Gustafson's Law and the Universal Scalability Law
Symptom Two measurements that both look like they break the previous post. The first: your Spark job takes four hours on ten nodes. You move it to a hundred nodes and it still takes four hours — but it is now processing ten times the data. T081 said the ceiling was 20×, and nobody hit a ceiling. ...