Re: Performance degradation between spark 0.9.3 and 1.3.1

2015-05-22 Thread tyronecai
may because of snappy-java, https://issues.apache.org/jira/browse/SPARK-5081 On May 23, 2015, at 1:23 AM, Josh Rosen rosenvi...@gmail.com wrote: I don't think that 0.9.3 has been released, so I'm assuming that you're running on branch-0.9. There's been over 4000 commits between 0.9.3 and

Performance degradation between spark 0.9.3 and 1.3.1

2015-05-22 Thread Shay Seng
Hi. I have a job that takes ~50min with Spark 0.9.3 and ~1.8hrs on Spark 1.3.1 on the same cluster. The only code difference between the two code bases is to fix the Seq - Iter changes that happened in the Spark 1.x series. Are there any other changes in the defaults from spark 0.9.3 - 1.3.1

Re: Performance degradation between spark 0.9.3 and 1.3.1

2015-05-22 Thread Josh Rosen
I don't think that 0.9.3 has been released, so I'm assuming that you're running on branch-0.9. There's been over 4000 commits between 0.9.3 and 1.3.1, so I'm afraid that this question doesn't have a concise answer: https://github.com/apache/spark/compare/branch-0.9...v1.3.1 To narrow down the