Hey Evan and TD, Spark's dependency graph in a maintenance release seems potentially harmful, especially upgrading a minor version (not just a patch version) like this. This could affect other downstream users. For instance, now without knowing their fastutil dependency gets bumped and they hit some new problem in fastutil 6.5.
- Patrick On Mon, Mar 24, 2014 at 12:02 AM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > @Shivaram, That is a useful patch but I am bit afraid merge it in. > Randomizing the executor has performance implications, especially for Spark > Streaming. The non-randomized ordering of allocating machines to tasks was > subtly helping to speed up certain window-based shuffle operations. For > example, corresponding shuffle partitions in multiple shuffles using the > same partitioner were likely to be co-located, that is, shuffle partition 0 > were likely to be on the same machine for multiple shuffles. While this is > the not a reliable mechanism to rely on, randomization may lead to > performance degradation. So I am afraid to merge this one without > understanding the consequences. > > @Evan, I have already cut a release! You can submit the PR and we can merge > it branch-0.9. If we have to cut another release, then we can include it. > > > > On Sun, Mar 23, 2014 at 11:42 PM, Evan Chan <e...@ooyala.com> wrote: > >> I also have a really minor fix for SPARK-1057 (upgrading fastutil), >> could that also make it in? >> >> -Evan >> >> >> On Sun, Mar 23, 2014 at 11:01 PM, Shivaram Venkataraman >> <shiva...@eecs.berkeley.edu> wrote: >> > Sorry this request is coming in a bit late, but would it be possible to >> > backport SPARK-979[1] to branch-0.9 ? This is the patch for randomizing >> > executor offers and I would like to use this in a release sooner rather >> > than later. >> > >> > Thanks >> > Shivaram >> > >> > [1] >> > >> https://github.com/apache/spark/commit/556c56689bbc32c6cec0d07b57bd3ec73ceb243e#diff-8ef3258646b0e6a4793d6ad99848eacd >> > >> > >> > On Thu, Mar 20, 2014 at 10:18 PM, Bhaskar Dutta <bhas...@gmail.com> >> wrote: >> > >> >> Thank You! We plan to test out 0.9.1 on YARN once it is out. >> >> >> >> Regards, >> >> Bhaskar >> >> >> >> On Fri, Mar 21, 2014 at 12:42 AM, Tom Graves <tgraves...@yahoo.com> >> wrote: >> >> >> >> > I'll pull [SPARK-1053] Should not require SPARK_YARN_APP_JAR when >> running >> >> > on YARN - JIRA and [SPARK-1051] On Yarn, executors don't doAs as >> >> > submitting user - JIRA in. The pyspark one I would consider more of >> an >> >> > enhancement so might not be appropriate for a point release. >> >> > >> >> > >> >> > [SPARK-1053] Should not require SPARK_YARN_APP_JAR when running on >> YA... >> >> > org.apache.spark.SparkException: env SPARK_YARN_APP_JAR is not set at >> >> > >> >> >> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:49) >> >> > at org.apache.spark.schedule... >> >> > View on spark-project.atlassian.net Preview by Yahoo >> >> > >> >> > >> >> > [SPARK-1051] On Yarn, executors don't doAs as submitting user - JIRA >> >> > This means that they can't write/read from files that the yarn user >> >> > doesn't have permissions to but the submitting user does. >> >> > View on spark-project.atlassian.net Preview by Yahoo >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Thursday, March 20, 2014 1:35 PM, Bhaskar Dutta <bhas...@gmail.com >> > >> >> > wrote: >> >> > >> >> > It will be great if >> >> > "SPARK-1101<https://spark-project.atlassian.net/browse/SPARK-1101>: >> >> > Umbrella >> >> > for hardening Spark on YARN" can get into 0.9.1. >> >> > >> >> > Thanks, >> >> > Bhaskar >> >> > >> >> > >> >> > On Thu, Mar 20, 2014 at 5:37 AM, Tathagata Das >> >> > <tathagata.das1...@gmail.com>wrote: >> >> > >> >> > > Hello everyone, >> >> > > >> >> > > Since the release of Spark 0.9, we have received a number of >> important >> >> > bug >> >> > > fixes and we would like to make a bug-fix release of Spark 0.9.1. We >> >> are >> >> > > going to cut a release candidate soon and we would love it if people >> >> test >> >> > > it out. We have backported several bug fixes into the 0.9 and >> updated >> >> > JIRA >> >> > > accordingly< >> >> > > >> >> > >> >> >> https://spark-project.atlassian.net/browse/SPARK-1275?jql=project%20in%20(SPARK%2C%20BLINKDB%2C%20MLI%2C%20MLLIB%2C%20SHARK%2C%20STREAMING%2C%20GRAPH%2C%20TACHYON)%20AND%20fixVersion%20%3D%200.9.1%20AND%20status%20in%20(Resolved%2C%20Closed) >> >> > > >. >> >> > > Please let me know if there are fixes that were not backported but >> you >> >> > > would like to see them in 0.9.1. >> >> > > >> >> > > Thanks! >> >> > > >> >> > > TD >> >> > > >> >> > >> >> >> >> >> >> -- >> -- >> Evan Chan >> Staff Engineer >> e...@ooyala.com | >>
