Okay I did some forensics with Sean Owen. Some things about this bug: 1. The underlying cause is that we added some code to make the tests of sub modules depend on the core tests. For unknown reasons this causes Spark to hit MSHADE-148 for *some* combinations of build profiles.
2. MSHADE-148 can be worked around by disabling building of "dependency reduced poms" because then the buggy code path is circumvented. Andrew Or did this in a patch on the 1.4 branch. However, that is not a tenable option for us because our *published* pom files require dependency reduction to substitute in the scala version correctly for the poms published to maven central. 3. As a result, Andrew Or reverted his patch recently, causing some package builds to start failing again (but publishing works now). 4. The reason this is not detected in our test harness or release build is that it is sensitive to the profiles enabled. The combination of profiles we enable in the test harness and release builds do not trigger this bug. The best path I see forward right now is to do the following: 1. Disable creation of dependency reduced poms by default (this doesn't matter for people doing a package build) so typical users won't have this bug. 2. Add a profile that re-enables that setting. 3. Use the above profile when publishing release artifacts to maven central. 4. Hope that we don't hit this bug for publishing. - Patrick On Fri, Jul 3, 2015 at 3:51 PM, Tarek Auel <tarek.a...@gmail.com> wrote: > Doesn't change anything for me. > > On Fri, Jul 3, 2015 at 3:45 PM Patrick Wendell <pwend...@gmail.com> wrote: >> >> Can you try using the built in maven "build/mvn..."? All of our builds >> are passing on Jenkins so I wonder if it's a maven version issue: >> >> https://amplab.cs.berkeley.edu/jenkins/view/Spark-QA-Compile/ >> >> - Patrick >> >> On Fri, Jul 3, 2015 at 3:14 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > Please take a look at SPARK-8781 >> > (https://github.com/apache/spark/pull/7193) >> > >> > Cheers >> > >> > On Fri, Jul 3, 2015 at 3:05 PM, Tarek Auel <tarek.a...@gmail.com> wrote: >> >> >> >> I found a solution, there might be a better one. >> >> >> >> https://github.com/apache/spark/pull/7217 >> >> >> >> On Fri, Jul 3, 2015 at 2:28 PM Robin East <robin.e...@xense.co.uk> >> >> wrote: >> >>> >> >>> Yes me too >> >>> >> >>> On 3 Jul 2015, at 22:21, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> >> >>> This is what I got (the last line was repeated non-stop): >> >>> >> >>> [INFO] Replacing original artifact with shaded artifact. >> >>> [INFO] Replacing >> >>> /home/hbase/spark/bagel/target/spark-bagel_2.10-1.5.0-SNAPSHOT.jar >> >>> with >> >>> >> >>> /home/hbase/spark/bagel/target/spark-bagel_2.10-1.5.0-SNAPSHOT-shaded.jar >> >>> [INFO] Dependency-reduced POM written at: >> >>> /home/hbase/spark/bagel/dependency-reduced-pom.xml >> >>> [INFO] Dependency-reduced POM written at: >> >>> /home/hbase/spark/bagel/dependency-reduced-pom.xml >> >>> >> >>> On Fri, Jul 3, 2015 at 1:13 PM, Tarek Auel <tarek.a...@gmail.com> >> >>> wrote: >> >>>> >> >>>> Hi all, >> >>>> >> >>>> I am trying to build the master, but it stucks and prints >> >>>> >> >>>> [INFO] Dependency-reduced POM written at: >> >>>> /Users/tarek/test/spark/bagel/dependency-reduced-pom.xml >> >>>> >> >>>> build command: mvn -DskipTests clean package >> >>>> >> >>>> Do others have the same issue? >> >>>> >> >>>> Regards, >> >>>> Tarek >> >>> >> >>> >> >>> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
