thats interesting... if i understand it correctly it would cause compatibility breaks for applications on top of spark, because those applications use the exact same current resolution logic (so basically they are maven apps), and the change would make them inconsistent?
by that logic all existing applications on top of spark that do not use maven are already in danger of incompatibly breaks? or am i completely misunderstanding? this makes the implications of spark switching to maven a while back somewhat more serious than i realized. on the other hand we use sbt for our spark apps and this has never been an issue for us, so i am not sure how real/serious this compatibility issue is. , assuming those applications because those applications are maven applications that currently use the same exact logic, because you publish a pom because those applications build on top of spark currently assume On Fri, Nov 6, 2015 at 5:35 PM, Patrick Wendell <pwend...@gmail.com> wrote: > I think we'd have to standardize on Maven-style resolution, or I'd at > least like to see that path explored first. The issue is if we switch the > standard now, it could cause compatibility breaks for applications on top > of Spark. > > On Fri, Nov 6, 2015 at 2:28 PM, Jakob Odersky <joder...@gmail.com> wrote: > >> Reposting to the list... >> >> Thanks for all the feedback everyone, I get a clearer picture of the >> reasoning and implications now. >> >> Koert, according to your post in this thread >> http://apache-spark-developers-list.1001551.n3.nabble.com/Master-build-fails-tt14895.html#a15023, >> it is apparently very easy to change the maven resolution mechanism to the >> ivy one. >> Patrick, would this not help with the problems you described? >> >> On 5 November 2015 at 23:23, Patrick Wendell <pwend...@gmail.com> wrote: >> >>> Hey Jakob, >>> >>> The builds in Spark are largely maintained by me, Sean, and Michael >>> Armbrust (for SBT). For historical reasons, Spark supports both a Maven and >>> SBT build. Maven is the build of reference for packaging Spark and is used >>> by many downstream packagers and to build all Spark releases. SBT is more >>> often used by developers. Both builds inherit from the same pom files (and >>> rely on the same profiles) to minimize maintenance complexity of Spark's >>> very complex dependency graph. >>> >>> If you are looking to make contributions that help with the build, I am >>> happy to point you towards some things that are consistent maintenance >>> headaches. There are two major pain points right now that I'd be thrilled >>> to see fixes for: >>> >>> 1. SBT relies on a different dependency conflict resolution strategy >>> than maven - causing all kinds of headaches for us. I have heard that newer >>> versions of SBT can (maybe?) use Maven as a dependency resolver instead of >>> Ivy. This would make our life so much better if it were possible, either by >>> virtue of upgrading SBT or somehow doing this ourselves. >>> >>> 2. We don't have a great way of auditing the net effect of dependency >>> changes when people make them in the build. I am working on a fairly clunky >>> patch to do this here: >>> >>> https://github.com/apache/spark/pull/8531 >>> >>> It could be done much more nicely using SBT, but only provided (1) is >>> solved. >>> >>> Doing a major overhaul of the sbt build to decouple it from pom files, >>> I'm not sure that's the best place to start, given that we need to continue >>> to support maven - the coupling is intentional. But getting involved in the >>> build in general would be completely welcome. >>> >>> - Patrick >>> >>> On Thu, Nov 5, 2015 at 10:53 PM, Sean Owen <so...@cloudera.com> wrote: >>> >>>> Maven isn't 'legacy', or supported for the benefit of third parties. >>>> SBT had some behaviors / problems that Maven didn't relative to what >>>> Spark needs. SBT is a development-time alternative only, and partly >>>> generated from the Maven build. >>>> >>>> On Fri, Nov 6, 2015 at 1:48 AM, Koert Kuipers <ko...@tresata.com> >>>> wrote: >>>> > People who do upstream builds of spark (think bigtop and hadoop >>>> distros) are >>>> > used to legacy systems like maven, so maven is the default build. I >>>> don't >>>> > think it will change. >>>> > >>>> > Any improvements for the sbt build are of course welcome (it is still >>>> used >>>> > by many developers), but i would not do anything that increases the >>>> burden >>>> > of maintaining two build systems. >>>> > >>>> > On Nov 5, 2015 18:38, "Jakob Odersky" <joder...@gmail.com> wrote: >>>> >> >>>> >> Hi everyone, >>>> >> in the process of learning Spark, I wanted to get an overview of the >>>> >> interaction between all of its sub-projects. I therefore decided to >>>> have a >>>> >> look at the build setup and its dependency management. >>>> >> Since I am alot more comfortable using sbt than maven, I decided to >>>> try to >>>> >> port the maven configuration to sbt (with the help of automated >>>> tools). >>>> >> This led me to a couple of observations and questions on the build >>>> system >>>> >> design: >>>> >> >>>> >> First, currently, there are two build systems, maven and sbt. Is >>>> there a >>>> >> preferred tool (or future direction to one)? >>>> >> >>>> >> Second, the sbt build also uses maven "profiles" requiring the use of >>>> >> specific commandline parameters when starting sbt. Furthermore, >>>> since it >>>> >> relies on maven poms, dependencies to the scala binary version >>>> (_2.xx) are >>>> >> hardcoded and require running an external script when switching >>>> versions. >>>> >> Sbt could leverage built-in constructs to support cross-compilation >>>> and >>>> >> emulate profiles with configurations and new build targets. This >>>> would >>>> >> remove external state from the build (in that no extra steps need to >>>> be >>>> >> performed in a particular order to generate artifacts for a new >>>> >> configuration) and therefore improve stability and build >>>> reproducibility >>>> >> (maybe even build performance). I was wondering if implementing such >>>> >> functionality for the sbt build would be welcome? >>>> >> >>>> >> thanks, >>>> >> --Jakob >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>> >>>> >>> >> >
