There was a lot of discussion that preceded our arriving at this statement in the Spark documentation: "Maven is the official build tool recommended for packaging Spark, and is the build of reference." https://spark.apache.org/docs/latest/building-spark.html#building-with-sbt
I'm not aware of anything new in the way of SBT tooling or your post, Jakob, that would lead us to reconsider the choice of Maven over SBT for the reference build of Spark. Of course, I'm by no means the sole and final authority on the matter, but at least I am not seeing anything in your suggested approach that hasn't already been considered. You're welcome to review the prior discussion and try to convince us that we've made the wrong choice, but I wouldn't expect that to be a quick and easy process. On Thu, Nov 5, 2015 at 4:44 PM, Ted Yu <yuzhih...@gmail.com> wrote: > See previous discussion: > http://search-hadoop.com/m/q3RTtPnPnzwOhBr > > FYI > > On Thu, Nov 5, 2015 at 4:30 PM, Stephen Boesch <java...@gmail.com> wrote: > >> Yes. The current dev/change-scala-version.sh mutates (/pollutes) the >> build environment by updating the pom.xml in each of the subprojects. If >> you were able to come up with a structure that avoids that approach it >> would be an improvement. >> >> 2015-11-05 15:38 GMT-08:00 Jakob Odersky <joder...@gmail.com>: >> >>> Hi everyone, >>> in the process of learning Spark, I wanted to get an overview of the >>> interaction between all of its sub-projects. I therefore decided to have a >>> look at the build setup and its dependency management. >>> Since I am alot more comfortable using sbt than maven, I decided to try >>> to port the maven configuration to sbt (with the help of automated tools). >>> This led me to a couple of observations and questions on the build >>> system design: >>> >>> First, currently, there are two build systems, maven and sbt. Is there a >>> preferred tool (or future direction to one)? >>> >>> Second, the sbt build also uses maven "profiles" requiring the use of >>> specific commandline parameters when starting sbt. Furthermore, since it >>> relies on maven poms, dependencies to the scala binary version (_2.xx) are >>> hardcoded and require running an external script when switching versions. >>> Sbt could leverage built-in constructs to support cross-compilation and >>> emulate profiles with configurations and new build targets. This would >>> remove external state from the build (in that no extra steps need to be >>> performed in a particular order to generate artifacts for a new >>> configuration) and therefore improve stability and build reproducibility >>> (maybe even build performance). I was wondering if implementing such >>> functionality for the sbt build would be welcome? >>> >>> thanks, >>> --Jakob >>> >> >> >
