I am not sure I fully understand this reasoning. I imagine that lift-json is only one of hundreds of packages that would have to be built if you wanted to build all of Spark's transitive dependencies from source.
Additionally, to make sure I understand the impact -- this is only intended to simplify the process of packaging Spark on a new OS distribution that disallows pulling in binaries? On Sun, Feb 9, 2014 at 4:50 PM, Will Benton <wi...@redhat.com> wrote: > Matei, sorry if I was unclear: I'm referring to downstream operating > system distributions (like Fedora or Debian) that have policies requiring > that all packages are built from source (using only tools already packaged > in the distribution). So end-users (and distributions with different > policies) don't have to build Lift to get the lift-json artifact, but it is > a concern for many open-source communities. > > best, > wb > > > ----- Original Message ----- > > From: "Matei Zaharia" <matei.zaha...@gmail.com> > > To: dev@spark.incubator.apache.org > > Sent: Sunday, February 9, 2014 4:29:20 PM > > Subject: Re: proposal: replace lift-json with spray-json > > > > Will, why are you saying that downstream distributes need to build all of > > Lift to package lift-json? Spark just downloads it from Maven Central, > where > > it's a JAR with no external dependencies. We don't have any dependency on > > the rest of lift. > > > > Matei > > > > On Feb 9, 2014, at 11:28 AM, Will Benton <wi...@redhat.com> wrote: > > > > > lift-json is a nice library, but Lift is a pretty heavyweight > dependency to > > > track just for its JSON support. (lift-json is relatively > self-contained > > > as a dependency from an end-user's perspective, but downstream > > > distributors need to build all of Lift in order to package the JSON > > > support.) I understand that this has come up before (cf. SPARK-883) > and > > > that the uncertain future of JSON support in the Scala standard > library is > > > the motivator for relying on an external library. > > > > > > I'm proposing replacing lift-json in Spark with something more > lightweight. > > > I've evaluated apparent project liveness and dependency scope for most > of > > > the current Scala JSON libraries and believe the best candidate is > > > spray-json (https://github.com/spray/spray-json), the JSON library > used by > > > the Spray HTTP toolkit. spray-json is Apache-licensed, actively > developed, > > > and builds and works independently of Spray with only one external > > > dependency. > > > > > > It looks to me like a pretty straightforward change (although > > > JsonProtocol.scala would be a little more verbose since it couldn't use > > > the Lift JSON DSL), and I'd like to do it. I'm writing now to ask for > > > some community feedback before making the change (and submitting a JIRA > > > and PR). If no one has any serious objections (to the effort in > general > > > or to to the choice of spark-json in particular), I'll go ahead and do > it, > > > but if anyone has concerns, I'd be happy to discuss and address them > > > before getting started. > > > > > > > > > thanks, > > > wb > > > > >
