Yup, we are making that work.
On Wed, Sep 4, 2013 at 9:19 PM, Matei Zaharia <[email protected]>wrote: > Ah, got it. You can do one other thing to shorten the list: package your > application into a single "assembly JAR". For SBT you can use this plugin: > https://github.com/sbt/sbt-assembly or for Maven use this: > http://maven.apache.org/plugins/maven-shade-plugin/. > > Matei > > On Sep 4, 2013, at 10:21 AM, Gary Malouf <[email protected]> wrote: > > That's how I do it now, list is getting lengthy but we are automating the > retrieving of the jars and list build up in ansible. > > > On Wed, Sep 4, 2013 at 12:55 PM, Matei Zaharia <[email protected]>wrote: > >> Hi Gary, >> >> Just to be clear, if you want to use third-party libraries in Spark (or >> even your own code), you *don't* need to modify SparkBuild.scala. Just pass >> a list of JARs containing your dependencies when you create your >> SparkContext. See >> http://spark.incubator.apache.org/docs/latest/quick-start.html for >> details. Spark will automatically ship those JARs to worker nodes and put >> them on the classpath for just this job. >> >> Matei >> >> On Sep 4, 2013, at 9:35 AM, Gary Malouf <[email protected]> wrote: >> >> To be clear, I was not referring to the Spark team rolling their own >> Date/Time utilities. In general though, I strongly disagree with just >> adding your own personal project dependencies to SparkBuild.scala - this >> quickly creates confusion and maintainability issues when one looks at >> upgrading. It appears we will just have to deal with ADD_JARS property for >> the foreseeable future. >> >> >> On Wed, Sep 4, 2013 at 12:17 PM, Mark Hamstra <[email protected]>wrote: >> >>> A couple of lines to include a build dependency and import a library vs. >>> all of the time to develop and maintain our own time-and-date code or all >>> of the user headache of having to work-around our choice the link in a >>> library that doesn't fit their particular needs. >>> >>> Until there is an obvious, stable and expected-in-almost-all-cases >>> third-party time-and-date library to chose, I strongly urge that we do not >>> bind Spark to a particular time-and-date library. (And there are a lot >>> better things that we could be doing with our time than developing on our >>> own yet another time-and-date implementation.) >>> >>> >>> On Wed, Sep 4, 2013 at 7:45 AM, Gary Malouf <[email protected]>wrote: >>> >>>> More setup that a user needs to do to reach his functional goals. >>>> >>>> >>>> On Wed, Sep 4, 2013 at 9:40 AM, Mark Hamstra >>>> <[email protected]>wrote: >>>> >>>>> Why? What is wrong with using the extant libraries? >>>>> >>>>> >>>>> >>>>> On Wed, Sep 4, 2013 at 6:37 AM, Gary Malouf <[email protected]>wrote: >>>>> >>>>>> Are there any built-in functions for timezone conversions? I can >>>>>> obviously bring in NScalaTime and other external libraries. However, >>>>>> being >>>>>> that this is probably a common need across companies I feel like it would >>>>>> make more sense to provide this out of the box. >>>>>> >>>>> >>>>> >>>> >>> >> >> > >
