I recommend you package all your dependencies (jars, .so’s, etc.) into a single uber-jar and then submit that. It’s much more convenient than trying to manage including everything in the --jars arg of spark-submit. If you build with maven than the shade plugin will do this for you nicely: https://maven.apache.org/plugins/maven-shade-plugin/
> On Mar 11, 2016, at 2:05 PM, Jacek Laskowski <ja...@japila.pl> wrote: > > Hi, > > For jars use spark-submit --jars. Dunno about so's. Could that work through > jars? > > Jacek > > 11.03.2016 8:07 PM "prateek arora" <prateek.arora...@gmail.com > <mailto:prateek.arora...@gmail.com>> napisał(a): > Hi > > I have multiple node cluster and my spark jobs depend on a native > library (.so files) and some jar files. > > Can some one please explain what are the best ways to distribute dependent > files across nodes? > > right now i copied dependent files in all nodes using chef tool . > > Regards > Prateek > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-distribute-dependent-files-so-jar-across-spark-worker-nodes-tp26464.html > > <http://apache-spark-user-list.1001560.n3.nabble.com/How-to-distribute-dependent-files-so-jar-across-spark-worker-nodes-tp26464.html> > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > <mailto:user-unsubscr...@spark.apache.org> > For additional commands, e-mail: user-h...@spark.apache.org > <mailto:user-h...@spark.apache.org> >
