It's strange for me, that having and support fat jar was never a important thing. We have next scenario - we have big application, where spark is just another library for data processing. So we can not create small jar and feed it to spark scripts - we need to call spark from application. And having fat jar as maven dependency is perfect. We have some spark installed on cluster(whatever cloudera put there), but often we need to patch spark for our needs, so we need to bring everything with us. Different departments use different spark versions - so we can not share jars on cluster easily. Yep, there are some disadvantages, but flexibility of changing spark process and deploying overcome these disadvantages.
So we probably would patch pom's as usual to create fat jar. 2016-06-14 12:23 GMT-07:00 Reynold Xin <r...@databricks.com>: > You just need to run normal packaging and all the scripts are now setup to > run without the assembly jars. > > > On Tuesday, June 14, 2016, Franklyn D'souza <franklyn.dso...@shopify.com> > wrote: > >> Just wondering where the spark-assembly jar has gone in 2.0. i've been >> reading that its been removed but i'm not sure what the new workflow is . >> > -- *Sincerely yoursEgor Pakhomov*
