yes, i am not sure what happens when building assembly jar and in my
understanding it just package all the dependency jars to a big one.
On 2014/9/2 16:45, Sean Owen wrote:
Hm, are you suggesting that the Spark distribution be a bag of 100
JARs? It doesn't quite seem reasonable. It does not remove version
conflicts, just pushes them to run-time, which isn't good. The
assembly is also necessary because that's where shading happens. In
development, you want to run against exactly what will be used in a
real Spark distro.
On Tue, Sep 2, 2014 at 9:39 AM, scwf <wangf...@huawei.com> wrote:
hi, all
I suggest spark not use assembly jar as default run-time
dependency(spark-submit/spark-class depend on assembly jar),use a library of
all 3rd dependency jar like hadoop/hive/hbase more reasonable.
1 assembly jar packaged all 3rd jars into a big one, so we need rebuild
this jar if we want to update the version of some component(such as hadoop)
2 in our practice with spark, sometimes we meet jar compatibility issue,
it is hard to diagnose compatibility issue with assembly jar
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
