Sorry, The quick reply didn't cc the dev list. Sean, sometimes I have to use the spark-shell to confirm some behavior change. In that case, I have to reassembly the whole project. is there another way around, not use the the big jar in development? For the original question, I have no comments.
-- Ye Xianjin Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, September 2, 2014 at 4:58 PM, Sean Owen wrote: > No, usually you unit-test your changes during development. That > doesn't require the assembly. Eventually you may wish to test some > change against the complete assembly. > > But that's a different question; I thought you were suggesting that > the assembly JAR should never be created. > > On Tue, Sep 2, 2014 at 9:53 AM, Ye Xianjin <advance...@gmail.com > (mailto:advance...@gmail.com)> wrote: > > Hi, Sean: > > In development, do I really need to reassembly the whole project even if I > > only change a line or two code in one component? > > I used to that but found time-consuming. > > > > -- > > Ye Xianjin > > Sent with Sparrow > > > > On Tuesday, September 2, 2014 at 4:45 PM, Sean Owen wrote: > > > > Hm, are you suggesting that the Spark distribution be a bag of 100 > > JARs? It doesn't quite seem reasonable. It does not remove version > > conflicts, just pushes them to run-time, which isn't good. The > > assembly is also necessary because that's where shading happens. In > > development, you want to run against exactly what will be used in a > > real Spark distro. > > > > On Tue, Sep 2, 2014 at 9:39 AM, scwf <wangf...@huawei.com > > (mailto:wangf...@huawei.com)> wrote: > > > > hi, all > > I suggest spark not use assembly jar as default run-time > > dependency(spark-submit/spark-class depend on assembly jar),use a library of > > all 3rd dependency jar like hadoop/hive/hbase more reasonable. > > > > 1 assembly jar packaged all 3rd jars into a big one, so we need rebuild > > this jar if we want to update the version of some component(such as hadoop) > > 2 in our practice with spark, sometimes we meet jar compatibility issue, > > it is hard to diagnose compatibility issue with assembly jar > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > (mailto:dev-unsubscr...@spark.apache.org) > > For additional commands, e-mail: dev-h...@spark.apache.org > > (mailto:dev-h...@spark.apache.org) > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > > (mailto:dev-unsubscr...@spark.apache.org) > > For additional commands, e-mail: dev-h...@spark.apache.org > > (mailto:dev-h...@spark.apache.org) > > > > >
