Re: how to run a dev spark project without fully rebuilding the fat jar ?

[Akhil Das]

You can use the --jars option to submit multiple jars using the
spark-submit, so you can simply build the jar that you have modified.

Thanks
Best Regards

On Thu, Oct 23, 2014 at 11:16 AM, Mohit Jaggi mohitja...@gmail.com wrote:

 i think you can give a list of jars - not just one - to spark-submit, so
 build only the one that has changed source code.

 On Wed, Oct 22, 2014 at 10:29 PM, Yang tedd...@gmail.com wrote:

 during tests, I often modify my code a little bit  and want to see the
 result.
 but spark-submit requires the full fat-jar, which takes quite a lot of
 time to build.

 I just need to run in --master local mode. is there a way to run it
 without rebuilding the fat jar?

 thanks
 Yang

how to run a dev spark project without fully rebuilding the fat jar ?

[Yang]

during tests, I often modify my code a little bit  and want to see the
result.
but spark-submit requires the full fat-jar, which takes quite a lot of time
to build.

I just need to run in --master local mode. is there a way to run it without
rebuilding the fat jar?

thanks
Yang

Re: how to run a dev spark project without fully rebuilding the fat jar ?

[Mohit Jaggi]

i think you can give a list of jars - not just one - to spark-submit, so
build only the one that has changed source code.

On Wed, Oct 22, 2014 at 10:29 PM, Yang tedd...@gmail.com wrote:

 during tests, I often modify my code a little bit  and want to see the
 result.
 but spark-submit requires the full fat-jar, which takes quite a lot of
 time to build.

 I just need to run in --master local mode. is there a way to run it
 without rebuilding the fat jar?

 thanks
 Yang