You can do that with a combination of the distributed cache and setting the
classpath, just like with hadoop. It is not as clean as it could be, but it
does work. - Bobby
On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan <ar...@apache.org>
wrote:
Shade and relocate the external modules sounds ok as a short term solution.
For the long term we should consider something like the second option to add
external modules without shipping uber jars.
Thanks,
Arun
On 7/22/16, 6:07 AM, "Jungtaek Lim" <kabh...@gmail.com> wrote:
>Hi devs,
>
>AFAIK, we had been struggled to resolve dependency issues for storm-core.
>As we all know, the strategy we have been using is shade & relocating.
>
>Now State and Storm SQL requires that some of external modules need to be
>included to extlib, which is the classpath workers refer.
>
>http://issues.apache.org/jira/browse/STORM-1881
>https://issues.apache.org/jira/browse/STORM-1435
>
>There're two issues here:
>- We don't make uber jar for external modules so users need to find and
>copy dependencies jars to extlib manually.
>- External modules also use Guava and Jackson and so on which are origin of
>version conflict issues.
>
>So we should apply the shade & relocating strategy for every external
>modules (at least storm-redis, storm-kafka, storm-sql-core,
>storm-sql-kafka), or introduce the way to add the dependency without adding
>them to extlib. (like --packages and --jar for Spark)
>
>Please express your opinions about this.
>
>Thanks,
>Jungtaek Lim (HeartSaVioR)
