Hi Priyank, first of all, this feature is similar (close) to what Spark provides. https://spark.apache.org/docs/2.0.0/submitting-applications.html#advanced-dependency-management
if you have additional jars which are not packed to uber topology jar, you can use --jars option to include them without repackaging topology jar. And I think I was not clear on submitter. I'm still trying to design that point in detail since resolving dependencies need eclipse aether libraries so thinking about avoiding to add dependency to storm-core. But it seems not that easy and clear. I'll update once I'm clear on this. Thanks, Jungtaek Lim (HeartSaVioR) 2016년 8월 3일 (수) 오전 7:43, Priyank Shah <[email protected]>님이 작성: > Hi Jungtaek, > > For adding jars and maven at submission, you have used the word Submitter. > Is Submitter the person running storm jar command or is Submitter the java > code that actually submits it to Nimbus? > Also, I did not quite understand the --jars option. If you could please > elaborate a little on that, that will be great > > Thanks > Priyank > > > > > > > On 8/2/16, 7:05 AM, "Jungtaek Lim" <[email protected]> wrote: > > >Ah, Satish you got the point. I meant copied version of files in > >supervisor, but itself can be isolated. > >I didn't think about removing blobs, and it seems not easy to do. > > > >Jungtaek Lim (HeartSaVIoR) > > > > > >2016년 8월 2일 (화) 오후 7:35, Satish Duggana <[email protected]>님이 작성: > > > >> Hi Jungtaek, > >> With the current proposal, are we removing blob store files referred by > a > >> topology when it is killed? > >> > >> Thanks, > >> Satish. > >> > >> On Tue, Aug 2, 2016 at 3:50 PM, Jungtaek Lim <[email protected]> wrote: > >> > >> > Hi Satish, > >> > > >> > Thanks for reviewing and share your idea. > >> > > >> > Yes this is shared dependencies vs isolated dependencies. > >> > If we name file of dependency to contain group name, artifact name, > and > >> > version, that can be shared. > >> > One downside of this approach is storage space since we don't know > when > >> > it's safe to delete without additional care, but I'm curious that disk > >> > fills up due to dependency blob jar files in normal situation. > >> > So I think we're OK to do this but I would like to see others > opinions. > >> > > >> > Btw, I'm designing details based on proposal. Will update to this > thread > >> if > >> > there're not covered things with initial design. > >> > > >> > Thanks, > >> > Jungtaek Lim (HeartSaVioR) > >> > > >> > 2016년 8월 2일 (화) 오후 6:58, Satish Duggana <[email protected]>님이 > 작성: > >> > > >> > > Hi Jungtaek, > >> > > Proposal looks good to me. Good that we are not going with other > >> > > alternative using mutable classloader etc. > >> > > > >> > > Good to have the mentioned config in proposal to add those jars > before > >> or > >> > > after storm core/libs. There is a property Config. > >> > > TOPOLOGY_CLASSPATH_BEGINNING which is to have that value as initial > >> > > classpath and that should continue to be working as expected even > with > >> > the > >> > > new configuration. > >> > > > >> > > One enhancement which we may want to add to the existing proposal. > >> > > When --packages are used, storm submitter can upload those > dependencies > >> > in > >> > > blob store with a defined naming convention so that same set of > >> packages > >> > > are not uploaded again and they can be used again for other > topologies > >> if > >> > > they use same package. > >> > > > >> > > Thanks, > >> > > Satish. > >> > > > >> > > > >> > > On Tue, Aug 2, 2016 at 7:25 AM, Jungtaek Lim <[email protected]> > >> wrote: > >> > > > >> > > > Hi dev, > >> > > > > >> > > > This is proposal review thread for submitting topology with adding > >> jars > >> > > and > >> > > > maven artifacts. This is also following up discussion thread for > >> > > > [DISCUSSION] > >> > > > Policy of resolving dependencies for non storm-core modules.[1] > >> > > > > >> > > > I've written design doc which also describes motivation on this. > >> > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/STORM/A.+Design+doc%3A+adding+jars+and+maven+artifacts+at+submission > >> > > > > >> > > > Please review this and comment to "this thread" instead of wiki > page > >> so > >> > > > that all devs can be notified for the update. > >> > > > > >> > > > Thanks, > >> > > > Jungtaek Lim (HeartSaVioR) > >> > > > > >> > > > [1] > >> > > > > >> > > > > >> > > > >> > > >> > http://mail-archives.apache.org/mod_mbox/storm-dev/201607.mbox/%3CCAF5108jByyJLTKrV_P4fS=dj8rsr_o5oubzqbviscggsc1c...@mail.gmail.com%3E > >> > > > > >> > > > >> > > >> >
