Hi, Did we find a way to comment on the proposal? How can we proceed with this discussion?
Thanks, Satish. On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <[email protected]> wrote: > I've made a design document for adding jars and maven artifacts at > submission. > Since we don't have any formats for this, I just borrowed KIP format. > That's not we would like to adopt that format, it's just me. > > > https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing > > Btw, I guess ASF policy requires that discussion history should be logged > to Apache Infra. > (So I only grant view permission who opens the doc via link.) > > So what's recommended way to do? Would we want to file an issue for JIRA > with attaching design doc to PDF and discuss there? > > - Jungtaek Lim > > 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <[email protected]>님이 작성: > > > Thought about it once more, and found that former approach still needs to > > add 'provided' scope libraries to extlib directory.. > > > > Along with thinking former approach, I've experimented a bit of latter > > approach by creating POC project. Since I don't know about copy and use > ASF > > project codes for personal use, I'd just share main class to see if this > > POC and my theory makes sense for us. > > https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6 > > 'zeppelin' package has some classes for transitive dependency resolver > > (copied from Zeppelin), and 'spark' package has some classes for > > classloader (yes also copied from Spark). > > > > Please share your experiences if you have knowledges regarding this area. > > > > - Jungtaek Lim > > > > 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <[email protected]>님이 > 작성: > > > >> You can do that with a combination of the distributed cache and setting > >> the classpath, just like with hadoop. It is not as clean as it could > be, > >> but it does work. - Bobby > >> > >> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan < > [email protected]> > >> wrote: > >> > >> > >> Shade and relocate the external modules sounds ok as a short term > >> solution. > >> > >> For the long term we should consider something like the second option to > >> add external modules without shipping uber jars. > >> > >> Thanks, > >> Arun > >> > >> On 7/22/16, 6:07 AM, "Jungtaek Lim" <[email protected]> wrote: > >> > >> >Hi devs, > >> > > >> >AFAIK, we had been struggled to resolve dependency issues for > storm-core. > >> >As we all know, the strategy we have been using is shade & relocating. > >> > > >> >Now State and Storm SQL requires that some of external modules need to > be > >> >included to extlib, which is the classpath workers refer. > >> > > >> >http://issues.apache.org/jira/browse/STORM-1881 > >> >https://issues.apache.org/jira/browse/STORM-1435 > >> > > >> >There're two issues here: > >> >- We don't make uber jar for external modules so users need to find and > >> >copy dependencies jars to extlib manually. > >> >- External modules also use Guava and Jackson and so on which are > origin > >> of > >> >version conflict issues. > >> > > >> >So we should apply the shade & relocating strategy for every external > >> >modules (at least storm-redis, storm-kafka, storm-sql-core, > >> >storm-sql-kafka), or introduce the way to add the dependency without > >> adding > >> >them to extlib. (like --packages and --jar for Spark) > >> > > >> >Please express your opinions about this. > >> > > >> >Thanks, > >> >Jungtaek Lim (HeartSaVioR) > >> > >> > >> > >> > >> > > > > >
