Thanks Taylor! Let me open a new thread for proposal and continue discussion from there.
Jungtaek Lim (HeartSaVioR) 2016년 8월 2일 (화) 오전 12:10, P. Taylor Goetz <[email protected]>님이 작성: > Jungtaek, I’ve given you permissions to the Storm wiki so you can post the > document there. > > -Taylor > > > On Aug 1, 2016, at 12:46 AM, Jungtaek Lim <[email protected]> wrote: > > > > Not yet. It would be good to continue discussion on this thread for now. > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) > > > > 2016년 8월 1일 (월) 오후 1:38, Satish Duggana <[email protected]>님이 작성: > > > >> Hi, > >> Did we find a way to comment on the proposal? How can we proceed with > this > >> discussion? > >> > >> Thanks, > >> Satish. > >> > >> > >> On Thu, Jul 28, 2016 at 9:42 AM, Jungtaek Lim <[email protected]> > wrote: > >> > >>> I've made a design document for adding jars and maven artifacts at > >>> submission. > >>> Since we don't have any formats for this, I just borrowed KIP format. > >>> That's not we would like to adopt that format, it's just me. > >>> > >>> > >>> > >> > https://docs.google.com/document/d/1f3Ed0Wa8aN2j7gtptT0BOKYMyBdpohLgERg4YebJJ7c/edit?usp=sharing > >>> > >>> Btw, I guess ASF policy requires that discussion history should be > logged > >>> to Apache Infra. > >>> (So I only grant view permission who opens the doc via link.) > >>> > >>> So what's recommended way to do? Would we want to file an issue for > JIRA > >>> with attaching design doc to PDF and discuss there? > >>> > >>> - Jungtaek Lim > >>> > >>> 2016년 7월 23일 (토) 오전 12:06, Jungtaek Lim <[email protected]>님이 작성: > >>> > >>>> Thought about it once more, and found that former approach still needs > >> to > >>>> add 'provided' scope libraries to extlib directory.. > >>>> > >>>> Along with thinking former approach, I've experimented a bit of latter > >>>> approach by creating POC project. Since I don't know about copy and > use > >>> ASF > >>>> project codes for personal use, I'd just share main class to see if > >> this > >>>> POC and my theory makes sense for us. > >>>> https://gist.github.com/HeartSaVioR/3639a9ee829fe1203b4a085a0fb069d6 > >>>> 'zeppelin' package has some classes for transitive dependency resolver > >>>> (copied from Zeppelin), and 'spark' package has some classes for > >>>> classloader (yes also copied from Spark). > >>>> > >>>> Please share your experiences if you have knowledges regarding this > >> area. > >>>> > >>>> - Jungtaek Lim > >>>> > >>>> 2016년 7월 22일 (금) 오후 10:59, Bobby Evans <[email protected] > >님이 > >>> 작성: > >>>> > >>>>> You can do that with a combination of the distributed cache and > >> setting > >>>>> the classpath, just like with hadoop. It is not as clean as it could > >>> be, > >>>>> but it does work. - Bobby > >>>>> > >>>>> On Thursday, July 21, 2016 11:09 PM, Arun Mahadevan < > >>> [email protected]> > >>>>> wrote: > >>>>> > >>>>> > >>>>> Shade and relocate the external modules sounds ok as a short term > >>>>> solution. > >>>>> > >>>>> For the long term we should consider something like the second option > >> to > >>>>> add external modules without shipping uber jars. > >>>>> > >>>>> Thanks, > >>>>> Arun > >>>>> > >>>>> On 7/22/16, 6:07 AM, "Jungtaek Lim" <[email protected]> wrote: > >>>>> > >>>>>> Hi devs, > >>>>>> > >>>>>> AFAIK, we had been struggled to resolve dependency issues for > >>> storm-core. > >>>>>> As we all know, the strategy we have been using is shade & > >> relocating. > >>>>>> > >>>>>> Now State and Storm SQL requires that some of external modules need > >> to > >>> be > >>>>>> included to extlib, which is the classpath workers refer. > >>>>>> > >>>>>> http://issues.apache.org/jira/browse/STORM-1881 > >>>>>> https://issues.apache.org/jira/browse/STORM-1435 > >>>>>> > >>>>>> There're two issues here: > >>>>>> - We don't make uber jar for external modules so users need to find > >> and > >>>>>> copy dependencies jars to extlib manually. > >>>>>> - External modules also use Guava and Jackson and so on which are > >>> origin > >>>>> of > >>>>>> version conflict issues. > >>>>>> > >>>>>> So we should apply the shade & relocating strategy for every > external > >>>>>> modules (at least storm-redis, storm-kafka, storm-sql-core, > >>>>>> storm-sql-kafka), or introduce the way to add the dependency without > >>>>> adding > >>>>>> them to extlib. (like --packages and --jar for Spark) > >>>>>> > >>>>>> Please express your opinions about this. > >>>>>> > >>>>>> Thanks, > >>>>>> Jungtaek Lim (HeartSaVioR) > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>> > >>>> > >>> > >> > >
