Hi All, I've create pull request for this: https://github.com/apache/incubator-apex-core/pull/208
Please review the pull request. Thanks, Chinmay. On Fri, Jan 15, 2016 at 11:40 PM, Chinmay Kolhatkar <[email protected] > wrote: > Hi Everyone, > > If there are no further comments, I'll create the pull request for this > change. > > Tushar, About the 2 parts you've mentioned, 2nd part is taken care of by > this Jira. I'll log another Jira for 1st and address it there. > > I'll come back with pull request link soon. > > Thanks, > Chinmay. > > > > On Wed, Jan 13, 2016 at 12:18 AM, Tushar Gosavi <[email protected]> > wrote: > >> One use case could be that if we want same application to work on >> different >> data (schema) without >> compiling the application again. In this case schema can be provided as a >> json description, >> with field name and their type. When application starts it will generate >> the classes using ASM >> and write it to the temp jar, and configure operators to use the >> classname. >> operators can do a Class.forName to load the class based on name and use >> those. This could enable to users to write generic applications and by >> providing schema definition and configuring operators based on field >> names. >> >> There are two parts >> 1) Generating classes based on schema definition. >> 2) Making generated class available to operators. >> >> - Tushar. >> >> >> >> On Tue, Jan 12, 2016 at 10:11 AM, Chinmay Kolhatkar < >> [email protected] >> > wrote: >> >> > Hi Thomas, >> > >> > The module that I'm working on will have fields added to the POJO by >> > current operator as per requirement of further processing stage. >> > >> > The fields added may or may not directly go into the final output POJO. >> > Hence the intermediate stream is mean to carry the required data for >> > processing of next stage. >> > >> > Thanks, >> > Chinmay. >> > >> > >> > On Tue, Jan 12, 2016 at 9:47 AM, Thomas Weise <[email protected]> >> > wrote: >> > >> > > Since the output POJO is defined by the user, why is there a need to >> > > generate additional classes for intermediate streams within the >> module? >> > > >> > > Can you give an example? >> > > >> > > >> > > On Mon, Jan 11, 2016 at 8:11 PM, Chinmay Kolhatkar < >> > > [email protected]> >> > > wrote: >> > > >> > > > Hi Thomas/David, >> > > > >> > > > Here is the details of the usecase. >> > > > I'm writing a module which has following definition: >> > > > >> > > > <Input POJO> -> O1 -> O2 -> O3 -> <Output POJO> >> > > > >> > > > O1, O2 & O3 are the operators inside the module and module take >> certain >> > > > POJO as input and given certain type if POJO as output. >> > > > >> > > > The input and output POJO will be provide by user. >> > > > But the streams between O1 & O2 and O2 & O3 are supposed to carry >> POJO >> > > > whose definition is unknown to/does not concern user. >> > > > >> > > > These streams within the module is supposed to contain some >> > > intermediately >> > > > processed POJOs, the definition of which will be generated by >> > parameters >> > > > provided to the module. >> > > > >> > > > Here the way I'm trying to achieve this is to generate a class and >> jar >> > > for >> > > > the intermediate streams and want to add those class in classpath. >> > > > >> > > > I am able to generate the class and jar for the intermediate >> streams in >> > > > populateDAG but I could not find any way to dynamically add those >> > classes >> > > > to classpath. >> > > > Also, as these are temporary classes, they need to be deleted once >> its >> > > > utilized. >> > > > >> > > > Hence for this request. >> > > > >> > > > Please suggest if there is a better/cleaner approach to achieve >> this. >> > > > >> > > > Thanks, >> > > > Chinmay. >> > > > >> > > > >> > > > On Tue, Jan 12, 2016 at 5:21 AM, David Yan <[email protected]> >> > > wrote: >> > > > >> > > > > If I remember correctly, all the necessary jar files, including >> the >> > > ones >> > > > > specified with -libjars (LIBJARS_CONF_KEY_NAME), are copied to >> HDFS >> > > when >> > > > > the launch command returns with an appid, so it should be safe to >> > > delete >> > > > > temporary jar files after the launch command returns. >> > > > > >> > > > > David >> > > > > >> > > > > On Thu, Jan 7, 2016 at 11:41 PM, Tushar Gosavi < >> > [email protected] >> > > > >> > > > > wrote: >> > > > > >> > > > > > +1 >> > > > > > In past we achieved this by writing generated class to a jar >> file >> > in >> > > > /tmp >> > > > > > directory and adding that jar in LIBJARS_CONF_KEY_NAME >> > > > > > conf.set(StramAppLauncher.LIBJARS_CONF_KEY_NAME, >> > > > > conf.get(StramAppLauncher. >> > > > > > LIBJARS_CONF_KEY_NAME) + "," + generatedPojoJar); >> > > > > > This is a kind of hack and generated jar file does not get >> cleaned >> > up >> > > > as >> > > > > we >> > > > > > don't get any callback after application is submitted in >> > > > > > StreamingApplication. It may be >> > > > > > good idea to provide a api to register temp jar which will get >> > copied >> > > > to >> > > > > > application directory, and deleted (optionally) from local >> machine >> > > > after >> > > > > > application is submitted. >> > > > > > >> > > > > > Regards, >> > > > > > -Tushar. >> > > > > > >> > > > > > >> > > > > > On Fri, Jan 8, 2016 at 12:07 PM, Chinmay Kolhatkar < >> > > > > > [email protected]> >> > > > > > wrote: >> > > > > > >> > > > > > > Hi Everyone, >> > > > > > > >> > > > > > > I have a requirement where I need to add certain classes to >> > > classpath >> > > > > > > during application started (during initialization). >> > > > > > > >> > > > > > > I get to know the classes to be added to classpath only via >> > > property >> > > > of >> > > > > > > application i.e. populateDAG method is the first point where I >> > get >> > > to >> > > > > > know >> > > > > > > what need to be added to classpath. >> > > > > > > The class I want to add primary defines what a tuple would >> look >> > > like. >> > > > > > > Basically properties will have the field information, >> populateDAG >> > > > needs >> > > > > > to >> > > > > > > generate the class and add it to classpath for all the >> operators. >> > > > > > > >> > > > > > > Creating a class is not a problem, but adding to classpath is. >> > > > > > > Is there a hook present in Apex platform where I can add given >> > > .class >> > > > > > file >> > > > > > > to the classpath in populateDAG? >> > > > > > > >> > > > > > > If not, I was wondering if this will be a positive addition to >> > the >> > > > > > > platform. >> > > > > > > I have some code already created for this. If everyone agrees >> > this >> > > to >> > > > > be >> > > > > > a >> > > > > > > good addition to platform, I'll be happy to work on this and >> > > > contribute >> > > > > > to >> > > > > > > the platform. >> > > > > > > >> > > > > > > Please share your thoughts. >> > > > > > > >> > > > > > > Thanks, >> > > > > > > Chinmay. >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
