One use case could be that if we want same application to work on different data (schema) without compiling the application again. In this case schema can be provided as a json description, with field name and their type. When application starts it will generate the classes using ASM and write it to the temp jar, and configure operators to use the classname. operators can do a Class.forName to load the class based on name and use those. This could enable to users to write generic applications and by providing schema definition and configuring operators based on field names.
There are two parts 1) Generating classes based on schema definition. 2) Making generated class available to operators. - Tushar. On Tue, Jan 12, 2016 at 10:11 AM, Chinmay Kolhatkar <[email protected] > wrote: > Hi Thomas, > > The module that I'm working on will have fields added to the POJO by > current operator as per requirement of further processing stage. > > The fields added may or may not directly go into the final output POJO. > Hence the intermediate stream is mean to carry the required data for > processing of next stage. > > Thanks, > Chinmay. > > > On Tue, Jan 12, 2016 at 9:47 AM, Thomas Weise <[email protected]> > wrote: > > > Since the output POJO is defined by the user, why is there a need to > > generate additional classes for intermediate streams within the module? > > > > Can you give an example? > > > > > > On Mon, Jan 11, 2016 at 8:11 PM, Chinmay Kolhatkar < > > [email protected]> > > wrote: > > > > > Hi Thomas/David, > > > > > > Here is the details of the usecase. > > > I'm writing a module which has following definition: > > > > > > <Input POJO> -> O1 -> O2 -> O3 -> <Output POJO> > > > > > > O1, O2 & O3 are the operators inside the module and module take certain > > > POJO as input and given certain type if POJO as output. > > > > > > The input and output POJO will be provide by user. > > > But the streams between O1 & O2 and O2 & O3 are supposed to carry POJO > > > whose definition is unknown to/does not concern user. > > > > > > These streams within the module is supposed to contain some > > intermediately > > > processed POJOs, the definition of which will be generated by > parameters > > > provided to the module. > > > > > > Here the way I'm trying to achieve this is to generate a class and jar > > for > > > the intermediate streams and want to add those class in classpath. > > > > > > I am able to generate the class and jar for the intermediate streams in > > > populateDAG but I could not find any way to dynamically add those > classes > > > to classpath. > > > Also, as these are temporary classes, they need to be deleted once its > > > utilized. > > > > > > Hence for this request. > > > > > > Please suggest if there is a better/cleaner approach to achieve this. > > > > > > Thanks, > > > Chinmay. > > > > > > > > > On Tue, Jan 12, 2016 at 5:21 AM, David Yan <[email protected]> > > wrote: > > > > > > > If I remember correctly, all the necessary jar files, including the > > ones > > > > specified with -libjars (LIBJARS_CONF_KEY_NAME), are copied to HDFS > > when > > > > the launch command returns with an appid, so it should be safe to > > delete > > > > temporary jar files after the launch command returns. > > > > > > > > David > > > > > > > > On Thu, Jan 7, 2016 at 11:41 PM, Tushar Gosavi < > [email protected] > > > > > > > wrote: > > > > > > > > > +1 > > > > > In past we achieved this by writing generated class to a jar file > in > > > /tmp > > > > > directory and adding that jar in LIBJARS_CONF_KEY_NAME > > > > > conf.set(StramAppLauncher.LIBJARS_CONF_KEY_NAME, > > > > conf.get(StramAppLauncher. > > > > > LIBJARS_CONF_KEY_NAME) + "," + generatedPojoJar); > > > > > This is a kind of hack and generated jar file does not get cleaned > up > > > as > > > > we > > > > > don't get any callback after application is submitted in > > > > > StreamingApplication. It may be > > > > > good idea to provide a api to register temp jar which will get > copied > > > to > > > > > application directory, and deleted (optionally) from local machine > > > after > > > > > application is submitted. > > > > > > > > > > Regards, > > > > > -Tushar. > > > > > > > > > > > > > > > On Fri, Jan 8, 2016 at 12:07 PM, Chinmay Kolhatkar < > > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi Everyone, > > > > > > > > > > > > I have a requirement where I need to add certain classes to > > classpath > > > > > > during application started (during initialization). > > > > > > > > > > > > I get to know the classes to be added to classpath only via > > property > > > of > > > > > > application i.e. populateDAG method is the first point where I > get > > to > > > > > know > > > > > > what need to be added to classpath. > > > > > > The class I want to add primary defines what a tuple would look > > like. > > > > > > Basically properties will have the field information, populateDAG > > > needs > > > > > to > > > > > > generate the class and add it to classpath for all the operators. > > > > > > > > > > > > Creating a class is not a problem, but adding to classpath is. > > > > > > Is there a hook present in Apex platform where I can add given > > .class > > > > > file > > > > > > to the classpath in populateDAG? > > > > > > > > > > > > If not, I was wondering if this will be a positive addition to > the > > > > > > platform. > > > > > > I have some code already created for this. If everyone agrees > this > > to > > > > be > > > > > a > > > > > > good addition to platform, I'll be happy to work on this and > > > contribute > > > > > to > > > > > > the platform. > > > > > > > > > > > > Please share your thoughts. > > > > > > > > > > > > Thanks, > > > > > > Chinmay. > > > > > > > > > > > > > > > > > > > > >
