Since the output POJO is defined by the user, why is there a need to generate additional classes for intermediate streams within the module?
Can you give an example? On Mon, Jan 11, 2016 at 8:11 PM, Chinmay Kolhatkar <[email protected]> wrote: > Hi Thomas/David, > > Here is the details of the usecase. > I'm writing a module which has following definition: > > <Input POJO> -> O1 -> O2 -> O3 -> <Output POJO> > > O1, O2 & O3 are the operators inside the module and module take certain > POJO as input and given certain type if POJO as output. > > The input and output POJO will be provide by user. > But the streams between O1 & O2 and O2 & O3 are supposed to carry POJO > whose definition is unknown to/does not concern user. > > These streams within the module is supposed to contain some intermediately > processed POJOs, the definition of which will be generated by parameters > provided to the module. > > Here the way I'm trying to achieve this is to generate a class and jar for > the intermediate streams and want to add those class in classpath. > > I am able to generate the class and jar for the intermediate streams in > populateDAG but I could not find any way to dynamically add those classes > to classpath. > Also, as these are temporary classes, they need to be deleted once its > utilized. > > Hence for this request. > > Please suggest if there is a better/cleaner approach to achieve this. > > Thanks, > Chinmay. > > > On Tue, Jan 12, 2016 at 5:21 AM, David Yan <[email protected]> wrote: > > > If I remember correctly, all the necessary jar files, including the ones > > specified with -libjars (LIBJARS_CONF_KEY_NAME), are copied to HDFS when > > the launch command returns with an appid, so it should be safe to delete > > temporary jar files after the launch command returns. > > > > David > > > > On Thu, Jan 7, 2016 at 11:41 PM, Tushar Gosavi <[email protected]> > > wrote: > > > > > +1 > > > In past we achieved this by writing generated class to a jar file in > /tmp > > > directory and adding that jar in LIBJARS_CONF_KEY_NAME > > > conf.set(StramAppLauncher.LIBJARS_CONF_KEY_NAME, > > conf.get(StramAppLauncher. > > > LIBJARS_CONF_KEY_NAME) + "," + generatedPojoJar); > > > This is a kind of hack and generated jar file does not get cleaned up > as > > we > > > don't get any callback after application is submitted in > > > StreamingApplication. It may be > > > good idea to provide a api to register temp jar which will get copied > to > > > application directory, and deleted (optionally) from local machine > after > > > application is submitted. > > > > > > Regards, > > > -Tushar. > > > > > > > > > On Fri, Jan 8, 2016 at 12:07 PM, Chinmay Kolhatkar < > > > [email protected]> > > > wrote: > > > > > > > Hi Everyone, > > > > > > > > I have a requirement where I need to add certain classes to classpath > > > > during application started (during initialization). > > > > > > > > I get to know the classes to be added to classpath only via property > of > > > > application i.e. populateDAG method is the first point where I get to > > > know > > > > what need to be added to classpath. > > > > The class I want to add primary defines what a tuple would look like. > > > > Basically properties will have the field information, populateDAG > needs > > > to > > > > generate the class and add it to classpath for all the operators. > > > > > > > > Creating a class is not a problem, but adding to classpath is. > > > > Is there a hook present in Apex platform where I can add given .class > > > file > > > > to the classpath in populateDAG? > > > > > > > > If not, I was wondering if this will be a positive addition to the > > > > platform. > > > > I have some code already created for this. If everyone agrees this to > > be > > > a > > > > good addition to platform, I'll be happy to work on this and > contribute > > > to > > > > the platform. > > > > > > > > Please share your thoughts. > > > > > > > > Thanks, > > > > Chinmay. > > > > > > > > > >
