Hello Community, I've created a Jira for this APEXMALHAR-2010 and also a PR for the same: https://github.com/apache/incubator-apex-malhar/pull/209
I would like to get this PR review and then after review, merged into apex malhar. If you get time, can you please review the code and provide your valuable feedback? Thanks in advance, Chinmay. On Thu, Mar 10, 2016 at 2:29 PM, Chinmay Kolhatkar <[email protected]> wrote: > Thanks everyone for the feedback. > > @Yogi, the reason for "transform" name is that the tuple will be > transformed here. > > If there are no further comments on this, I'll start working on this and > will create a PR soon. > First PR I'll create is for enhancements to PojoUtils as mentioned above > and second PR will be for transform operator. > > Thanks, > Chinmay, > > > On Tue, Mar 8, 2016 at 5:48 PM, Devendra Tagare <[email protected] > > wrote: > >> +1 >> >> Dev >> >> >> On Tue, Mar 8, 2016 at 5:38 PM, Yogi Devendra <[email protected]> >> wrote: >> >> > Forgot to add in earlier email: >> > >> > +1 for this operator. >> > >> > ~ Yogi >> > >> > On 8 March 2016 at 17:03, Yogi Devendra <[email protected]> >> wrote: >> > >> > > Can we think of better name than Transform operator? >> > > May be: >> > > Expression operator >> > > >> > > Reason: >> > > We have many operators which are doing 'transform'. So, this name >> looks >> > > very generic. >> > > >> > > In fact, from the users perspective, I always consider any operator >> like >> > a >> > > black-box doing some transformation. >> > > >> > > ~ Yogi >> > > >> > > On 8 March 2016 at 16:46, Mohit Jotwani <[email protected]> >> wrote: >> > > >> > >> This can have as many util functions added to it to perform various >> > >> transformations to the tuple. >> > >> >> > >> +1 >> > >> >> > >> Regards, >> > >> Mohit >> > >> On 8 Mar 2016 16:42, "Sandeep Deshmukh" <[email protected]> >> > wrote: >> > >> >> > >> > Looks good to me. >> > >> > >> > >> > Regards, >> > >> > Sandeep >> > >> > >> > >> > On Mon, Mar 7, 2016 at 9:51 PM, Chinmay Kolhatkar < >> [email protected] >> > > >> > >> > wrote: >> > >> > >> > >> > > Dear Community, >> > >> > > >> > >> > > We're creating a transform operator which will allow the apex >> users >> > to >> > >> > > transform data over a stream using this operator. >> > >> > > >> > >> > > Use Case: >> > >> > > ---- >> > >> > > When the data has been received from input source, one can >> transform >> > >> data >> > >> > > using transform operator and then dump it to destination. >> > >> > > >> > >> > > Transform means: >> > >> > > ---- >> > >> > > 1. Conversion of fields from one type to another. Eg. int to >> > Integer, >> > >> > epoc >> > >> > > to java.util.Date object, int to String or String to int etc. >> > >> > > 2. Deriving new fields from one or more input fields. >> > >> > > For eg. Input contains 2 fields viz. "firstname", "lastname" >> and >> > >> > > derived field is "fullname" which is combination of firstname and >> > >> > lastname >> > >> > > in some way. >> > >> > > Another example is date of birth present in input tuple, >> > generate >> > >> age >> > >> > > from it. >> > >> > > 3. Change in schema from input to output. For eg. Input schema is >> > >> subset >> > >> > of >> > >> > > output schema and output schema contains some extra derived >> fields >> > >> from >> > >> > > input fields. >> > >> > > >> > >> > > >> > >> > > Functionality: >> > >> > > ---- >> > >> > > 1. Transform operator will access POJO as input tuple and emit >> > >> > another/same >> > >> > > POJO output tuple. >> > >> > > 2. Operator needs to be configured with input tuple schema and >> > output >> > >> > tuple >> > >> > > schema for this. This can be done via TUPLE_CLASS attribute on >> > ports. >> > >> > > 3. Generation of output tuple from input tuple can be specified >> > using >> > >> > > simple java based expressions. >> > >> > > For eg. >> > >> > > outfield1 = <expression using fields of input POJO> >> > >> > > outfield2 = <expression using fields of output POJO> >> > >> > > 4. If no expression is mentioned for a certain output field, >> then a >> > >> > > matching field (name and type) will be picked input POJO and >> > contents >> > >> > will >> > >> > > be copied to output POJO. >> > >> > > Otherwise, the output field will be left to default empty value. >> > >> > > >> > >> > > >> > >> > > Expression Support in PojoUtils: >> > >> > > ---- >> > >> > > 1. For achieving above functionality, I thought about extending >> > >> PojoUtils >> > >> > > which already support much of what is required. Then this utility >> > can >> > >> be >> > >> > > used in Transform operator. >> > >> > > 2. Couple of minor enhancements/fixing issues required to this >> > >> utility: >> > >> > > a. Multiple POJO support is not required. Hence will stick to >> > >> single >> > >> > > POJO. >> > >> > > b. PojoUtils does not work well with resolution of getter >> > methods >> > >> > when >> > >> > > more than one fields are mentioned in expression. Will need to >> fix >> > >> that. >> > >> > > c. Following interface can be added to PojoUtils. This will >> be >> > the >> > >> > > return method that can be called for executing the expression >> > >> > > interface Expression<O> >> > >> > > { >> > >> > > O execute(Object obj); >> > >> > > } >> > >> > > d. Following methods can be added to PojoUtils for addressing >> > >> > > expression support: >> > >> > > // Evaluates given expression >> > >> > > Expression evaluateExpression(String expr, Class >> > >> inputObjectType, >> > >> > > Class returnType); >> > >> > > >> > >> > > // Adds a custom method that can be used in an >> expression. >> > >> > > // The one who's using this library can utility this >> method >> > to >> > >> > > provide predefined functionality to user. >> > >> > > void registerCustomMethod(String qualifiedFunc) >> > >> > > >> > >> > > >> > >> > > Please share your valuable inputs on this. >> > >> > > >> > >> > > Thanks, >> > >> > > Chinmay. >> > >> > > >> > >> > >> > >> >> > > >> > > >> > >> > >
