I will send an invite for next week with Sanjiva and others
On Wed, Aug 14, 2013 at 3:05 PM, Malith Dhanushka <mal...@wso2.com> wrote: > Hi Srinath, > > This has been reviewed with Anjana by the time we were finalizing BAM > 2.4.0 M1. Shall I arrange a review with you? > > Thanks, > Malith > > > > > On Wed, Aug 14, 2013 at 2:33 PM, Srinath Perera <srin...@wso2.com> wrote: > >> Have we reviewed this? >> >> >> On Wed, Aug 14, 2013 at 1:14 PM, Malith Dhanushka <mal...@wso2.com>wrote: >> >>> Hi Maninda, >>> >>> Of course, we can add them as set of built-in analyzers and it will >>> improve the scope of Hive scripts. >>> >>> Thanks, >>> Malith >>> >>> >>> On Wed, Aug 14, 2013 at 11:55 AM, Maninda Edirisooriya <mani...@wso2.com >>> > wrote: >>> >>>> Hi Malith, >>>> >>>> Nice work. Using this new feature we can implement the requirements I >>>> mentioned in the Architecture mail titled "Making BAM more useful platform >>>> with well defined Class Analyzers / UDFs". WDYT? >>>> >>>> * >>>> Maninda Edirisooriya* >>>> Software Engineer >>>> *WSO2, Inc. >>>> *lean.enterprise.middleware. >>>> >>>> *Blog* : http://maninda.blogspot.com/ >>>> *Phone* : +94 777603226 >>>> >>>> >>>> On Tue, Aug 13, 2013 at 8:24 PM, Malith Dhanushka <mal...@wso2.com>wrote: >>>> >>>>> Hi all, >>>>> >>>>> I have modified the implementation according to above description and >>>>> following is the modified version, >>>>> >>>>> - analyzer-config.xml contains the mapping details. >>>>> >>>>> ex : - >>>>> >>>>> analyzer-config.xml >>>>> >>>>> *<analyzerConfig xmlns="http://wso2.org/carbon/analytics";>* >>>>> * <analyzers>* >>>>> * <analyzer>* >>>>> * <name>foo</name>* >>>>> * >>>>> <class>org.wso2.carbon.analytics.hive.extension.builtin.FooAnalyzer</class> >>>>> * >>>>> * <parameters>bar,bat1,*</parameters>* >>>>> * </analyzer>* >>>>> * </analyzers>* >>>>> *</analyzerConfig>* >>>>> >>>>> parameter description, >>>>> >>>>> *name *- alias name which maps to the class analyzer >>>>> >>>>> *class* - class analyzer >>>>> >>>>> *parameters* - parameters that are accepted by class analyzer >>>>> >>>>> - This can be utilized in hive script as follows, >>>>> >>>>> syntax >>>>> >>>>> *analyzer foo(bar="value",bar1="value1",*)*; >>>>> >>>>> Currently there is one built-in analyzer which is resolvePath analyzer >>>>> and more will be added by considering other common use cases. >>>>> >>>>> Thanks, >>>>> Malith >>>>> >>>>> >>>>> On Mon, Aug 5, 2013 at 11:36 AM, Anjana Fernando <anj...@wso2.com>wrote: >>>>> >>>>>> On Mon, Aug 5, 2013 at 11:30 AM, Maninda Edirisooriya < >>>>>> mani...@wso2.com> wrote: >>>>>> >>>>>>> On Mon, Aug 5, 2013 at 7:58 AM, Malith Dhanushka <mal...@wso2.com>wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Implementation of the above suggested approach is in the final >>>>>>>> stage. But I had a minor clarification of the implementation with >>>>>>>> Anjana. >>>>>>>> There we came across following drawbacks of the implemented approach, >>>>>>>> >>>>>>>> - Annotation should not be coupled with a class analyzer, rather it >>>>>>>> should be a run time property injector to hive scripts. >>>>>>>> >>>>>>> Then won't the annotation feature will be bounded to Hive language? >>>>>>> If so we will not be able to integrate more languages to annotations in >>>>>>> future. >>>>>>> >>>>>> >>>>>> It won't be .. the current implementation happens to only support >>>>>> Hive at the moment, since that's what we have now. Not coupling with >>>>>> class >>>>>> analyzer itself is a sign of that, because a class analyzer is anyway a >>>>>> Hive functionality we have. >>>>>> >>>>>> Cheers, >>>>>> Anjana. >>>>>> >>>>>> >>>>>>> >>>>>>>> - Abstract Annotation class adds unnecessary complications for user >>>>>>>> to write a custom annotation. >>>>>>>> >>>>>>>> So by considering above information I am going to modify the >>>>>>>> current implementation by adhering to those factors. If there any other >>>>>>>> concerns and suggestions please feel free to add. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Malith >>>>>>>> >>>>>>>> >>>>>>>> On Wed, Jul 10, 2013 at 1:26 PM, Malith Dhanushka >>>>>>>> <mal...@wso2.com>wrote: >>>>>>>> >>>>>>>>> Hi Maninda, >>>>>>>>> >>>>>>>>> On Wed, Jul 10, 2013 at 11:37 AM, Maninda Edirisooriya < >>>>>>>>> mani...@wso2.com> wrote: >>>>>>>>> >>>>>>>>>> This is nice. Can we use these annotations to unify the >>>>>>>>>> languages, Hive and Siddhi? If we can use this annotation framework >>>>>>>>>> as a >>>>>>>>>> platform to support different languages it will be very useful when >>>>>>>>>> it >>>>>>>>>> comes to integrating other Hadoop related languages like Mahout and >>>>>>>>>> Pig. >>>>>>>>>> That means we can separate each language relates script using >>>>>>>>>> annotations. >>>>>>>>>> This will solve the problem of unifying all the languages into a >>>>>>>>>> single >>>>>>>>>> language. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Interesting suggestion and yes ,this is gettable via annotations. >>>>>>>>> But the only limitation is that the current script implementation is >>>>>>>>> only >>>>>>>>> available for hive. So in order to achieve language unification via >>>>>>>>> annotations firstly we need to have a unified script implementation >>>>>>>>> for >>>>>>>>> each underlying engine (ie- Siddhi, Mahout, pig). >>>>>>>>> >>>>>>>>>> >>>>>>>>>> And also using this annotation framework we can create a generic >>>>>>>>>> Process Flow model on the data. For example we can execute several >>>>>>>>>> Hive >>>>>>>>>> scripts in parallel using a annotation block. And a barrier can be >>>>>>>>>> introduced if all the parallel scripts should be finished before we >>>>>>>>>> move >>>>>>>>>> onto the next script and so on. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Yes, this can be added as a built in annotation. >>>>>>>>> >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Other than that we can provide a default set of class analysers >>>>>>>>>> as we have discussed in a previous mail. The value of annotations is >>>>>>>>>> that >>>>>>>>>> we can provide the available set of class analysers out of the box. >>>>>>>>>> Any >>>>>>>>>> idea about the syntax? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Each annotation is associated with a particular class analyzer, >>>>>>>>> which process the given parameters via the annotation. So we can wrap >>>>>>>>> that >>>>>>>>> default set of class analyzers and expose them as set of built in >>>>>>>>> annotations and can stick to the same syntax as previous, >>>>>>>>> >>>>>>>>> @script.foo(bar="value", bar1="value1",*) >>>>>>>>> >>>>>>>>> >>>>>>>>>> * >>>>>>>>>> Maninda Edirisooriya* >>>>>>>>>> Software Engineer >>>>>>>>>> *WSO2, Inc. >>>>>>>>>> *lean.enterprise.middleware. >>>>>>>>>> >>>>>>>>>> *Blog* : http://maninda.blogspot.com/ >>>>>>>>>> *Phone* : +94 777603226 >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tue, Jul 9, 2013 at 11:23 AM, Malith Dhanushka < >>>>>>>>>> mal...@wso2.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I have started implementing the $Subject. The idea of having an >>>>>>>>>>> annotation facility is to carryout some pre-processing of Hive >>>>>>>>>>> queries >>>>>>>>>>> before they are being passed to the Hive engine. Currently we >>>>>>>>>>> already have >>>>>>>>>>> a "class analyzer" which can be used execute some custom logic as a >>>>>>>>>>> part of >>>>>>>>>>> a Hive script. But the main use case of annotations is to inject >>>>>>>>>>> run-time >>>>>>>>>>> properties to Hive execution context before the actual queries are >>>>>>>>>>> carried >>>>>>>>>>> out by Hive. The annotation facility would be building upon this by >>>>>>>>>>> having >>>>>>>>>>> set of such common analyzers which can manipulate the Hive queries >>>>>>>>>>> or Hive >>>>>>>>>>> execution context which it is passed to Hive query engine. >>>>>>>>>>> >>>>>>>>>>> Annotation Syntax, >>>>>>>>>>> >>>>>>>>>>> *@script.foo(bar="value", bar1="value1",*)* >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Annotation scheme will be externalized by giving *abstract >>>>>>>>>>> implementation of annotation* and *annotation-config.xml* file >>>>>>>>>>> to provide the annotation configuration which allows third party >>>>>>>>>>> annotations to be included to the system. >>>>>>>>>>> >>>>>>>>>>> *annotation-config.xml* >>>>>>>>>>> >>>>>>>>>>> <annotation> >>>>>>>>>>> <name>foo</name> >>>>>>>>>>> >>>>>>>>>>> <class>org.wso2.carbon.analytics.hive.extension.annotation.foo</class> >>>>>>>>>>> <analyzer>org.wso2.carbon.analytics.hive.extension.foo</analyzer> >>>>>>>>>>> </annotation> >>>>>>>>>>> >>>>>>>>>>> <annotation> >>>>>>>>>>> ................................ >>>>>>>>>>> </annotation> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Potential use case for this in incremental data processing where >>>>>>>>>>> any query associated with "*@script.incremental(foo="value1", >>>>>>>>>>> bar="value2",*)*" would flag and setup the properties those are >>>>>>>>>>> required to present in order for that particular query to be >>>>>>>>>>> executed in an >>>>>>>>>>> incremental manner.There can be many other useful additions as well. >>>>>>>>>>> >>>>>>>>>>> Any suggestions, thoughts are welcome. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Malith Dhanushka >>>>>>>>>>> >>>>>>>>>>> Engineer - Data Technologies >>>>>>>>>>> *WSO2, Inc. : wso2.com* >>>>>>>>>>> >>>>>>>>>>> *Mobile* : +94 716 506 693 >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Architecture mailing list >>>>>>>>>>> Architecture@wso2.org >>>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Architecture mailing list >>>>>>>>>> Architecture@wso2.org >>>>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Malith Dhanushka >>>>>>>>> >>>>>>>>> Engineer - Data Technologies >>>>>>>>> *WSO2, Inc. : wso2.com* >>>>>>>>> >>>>>>>>> *Mobile* : +94 716 506 693 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Malith Dhanushka >>>>>>>> >>>>>>>> Engineer - Data Technologies >>>>>>>> *WSO2, Inc. : wso2.com* >>>>>>>> >>>>>>>> *Mobile* : +94 716 506 693 >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> Architecture mailing list >>>>>>>> Architecture@wso2.org >>>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Architecture mailing list >>>>>>> Architecture@wso2.org >>>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Anjana Fernando* >>>>>> Technical Lead >>>>>> WSO2 Inc. | http://wso2.com >>>>>> lean . enterprise . middleware >>>>>> >>>>>> _______________________________________________ >>>>>> Architecture mailing list >>>>>> Architecture@wso2.org >>>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Malith Dhanushka >>>>> >>>>> Engineer - Data Technologies >>>>> *WSO2, Inc. : wso2.com* >>>>> >>>>> *Mobile* : +94 716 506 693 >>>>> >>>>> _______________________________________________ >>>>> Architecture mailing list >>>>> Architecture@wso2.org >>>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Architecture mailing list >>>> Architecture@wso2.org >>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>>> >>>> >>> >>> >>> -- >>> Malith Dhanushka >>> >>> Engineer - Data Technologies >>> *WSO2, Inc. : wso2.com* >>> >>> *Mobile* : +94 716 506 693 >>> >>> _______________________________________________ >>> Architecture mailing list >>> Architecture@wso2.org >>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >>> >>> >> >> >> -- >> ============================ >> Srinath Perera, Ph.D. >> http://people.apache.org/~hemapani/ >> http://srinathsview.blogspot.com/ >> >> _______________________________________________ >> Architecture mailing list >> Architecture@wso2.org >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Malith Dhanushka > > Engineer - Data Technologies > *WSO2, Inc. : wso2.com* > > *Mobile* : +94 716 506 693 > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- ============================ Srinath Perera, Ph.D. Director, Research, WSO2 Inc. Visiting Faculty, University of Moratuwa Member, Apache Software Foundation Research Scientist, Lanka Software Foundation Blog: http://srinathsview.blogspot.com/ Photos: http://www.flickr.com/photos/hemapani/ Phone: 0772360902
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
