Okay Sure. We can have a hangout On Sun, Jun 28, 2015 at 11:15 AM, Nirmal Fernando <nir...@wso2.com> wrote:
> It'll be good if we can have it before mid evaluations. If you can't make > it to Trace, we can have a hangout? > > On Sun, Jun 28, 2015 at 11:11 AM, Danula Eranjith <hmdanu...@gmail.com> > wrote: > >> It would be difficult for me to make it tomorrow. >> How about Thursday (02/07) at Trace? anytime after 11.30 am would be >> great. >> >> On Sun, Jun 28, 2015 at 10:09 AM, Nirmal Fernando <nir...@wso2.com> >> wrote: >> >>> +1 shall we have it tomorrow at Trace? >>> >>> On Sun, Jun 28, 2015 at 9:45 AM, Supun Sethunga <sup...@wso2.com> wrote: >>> >>>> Can you arrange a time around this week? Please check with Nirmal too. >>>> >>>> On Sun, Jun 28, 2015 at 9:31 AM, Danula Eranjith <hmdanu...@gmail.com> >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> No, We haven't done a review yet. >>>>> It would be great if we could have one so that I can discuss with you >>>>> all and clarify the next steps of the implementation as you mentioned. >>>>> >>>>> Thanks >>>>> Danula >>>>> >>>>> On Sun, Jun 28, 2015 at 9:25 AM, Supun Sethunga <sup...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Danula, >>>>>> >>>>>> Did we have a review for the work done so far? If not, shall we have >>>>>> a one? We can clear out any doubts and issues as well.. >>>>>> >>>>>> Thanks, >>>>>> Supun >>>>>> >>>>>> On Wed, Jun 24, 2015 at 6:42 AM, Nirmal Fernando <nir...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Danula, >>>>>>> >>>>>>> Thanks for the update, keep them coming. >>>>>>> >>>>>>> On a JavaRDD you can perform a collect() to get a list, AFAIR. Yes, >>>>>>> this is costly, since it would load whole dataset into memory. So, is >>>>>>> this >>>>>>> an operation which involves multiple rows? >>>>>>> >>>>>>> On Tue, Jun 23, 2015 at 2:15 PM, Danula Eranjith < >>>>>>> hmdanu...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi Supun, >>>>>>>> >>>>>>>> I modified the "Fill" operation to add what you mentioned. >>>>>>>> >>>>>>>> I used a workaround to to implement certain parts of the operations >>>>>>>> such as filling with values from rows above and below. >>>>>>>> I created a List Implementation using toArray() method in JavaRDD >>>>>>>> and then converted it back to a JavaRDD after the operation. >>>>>>>> >>>>>>>> This will be inefficient (in terms of both memory and time) when >>>>>>>> working with very large data sets. But I think its important to have >>>>>>>> these >>>>>>>> features included. Otherwise a user would be left with very limited >>>>>>>> set of >>>>>>>> operations. >>>>>>>> >>>>>>>> Please let me know if you have a different opinion on this. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Danula >>>>>>>> >>>>>>>> On Tue, Jun 16, 2015 at 9:44 AM, Supun Sethunga <sup...@wso2.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Somehow there are issues in implementing certain wrangler >>>>>>>>>> functions due to limitations in JavaRDD used in spark >>>>>>>>>> e.g. - >>>>>>>>>> Fill operation - when filling with values from rows above and >>>>>>>>>> below >>>>>>>>>> Fold operation >>>>>>>>> >>>>>>>>> >>>>>>>>> Agree, since rows will get executed randomly with spark, inter-row >>>>>>>>> operations are not very meaningful. >>>>>>>>> But you can slightly modify the implementation of the "Fill" >>>>>>>>> operation, such as, to fill values based on an >>>>>>>>> expression/static-value/mean >>>>>>>>> etc. (not depending on other rows).. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Supun >>>>>>>>> >>>>>>>>> On Tue, Jun 16, 2015 at 9:27 AM, Supun Sethunga <sup...@wso2.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi Danula, >>>>>>>>>> >>>>>>>>>> Sorry for the late reply. Have you got the details you were >>>>>>>>>> looking for? >>>>>>>>>> >>>>>>>>>> It would be great if I could get to know which wrangler >>>>>>>>>>> operations are important for a user of the ML >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Other than the ones you have mentioned in the proposal, think its >>>>>>>>>> better to have "Translate" operation as well (to create a new >>>>>>>>>> column based on an existing column). >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Supun >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Jun 4, 2015 at 10:11 PM, Danula Eranjith < >>>>>>>>>> hmdanu...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> I am currently working on generating spark transformations >>>>>>>>>>> related to the operations available in the data wrangler. >>>>>>>>>>> >>>>>>>>>>> Data wrangler provides sufficient parameters to re-create these >>>>>>>>>>> at spark.I have successfully implemented delete and split >>>>>>>>>>> operations of >>>>>>>>>>> wrangler in spark. >>>>>>>>>>> >>>>>>>>>>> Once this phase is completed, I can either directly generate >>>>>>>>>>> these scripts at wrangler or use the javascript output and convert >>>>>>>>>>> it to >>>>>>>>>>> spark depending on the implementation. >>>>>>>>>>> >>>>>>>>>>> Somehow there are issues in implementing certain wrangler >>>>>>>>>>> functions due to limitations in JavaRDD used in spark >>>>>>>>>>> >>>>>>>>>>> e.g. - >>>>>>>>>>> Fill operation - when filling with values from rows above and >>>>>>>>>>> below >>>>>>>>>>> Fold operation >>>>>>>>>>> >>>>>>>>>>> It would be great if I could get to know which wrangler >>>>>>>>>>> operations are important for a user of the ML >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Danula >>>>>>>>>>> >>>>>>>>>>> On Wed, Jun 3, 2015 at 8:30 AM, Nirmal Fernando <nir...@wso2.com >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Danula, >>>>>>>>>>>> >>>>>>>>>>>> Please send an update of your work thus far. >>>>>>>>>>>> >>>>>>>>>>>> On Sun, May 10, 2015 at 2:30 PM, Nirmal Fernando < >>>>>>>>>>>> nir...@wso2.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Danula, >>>>>>>>>>>>> >>>>>>>>>>>>> Welcome to GSoC 15' ! Can you do some research on directly >>>>>>>>>>>>> generating spark transformations using Wrangler and come up with >>>>>>>>>>>>> a summary ? >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, May 8, 2015 at 11:03 AM, Danula Eranjith < >>>>>>>>>>>>> hmdanu...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you for selecting my proposal [1] >>>>>>>>>>>>>> <https://docs.google.com/document/d/18NFa23CrhXqnHrkl_AuRz3sQ3Axg7SEmiA7l66Hl9_0/edit?usp=sharing> >>>>>>>>>>>>>> for GSoC 2015. I am really looking forward to work with you all >>>>>>>>>>>>>> and >>>>>>>>>>>>>> contribute to WSO2. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I have already completed my primary research on wrangler and >>>>>>>>>>>>>> would like to meet you to get feedback on the proposed >>>>>>>>>>>>>> architecture. I am >>>>>>>>>>>>>> planning to start working on the project before 25th of May. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you, >>>>>>>>>>>>>> Danula >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] - >>>>>>>>>>>>>> https://docs.google.com/document/d/18NFa23CrhXqnHrkl_AuRz3sQ3Axg7SEmiA7l66Hl9_0/edit?usp=sharing >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>>> Nirmal >>>>>>>>>>>>> >>>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>> Nirmal >>>>>>>>>>>> >>>>>>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Supun Sethunga* >>>>>>>>>> Software Engineer >>>>>>>>>> WSO2, Inc. >>>>>>>>>> http://wso2.com/ >>>>>>>>>> lean | enterprise | middleware >>>>>>>>>> Mobile : +94 716546324 >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Supun Sethunga* >>>>>>>>> Software Engineer >>>>>>>>> WSO2, Inc. >>>>>>>>> http://wso2.com/ >>>>>>>>> lean | enterprise | middleware >>>>>>>>> Mobile : +94 716546324 >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Supun Sethunga* >>>>>> Software Engineer >>>>>> WSO2, Inc. >>>>>> http://wso2.com/ >>>>>> lean | enterprise | middleware >>>>>> Mobile : +94 716546324 >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> *Supun Sethunga* >>>> Software Engineer >>>> WSO2, Inc. >>>> http://wso2.com/ >>>> lean | enterprise | middleware >>>> Mobile : +94 716546324 >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
