Re: A proposal for Malhar

Thomas thanks for the suggestions and the comments in the document. I will take another look at the ones that I had shortlisted in the document to keep. Within that subset, would it be ok to leave the ones that don't have a large state problem, for the time being, till we have replacement

Re: A proposal for Malhar

There are a bunch of operators that don't have proper state management and also don't support generic windowing (event time etc.). I would suggest to move those out or deprecate them. The new windowing and state management support along with the appropriate aggregators is going to make them

Re: A proposal for Malhar

Hi, Friendly Reminder : I created a shared google sheet and tracked the various details of operators. The sheet contains information about operators under lib/algo, lib/math & lib/streamquery. Link is https://docs.google.com/a/ datatorrent.com/spreadsheets/d/15IjMa-vYK6Wru4kZnUGIgg6sQAptDJ_

Re: A proposal for Malhar

Added comments, also recommend having the misc folder for the remaining operators in contrib according to proposed guidelines https://github.com/apache/apex-site/pull/44 On Mon, Aug 1, 2016 at 12:22 AM, Lakshmi Velineni wrote: > Hi > > I also added recommendation for

Re: A proposal for Malhar

Hi, I also added recommendation for each operator . Please take a look. thanks On Thu, Jul 28, 2016 at 11:03 AM, Lakshmi Velineni wrote: > Hi, > > I created a shared google sheet and tracked the various details of > operators. Currently, the sheet contains information

Re: A proposal for Malhar

A document for malhar contribution guidelines has been prepared and submitted in a pull request https://github.com/apache/apex-site/pull/44 Thanks On Tue, Jun 7, 2016 at 2:27 PM, Pramod Immaneni wrote: > I wanted to close the loop on this discussion. In general

Re: A proposal for Malhar

+1. This is a really good starting point to cleanup malhar. On Wed, Jul 13, 2016 at 3:06 AM, David Yan wrote: > Hi Lakshmi, > > Thanks for volunteering. > > I think Pramod's suggestion of putting the operators into 3 buckets and > Siyuan's suggestion of starting a shared

Re: A proposal for Malhar

Hi Lakshmi, Thanks for volunteering. I think Pramod's suggestion of putting the operators into 3 buckets and Siyuan's suggestion of starting a shared Google Sheet that tracks individual operators are both good, with the exception that lib/streamquery is one unit and we probably do not need to

Re: A proposal for Malhar

I am interested to work on this. Regards, Lakshmi prasanna On Tue, Jul 12, 2016 at 1:55 PM, hsy...@gmail.com wrote: > Why not have a shared google sheet with a list of operators and options > that we want to do with it. > I think it's case by case. > But retire unused or

Re: A proposal for Malhar

Why not have a shared google sheet with a list of operators and options that we want to do with it. I think it's case by case. But retire unused or obsolete operators is important and we should do it sooner rather than later. Regards, Siyuan On Tue, Jul 12, 2016 at 1:09 PM, Amol Kekre

Re: A proposal for Malhar

I would suggest we go through the operators in those packages on an individual basis and grade them into 3 buckets, those that meet the level we expect from the operators (could be few of them), those that are potentially useful but need additional work and those that we don't think would be

Re: A proposal for Malhar

My vote is to do 2&3 Thks Amol On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh < vkottapa...@directv.com> wrote: > +1 for deprecating the packages listed below. > > -Original Message- > From: hsy...@gmail.com [mailto:hsy...@gmail.com] > Sent: Tuesday, July 12, 2016 12:01 PM > >

Re: A proposal for Malhar

+1 for options 2 and 3 On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh < vkottapa...@directv.com> wrote: > +1 for deprecating the packages listed below. > > -Original Message- > From: hsy...@gmail.com [mailto:hsy...@gmail.com] > Sent: Tuesday, July 12, 2016 12:01 PM > > +1 > > On

RE: A proposal for Malhar

+1 for deprecating the packages listed below. -Original Message- From: hsy...@gmail.com [mailto:hsy...@gmail.com] Sent: Tuesday, July 12, 2016 12:01 PM +1 On Tue, Jul 12, 2016 at 11:53 AM, David Yan wrote: > Hi all, > > I would like to renew the discussion of

Re: A proposal for Malhar

+1 On Tue, Jul 12, 2016 at 11:53 AM, David Yan wrote: > Hi all, > > I would like to renew the discussion of retiring operators in Malhar. > > As stated before, the reason why we would like to retire operators in > Malhar is because some of them were written a long time

Re: A proposal for Malhar

I wanted to close the loop on this discussion. In general everyone seemed to be favorable to this idea with no serious objections. Folks had good suggestions like documenting capabilities of operators, come up well defined criteria for graduation of operators and what those criteria may be and

Re: A proposal for Malhar

The two ideas are not conflicting, but rather complementing. On the contrary, putting a new process for people trying to contribute while NOT addressing the old unused subpar operators in the repository is what is conflicting. Keep in mind that when people try to contribute, they always look at

Re: A proposal for Malhar

Instead of creating a new space for evolving operators, why not create a new space for mature operators and graduate existing mature operators? That way, it solves both the items of discussion - having separation between evolving and mature operators. - not having evolving operators in mature

Re: A proposal for Malhar

It is important that the discussion happens, regardless in which thread. If we want to establish criteria for operators to be promoted out of contrib then we need to make sure those already out there match the bar. Today there are many operators that don't. Also, the incubator space needs to be

Re: A proposal for Malhar

Agreed Thks Amol On Fri, May 27, 2016 at 4:14 PM, Pramod Immaneni wrote: > Amol, > > I would suggest starting a separate thread for that discussion. > > Thanks > > On Fri, May 27, 2016 at 4:05 PM, Amol Kekre wrote: > > > Yes there are two

Re: A proposal for Malhar

On Fri, May 27, 2016 at 3:40 PM, Sandesh Hegde wrote: > +1 for removing the not-used operators. > > So we are creating a process for operator writers who don't want to > understand the platform, yet wants to contribute? How big is that set? > If we tell the app-user,

Re: A proposal for Malhar

+1 for removing the not-used operators. So we are creating a process for operator writers who don't want to understand the platform, yet wants to contribute? How big is that set? If we tell the app-user, here is the code which has not passed all the checklist, will they be ready to use that in

Re: A proposal for Malhar

On Fri, May 27, 2016 at 2:30 PM, Gaurav Gupta wrote: > Pramod, > > By that logic I would say let's put all partitionable operators into one > folder, non-partitionable operators in another and so on... > Remember the original goal of making it easier for new members to

Re: A proposal for Malhar

A separate directory conveys "new code; not cleared all checklisted items" much better and stronger than evolving within the same dir. There are examples like PiggyBank that achieved similar results. As Hitesh pointed out, licensing compliance will need to be checked. Thks, Amol On Fri, May 27,

Re: A proposal for Malhar

Pramod, By that logic I would say let's put all partitionable operators into one folder, non-partitionable operators in another and so on... When I look at hadoop code I see these annotations being used at class level and not at package/folder level. Thanks On Fri, May 27, 2016 at 2:10 PM,

Re: A proposal for Malhar

I am in favor of contrib too as long as it did not imply "license incompatible" issue in other Apache projects. I found the following https://mail-archives.apache.org/mod_mbox/pig-user/201205.mbox/%3CCAB-acjMTZUy=8skp+aqdm4rj1ordbxynvjnmsfymjsdfdvp...@mail.gmail.com%3E

Re: A proposal for Malhar

On Fri, May 27, 2016 at 1:05 PM, Gaurav Gupta wrote: > Can same goal not be achieved by > using org.apache.hadoop.classification.InterfaceStability.Evolving / > org.apache.hadoop.classification.InterfaceStability.Unstable annotation? > I think it is important to

Re: A proposal for Malhar

> > > > This checklist would be defined by the community and would track the > > > > current development state of the operator. That way there are no > > > unexpected > > > > surprises. > > > > > > > > > > > > > > &g

Re: A proposal for Malhar

I think you misunderstood what I said. For operators that will never be used and have no potential to improve, my opinion is to remove them completely. I am not against using the annotations in place of the incubator space. David On Fri, May 27, 2016 at 1:37 PM, Gaurav Gupta

Re: A proposal for Malhar

David, Why do you want to have operators is new incubator space that will never be used? My question what is this new incubator space going to provide that can't be achieved by annotations? Thanks Gaurav On Fri, May 27, 2016 at 1:20 PM, David Yan wrote: > Yes, we can

Re: A proposal for Malhar

> > > > > > > > > Malhar in its current state, has way too many operators that fall in > the > > > "non-production quality" category. We should make it obvious to users > > that > > > which operators are up to par, and which operators are not, and maybe > > even > > > remove those that are likely

Re: A proposal for Malhar

On Fri, May 27, 2016 at 10:59 AM, Priyanka Gugale wrote: > I like the idea. I am just worried about left with unreadable code as time > passes. As few people already pointed out, I would suggest there should be > kind of checklist which they should mark as

Re: A proposal for Malhar

On Fri, May 27, 2016 at 10:31 AM, Thomas Weise wrote: > > > > > > > > Malhar in its current state, has way too many operators that fall in > the > > > "non-production quality" category. We should make it obvious to users > > that > > > which operators are up to par, and

Re: A proposal for Malhar

Comments inline. On Fri, May 27, 2016 at 7:13 AM, Amol Kekre wrote: > This is a very good idea and will greatly help contributors. The > requirements to submit code to this Malhar folder should be very minimal. A > few that come to my mind > Thanks > > - Should compile

Re: A proposal for Malhar

My comment inline On Fri, May 27, 2016 at 9:00 AM, David Yan wrote: > This is an important change because not only will it help contributors who > want to contribute to Apex Malhar, it also helps users on deciding which > operators they can use. > Thanks > > Malhar in

Re: A proposal for Malhar

the class. > > > > > > This checklist would be defined by the community and would track the > > > current development state of the operator. That way there are no > > unexpected > > > surprises. > > > > > > > > > > > > Sent

Re: A proposal for Malhar

t; > Sent with Good (www.good.com) > > From: Amol Kekre <a...@datatorrent.com> > Sent: Friday, May 27, 2016 10:13:06 AM > To: dev@apex.apache.org > Cc: d...@apex.incubator.apache.org > Subject: Re: A proposal for Malhar > > This is a very good idea and will greatly h

RE: A proposal for Malhar

surprises. Sent with Good (www.good.com) From: Amol Kekre <a...@datatorrent.com> Sent: Friday, May 27, 2016 10:13:06 AM To: dev@apex.apache.org Cc: d...@apex.incubator.apache.org Subject: Re: A proposal for Malhar This is a very good idea and will greatl

Re: A proposal for Malhar

That's a good proposal. Sounds like an "incubator" space inside Malhar. Originally that was the intention behind the contrib module. But now it contains many connectors that should be promoted to their own modules. So a possible route would be to use contrib for early/evolving code and then

Re: A proposal for Malhar

This is a very good idea and will greatly help contributors. The requirements to submit code to this Malhar folder should be very minimal. A few that come to my mind - Should compile - License of the external lib (if any) should be Apache compliant license // Need to see if this is part of ASF