Why not have a shared google sheet with a list of operators and options that we want to do with it. I think it's case by case. But retire unused or obsolete operators is important and we should do it sooner rather than later.
Regards, Siyuan On Tue, Jul 12, 2016 at 1:09 PM, Amol Kekre <a...@datatorrent.com> wrote: > > My vote is to do 2&3 > > Thks > Amol > > > On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh < > vkottapa...@directv.com> wrote: > >> +1 for deprecating the packages listed below. >> >> -----Original Message----- >> From: hsy...@gmail.com [mailto:hsy...@gmail.com] >> Sent: Tuesday, July 12, 2016 12:01 PM >> >> +1 >> >> On Tue, Jul 12, 2016 at 11:53 AM, David Yan <da...@datatorrent.com> >> wrote: >> >> > Hi all, >> > >> > I would like to renew the discussion of retiring operators in Malhar. >> > >> > As stated before, the reason why we would like to retire operators in >> > Malhar is because some of them were written a long time ago before >> > Apache incubation, and they do not pertain to real use cases, are not >> > up to par in code quality, have no potential for improvement, and >> > probably completely unused by anybody. >> > >> > We do not want contributors to use them as a model of their >> > contribution, or users to use them thinking they are of quality, and >> then hit a wall. >> > Both scenarios are not beneficial to the reputation of Apex. >> > >> > The initial 3 packages that we would like to target are *lib/algo*, >> > *lib/math*, and *lib/streamquery*. >> >> > >> > I'm adding this thread to the users list. Please speak up if you are >> > using any operator in these 3 packages. We would like to hear from you. >> > >> > These are the options I can think of for retiring those operators: >> > >> > 1) Completely remove them from the malhar repository. >> > 2) Move them from malhar-library into a separate artifact called >> > malhar-misc >> > 3) Mark them deprecated and add to their javadoc that they are no >> > longer supported >> > >> > Note that 2 and 3 are not mutually exclusive. Any thoughts? >> > >> > David >> > >> > On Tue, Jun 7, 2016 at 2:27 PM, Pramod Immaneni >> > <pra...@datatorrent.com> >> > wrote: >> > >> >> I wanted to close the loop on this discussion. In general everyone >> >> seemed to be favorable to this idea with no serious objections. Folks >> >> had good suggestions like documenting capabilities of operators, come >> >> up well defined criteria for graduation of operators and what those >> >> criteria may be and what to do with existing operators that may not >> >> yet be mature or unused. >> >> >> >> I am going to summarize the key points that resulted from the >> >> discussion and would like to proceed with them. >> >> >> >> - Operators that do not yet provide the key platform capabilities to >> >> make an operator useful across different applications such as >> >> reusability, >> >> partitioning static or dynamic, idempotency, exactly once will >> still be >> >> accepted as long as they are functionally correct, have unit tests >> >> and will >> >> go into a separate module. >> >> - Contrib module was suggested as a place where new contributions >> go in >> >> that don't yet have all the platform capabilities and are not yet >> >> mature. >> >> If there are no other suggestions we will go with this one. >> >> - It was suggested the operators documentation list those platform >> >> capabilities it currently provides from the list above. I will >> >> document a >> >> structure for this in the contribution guidelines. >> >> - Folks wanted to know what would be the criteria to graduate an >> >> operator to the big leagues :). I will kick-off a separate thread >> >> for it as >> >> I think it requires its own discussion and hopefully we can come >> >> up with a >> >> set of guidelines for it. >> >> - David brought up state of some of the existing operators and their >> >> retirement and the layout of operators in Malhar in general and how >> it >> >> causes problems with development. I will ask him to lead the >> >> discussion on >> >> that. >> >> >> >> Thanks >> >> >> >> On Fri, May 27, 2016 at 7:47 PM, David Yan <da...@datatorrent.com> >> wrote: >> >> >> >> > The two ideas are not conflicting, but rather complementing. >> >> > >> >> > On the contrary, putting a new process for people trying to >> >> > contribute while NOT addressing the old unused subpar operators in >> >> > the repository >> >> is >> >> > what is conflicting. >> >> > >> >> > Keep in mind that when people try to contribute, they always look >> >> > at the existing operators already in the repository as examples and >> >> > likely a >> >> model >> >> > for their new operators. >> >> > >> >> > David >> >> > >> >> > >> >> > On Fri, May 27, 2016 at 4:05 PM, Amol Kekre <a...@datatorrent.com> >> >> wrote: >> >> > >> >> > > Yes there are two conflicting threads now. The original thread >> >> > > was to >> >> > open >> >> > > up a way for contributors to submit code in a dir (contrib?) as >> >> > > long >> >> as >> >> > > license part of taken care of. >> >> > > >> >> > > On the thread of removing non-used operators -> How do we know >> >> > > what is being used? >> >> > > >> >> > > Thks, >> >> > > Amol >> >> > > >> >> > > >> >> > > On Fri, May 27, 2016 at 3:40 PM, Sandesh Hegde < >> >> sand...@datatorrent.com> >> >> > > wrote: >> >> > > >> >> > > > +1 for removing the not-used operators. >> >> > > > >> >> > > > So we are creating a process for operator writers who don't >> >> > > > want to understand the platform, yet wants to contribute? How >> >> > > > big is that >> >> set? >> >> > > > If we tell the app-user, here is the code which has not passed >> >> > > > all >> >> the >> >> > > > checklist, will they be ready to use that in production? >> >> > > > >> >> > > > This thread has 2 conflicting forces, reduce the operators and >> >> > > > make >> >> it >> >> > > easy >> >> > > > to add more operators. >> >> > > > >> >> > > > >> >> > > > >> >> > > > On Fri, May 27, 2016 at 3:03 PM Pramod Immaneni < >> >> > pra...@datatorrent.com> >> >> > > > wrote: >> >> > > > >> >> > > > > On Fri, May 27, 2016 at 2:30 PM, Gaurav Gupta < >> >> > > gaurav.gopi...@gmail.com> >> >> > > > > wrote: >> >> > > > > >> >> > > > > > Pramod, >> >> > > > > > >> >> > > > > > By that logic I would say let's put all partitionable >> >> > > > > > operators >> >> > into >> >> > > > one >> >> > > > > > folder, non-partitionable operators in another and so on... >> >> > > > > > >> >> > > > > >> >> > > > > Remember the original goal of making it easier for new >> >> > > > > members to contribute and managing those contributions to >> >> > > > > maturity. It is >> >> not a >> >> > > > > functional level separation. >> >> > > > > >> >> > > > > >> >> > > > > > When I look at hadoop code I see these annotations being >> >> > > > > > used at >> >> > > class >> >> > > > > > level and not at package/folder level. >> >> > > > > >> >> > > > > >> >> > > > > I had a typo in my email, I meant to say "think of this like >> >> > > > > a >> >> > > folder..." >> >> > > > > as an analogy and not literally. >> >> > > > > >> >> > > > > Thanks >> >> > > > > >> >> > > > > >> >> > > > > > Thanks >> >> > > > > > >> >> > > > > > On Fri, May 27, 2016 at 2:10 PM, Pramod Immaneni < >> >> > > > pra...@datatorrent.com >> >> > > > > > >> >> > > > > > wrote: >> >> > > > > > >> >> > > > > > > On Fri, May 27, 2016 at 1:05 PM, Gaurav Gupta < >> >> > > > > gaurav.gopi...@gmail.com> >> >> > > > > > > wrote: >> >> > > > > > > >> >> > > > > > > > Can same goal not be achieved by using >> >> > > org.apache.hadoop.classification.InterfaceStability.Evolving >> >> > > > / >> >> > > > > > > > org.apache.hadoop.classification.InterfaceStability.Uns >> >> > > > > > > > table >> >> > > > > > annotation? >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > > I think it is important to localize the additions in one >> >> place so >> >> > > > that >> >> > > > > it >> >> > > > > > > becomes clearer to users about the maturity level of >> >> > > > > > > these, >> >> > easier >> >> > > > for >> >> > > > > > > developers to track them towards the path to maturity and >> >> > > > > > > also >> >> > > > > provides a >> >> > > > > > > clearer directive for committers and contributors on >> >> acceptance >> >> > of >> >> > > > new >> >> > > > > > > submissions. Relying on the annotations alone makes them >> >> spread >> >> > all >> >> > > > > over >> >> > > > > > > the place and adds an additional layer of difficulty in >> >> > > > identification >> >> > > > > > not >> >> > > > > > > just for users but also for developers who want to find >> >> > > > > > > such >> >> > > > operators >> >> > > > > > and >> >> > > > > > > improve them. This of this like a folder level annotation >> >> where >> >> > > > > > everything >> >> > > > > > > under this folder is unstable or evolving. >> >> > > > > > > >> >> > > > > > > Thanks >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > > >> >> > > > > > > > On Fri, May 27, 2016 at 12:35 PM, David Yan < >> >> > > da...@datatorrent.com >> >> > > > > >> >> > > > > > > wrote: >> >> > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > > >> >> > > > > > > > > > > > Malhar in its current state, has way too many >> >> operators >> >> > > > that >> >> > > > > > fall >> >> > > > > > > > in >> >> > > > > > > > > > the >> >> > > > > > > > > > > > "non-production quality" category. We should >> >> > > > > > > > > > > > make it >> >> > > > obvious >> >> > > > > to >> >> > > > > > > > users >> >> > > > > > > > > > > that >> >> > > > > > > > > > > > which operators are up to par, and which >> >> > > > > > > > > > > > operators >> >> are >> >> > > not, >> >> > > > > and >> >> > > > > > > > maybe >> >> > > > > > > > > > > even >> >> > > > > > > > > > > > remove those that are likely not ever used in a >> >> > > > > > > > > > > > real >> >> > use >> >> > > > > case. >> >> > > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > > I am ambivalent about revisiting older operators >> >> > > > > > > > > > > and >> >> > doing >> >> > > > this >> >> > > > > > > > > exercise >> >> > > > > > > > > > as >> >> > > > > > > > > > > this can cause unnecessary tensions. My original >> >> intent >> >> > is >> >> > > > for >> >> > > > > > > > > > > contributions going forward. >> >> > > > > > > > > > > >> >> > > > > > > > > > > >> >> > > > > > > > > > IMO it is important to address this as well. >> >> > > > > > > > > > Operators >> >> > > outside >> >> > > > > the >> >> > > > > > > play >> >> > > > > > > > > > area should be of well known quality. >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > I think this is important, and I don't anticipate >> >> > > > > > > > > much >> >> > tension >> >> > > if >> >> > > > > we >> >> > > > > > > > > establish clear criteria. >> >> > > > > > > > > It's not helpful if we let the old subpar operators >> >> > > > > > > > > stay >> >> and >> >> > > put >> >> > > > up >> >> > > > > > the >> >> > > > > > > > > bars for new operators. >> >> > > > > > > > > >> >> > > > > > > > > David >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> > >> > >
