+1 for options 2 and 3 On Tue, Jul 12, 2016 at 12:14 PM, Kottapalli, Venkatesh < vkottapa...@directv.com> wrote:
> +1 for deprecating the packages listed below. > > -----Original Message----- > From: hsy...@gmail.com [mailto:hsy...@gmail.com] > Sent: Tuesday, July 12, 2016 12:01 PM > > +1 > > On Tue, Jul 12, 2016 at 11:53 AM, David Yan <da...@datatorrent.com> wrote: > > > Hi all, > > > > I would like to renew the discussion of retiring operators in Malhar. > > > > As stated before, the reason why we would like to retire operators in > > Malhar is because some of them were written a long time ago before > > Apache incubation, and they do not pertain to real use cases, are not > > up to par in code quality, have no potential for improvement, and > > probably completely unused by anybody. > > > > We do not want contributors to use them as a model of their > > contribution, or users to use them thinking they are of quality, and > then hit a wall. > > Both scenarios are not beneficial to the reputation of Apex. > > > > The initial 3 packages that we would like to target are *lib/algo*, > > *lib/math*, and *lib/streamquery*. > > > > I'm adding this thread to the users list. Please speak up if you are > > using any operator in these 3 packages. We would like to hear from you. > > > > These are the options I can think of for retiring those operators: > > > > 1) Completely remove them from the malhar repository. > > 2) Move them from malhar-library into a separate artifact called > > malhar-misc > > 3) Mark them deprecated and add to their javadoc that they are no > > longer supported > > > > Note that 2 and 3 are not mutually exclusive. Any thoughts? > > > > David > > > > On Tue, Jun 7, 2016 at 2:27 PM, Pramod Immaneni > > <pra...@datatorrent.com> > > wrote: > > > >> I wanted to close the loop on this discussion. In general everyone > >> seemed to be favorable to this idea with no serious objections. Folks > >> had good suggestions like documenting capabilities of operators, come > >> up well defined criteria for graduation of operators and what those > >> criteria may be and what to do with existing operators that may not > >> yet be mature or unused. > >> > >> I am going to summarize the key points that resulted from the > >> discussion and would like to proceed with them. > >> > >> - Operators that do not yet provide the key platform capabilities to > >> make an operator useful across different applications such as > >> reusability, > >> partitioning static or dynamic, idempotency, exactly once will still > be > >> accepted as long as they are functionally correct, have unit tests > >> and will > >> go into a separate module. > >> - Contrib module was suggested as a place where new contributions go > in > >> that don't yet have all the platform capabilities and are not yet > >> mature. > >> If there are no other suggestions we will go with this one. > >> - It was suggested the operators documentation list those platform > >> capabilities it currently provides from the list above. I will > >> document a > >> structure for this in the contribution guidelines. > >> - Folks wanted to know what would be the criteria to graduate an > >> operator to the big leagues :). I will kick-off a separate thread > >> for it as > >> I think it requires its own discussion and hopefully we can come > >> up with a > >> set of guidelines for it. > >> - David brought up state of some of the existing operators and their > >> retirement and the layout of operators in Malhar in general and how > it > >> causes problems with development. I will ask him to lead the > >> discussion on > >> that. > >> > >> Thanks > >> > >> On Fri, May 27, 2016 at 7:47 PM, David Yan <da...@datatorrent.com> > wrote: > >> > >> > The two ideas are not conflicting, but rather complementing. > >> > > >> > On the contrary, putting a new process for people trying to > >> > contribute while NOT addressing the old unused subpar operators in > >> > the repository > >> is > >> > what is conflicting. > >> > > >> > Keep in mind that when people try to contribute, they always look > >> > at the existing operators already in the repository as examples and > >> > likely a > >> model > >> > for their new operators. > >> > > >> > David > >> > > >> > > >> > On Fri, May 27, 2016 at 4:05 PM, Amol Kekre <a...@datatorrent.com> > >> wrote: > >> > > >> > > Yes there are two conflicting threads now. The original thread > >> > > was to > >> > open > >> > > up a way for contributors to submit code in a dir (contrib?) as > >> > > long > >> as > >> > > license part of taken care of. > >> > > > >> > > On the thread of removing non-used operators -> How do we know > >> > > what is being used? > >> > > > >> > > Thks, > >> > > Amol > >> > > > >> > > > >> > > On Fri, May 27, 2016 at 3:40 PM, Sandesh Hegde < > >> sand...@datatorrent.com> > >> > > wrote: > >> > > > >> > > > +1 for removing the not-used operators. > >> > > > > >> > > > So we are creating a process for operator writers who don't > >> > > > want to understand the platform, yet wants to contribute? How > >> > > > big is that > >> set? > >> > > > If we tell the app-user, here is the code which has not passed > >> > > > all > >> the > >> > > > checklist, will they be ready to use that in production? > >> > > > > >> > > > This thread has 2 conflicting forces, reduce the operators and > >> > > > make > >> it > >> > > easy > >> > > > to add more operators. > >> > > > > >> > > > > >> > > > > >> > > > On Fri, May 27, 2016 at 3:03 PM Pramod Immaneni < > >> > pra...@datatorrent.com> > >> > > > wrote: > >> > > > > >> > > > > On Fri, May 27, 2016 at 2:30 PM, Gaurav Gupta < > >> > > gaurav.gopi...@gmail.com> > >> > > > > wrote: > >> > > > > > >> > > > > > Pramod, > >> > > > > > > >> > > > > > By that logic I would say let's put all partitionable > >> > > > > > operators > >> > into > >> > > > one > >> > > > > > folder, non-partitionable operators in another and so on... > >> > > > > > > >> > > > > > >> > > > > Remember the original goal of making it easier for new > >> > > > > members to contribute and managing those contributions to > >> > > > > maturity. It is > >> not a > >> > > > > functional level separation. > >> > > > > > >> > > > > > >> > > > > > When I look at hadoop code I see these annotations being > >> > > > > > used at > >> > > class > >> > > > > > level and not at package/folder level. > >> > > > > > >> > > > > > >> > > > > I had a typo in my email, I meant to say "think of this like > >> > > > > a > >> > > folder..." > >> > > > > as an analogy and not literally. > >> > > > > > >> > > > > Thanks > >> > > > > > >> > > > > > >> > > > > > Thanks > >> > > > > > > >> > > > > > On Fri, May 27, 2016 at 2:10 PM, Pramod Immaneni < > >> > > > pra...@datatorrent.com > >> > > > > > > >> > > > > > wrote: > >> > > > > > > >> > > > > > > On Fri, May 27, 2016 at 1:05 PM, Gaurav Gupta < > >> > > > > gaurav.gopi...@gmail.com> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Can same goal not be achieved by using > >> > > org.apache.hadoop.classification.InterfaceStability.Evolving > >> > > > / > >> > > > > > > > org.apache.hadoop.classification.InterfaceStability.Uns > >> > > > > > > > table > >> > > > > > annotation? > >> > > > > > > > > >> > > > > > > > >> > > > > > > I think it is important to localize the additions in one > >> place so > >> > > > that > >> > > > > it > >> > > > > > > becomes clearer to users about the maturity level of > >> > > > > > > these, > >> > easier > >> > > > for > >> > > > > > > developers to track them towards the path to maturity and > >> > > > > > > also > >> > > > > provides a > >> > > > > > > clearer directive for committers and contributors on > >> acceptance > >> > of > >> > > > new > >> > > > > > > submissions. Relying on the annotations alone makes them > >> spread > >> > all > >> > > > > over > >> > > > > > > the place and adds an additional layer of difficulty in > >> > > > identification > >> > > > > > not > >> > > > > > > just for users but also for developers who want to find > >> > > > > > > such > >> > > > operators > >> > > > > > and > >> > > > > > > improve them. This of this like a folder level annotation > >> where > >> > > > > > everything > >> > > > > > > under this folder is unstable or evolving. > >> > > > > > > > >> > > > > > > Thanks > >> > > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > On Fri, May 27, 2016 at 12:35 PM, David Yan < > >> > > da...@datatorrent.com > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > Malhar in its current state, has way too many > >> operators > >> > > > that > >> > > > > > fall > >> > > > > > > > in > >> > > > > > > > > > the > >> > > > > > > > > > > > "non-production quality" category. We should > >> > > > > > > > > > > > make it > >> > > > obvious > >> > > > > to > >> > > > > > > > users > >> > > > > > > > > > > that > >> > > > > > > > > > > > which operators are up to par, and which > >> > > > > > > > > > > > operators > >> are > >> > > not, > >> > > > > and > >> > > > > > > > maybe > >> > > > > > > > > > > even > >> > > > > > > > > > > > remove those that are likely not ever used in a > >> > > > > > > > > > > > real > >> > use > >> > > > > case. > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > I am ambivalent about revisiting older operators > >> > > > > > > > > > > and > >> > doing > >> > > > this > >> > > > > > > > > exercise > >> > > > > > > > > > as > >> > > > > > > > > > > this can cause unnecessary tensions. My original > >> intent > >> > is > >> > > > for > >> > > > > > > > > > > contributions going forward. > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > IMO it is important to address this as well. > >> > > > > > > > > > Operators > >> > > outside > >> > > > > the > >> > > > > > > play > >> > > > > > > > > > area should be of well known quality. > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > I think this is important, and I don't anticipate > >> > > > > > > > > much > >> > tension > >> > > if > >> > > > > we > >> > > > > > > > > establish clear criteria. > >> > > > > > > > > It's not helpful if we let the old subpar operators > >> > > > > > > > > stay > >> and > >> > > put > >> > > > up > >> > > > > > the > >> > > > > > > > > bars for new operators. > >> > > > > > > > > > >> > > > > > > > > David > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
