+1 for deprecating operators/hooks as plugins, let's use Python's good old python packages and maybe python "entry points" if we want to inject them in "airflow.operators"/"airflow.hooks" (which is probably not necessary)
On Tue, Sep 18, 2018 at 2:12 AM Ash Berlin-Taylor <a...@apache.org> wrote: > Operators and hooks don't need any special plugin system - simply having > them as as separate Python modules which are imported using normal python > semantics is enough. > > In fact now that I think about it: I want to deprecate the plugins > registering hooks/operators etc and limit it to only bits which a simple > python import can't manage - which I think is only anything that needs to > be registered with another system, such as custom routes in the web UI. > > I'll draft an AIP for this soon. > > -ash > > > > On 18 Sep 2018, at 00:50, George Leslie-Waksman <waks...@gmail.com> > wrote: > > > > Given we have a plugin system, could we alternatively move away from > > keeping non-core supported code outside of the core project/repo? > > > > It would hugely decrease the surface area of the main repository and > > testing infrastructure to get most of the contrib code out to its own > place. > > > > Further, it would decrease the committer burden of having to > approve/merge > > code that is not supposed to be their responsibility. > > > > On Mon, Sep 17, 2018 at 4:37 PM Tim Swast <sw...@google.com.invalid> > wrote: > > > >>> Individual operators and hooks living in separate repositories on > github > >> (or possibly other Apache projects), which are then distributed by pip > and > >> installed as libraries seems like it would scale better. > >> > >> Pandas did this about a year ago, and it's seemed to have worked well. > For > >> example, pandas.read_gbq is a very thin wrapper around > pandas_gbq.read_gbq > >> (distributed as a separate package). It has made it easier for me to > track > >> issues corresponding to my area of expertise. > >> > >> On Sun, Sep 16, 2018 at 1:25 PM Jakob Homan <jgho...@gmail.com> wrote: > >> > >>>> My understanding as a contributor is that if a hook/operator is in > >> core, > >>> it > >>>> means that a committer is willing to take personal responsibility to > >>>> maintain it (or at least help maintain it), and everything else goes > in > >>>> contrib. > >>> > >>> That's not correct. All of the code is owned by the entire > >>> community[1]; no one person is responsible for any of it. There's no > >>> silos, fiefdoms, walled gardens, etc. If the community cannot support > >>> a piece of code it should be deprecated and subsequently removed. > >>> > >>> Contrib sections are almost always problematic for this reason. > >>> Hadoop ended up abandoning its. Because Airflow acts as a gathering > >>> point for so many disparate technologies (databases, storage systems, > >>> compute engines, etc.), trying to keep all of them corralled and up to > >>> date will be very difficult. Individual operators and hooks living in > >>> separate repositories on github (or possibly other Apache projects), > >>> which are then distributed by pip and installed as libraries seems > >>> like it would scale better. > >>> > >>> -Jakob > >>> > >>> [1] > https://blogs.apache.org/foundation/entry/success-at-apache-a-newbie > >>> > >>> On 15 September 2018 at 13:29, Jeff Payne <jpa...@bombora.com> wrote: > >>>> How many operators are added to contrib per month? Is it too many to > >>> make the decision case by case? If so, then the above mentioned rule > >> sounds > >>> fairly reasonable. However, if that's the rule, shouldn't a bunch of > >>> existing modules be moved from contrib to core? > >>>> > >>>> Get Outlook for Android<https://aka.ms/ghei36> > >>>> > >>>> ________________________________ > >>>> From: Taylor Edmiston <tedmis...@gmail.com> > >>>> Sent: Saturday, September 15, 2018 1:13:47 PM > >>>> To: dev@airflow.incubator.apache.org > >>>> Subject: Re: Guidelines on Contrib vs Non-contrib > >>>> > >>>> My understanding as a contributor is that if a hook/operator is in > >> core, > >>> it > >>>> means that a committer is willing to take personal responsibility to > >>>> maintain it (or at least help maintain it), and everything else goes > in > >>>> contrib. > >>>> > >>>> *Taylor Edmiston* > >>>> Blog <https://blog.tedmiston.com/> | LinkedIn > >>>> <https://www.linkedin.com/in/tedmiston/> | Stack Overflow > >>>> <https://stackoverflow.com/users/149428/taylor-edmiston> | Developer > >>> Story > >>>> <https://stackoverflow.com/story/taylor> > >>>> > >>>> > >>>> > >>>> On Sat, Sep 15, 2018 at 2:02 PM Kaxil Naik <kaxiln...@gmail.com> > >> wrote: > >>>> > >>>>> Hi, all (mainly contributors), > >>>>> > >>>>> Can we decide on a common guideline on when a hook/operator should go > >>> under > >>>>> contrib vs core? > >>>>> > >>>>> Regards, > >>>>> > >>>>> *Kaxil Naik* > >>>>> *Big Data Consultant *@ *Data Reply UK* > >>>>> *Certified *Google Cloud Data Engineer | *Certified* Apache Spark & > >>> Neo4j > >>>>> Developer > >>>>> *Phone: *+44 (0) 74820 88992 > >>>>> *LinkedIn*: https://www.linkedin.com/in/kaxil > >>>>> > >>> > >> -- > >> * • **Tim Swast* > >> * • *Software Friendliness Engineer > >> * • *Google Cloud Developer Relations > >> * • *Seattle, WA, USA > >> > >
