There is something to be said for the ease of putting plugins in the airflow home folder too that I would still like to keep, as it makes deploying site specific plugins much easier.
On 14 April 2019 04:13:52 BST, Jarek Potiuk <jarek.pot...@polidea.com> wrote: >+1 for using more entrypoints as discovery mechanism. Maybe we should >also >start to use the 'console_scripts' entrypoint group for "airflow" >binary >instead of copying the binary airflow script ( >https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point >) as well as for plugginable extensions. > >J. > >On Sat, Apr 13, 2019 at 11:15 PM Maxime Beauchemin < >maximebeauche...@gmail.com> wrote: > >> I'd say it could be great to deprecate the whole plugin system and >use >> Python's "entry points" instead. I just didn't know that was an >option and >> the standard way to do this when I originally wrote it... The current >> plugin system is a minefield for circular dependencies... >> >> On Sat, Apr 13, 2019 at 10:50 AM Jarek Potiuk ><jarek.pot...@polidea.com> >> wrote: >> >> > One comment here - The current connections.py file screams to get >dynamic >> > discovery similar to what sqlalchemy does for dialects (for >example). >> > >> > But on the other hand, I am not sure if this "registration" is >really >> > needed. Maybe I am totally wrong, but somehow I don't find it >> particularly >> > useful to get hook by connection_type. >> > >> > I believe connection_type is useful for the UI to be able to select >the >> > connection when you create it and have different UI fields defined >by >> > connection. But nearly universally in other places, when operator >is >> > instantiated, it also instantiates the hook it expects to use, >rather >> than >> > rely on get_hook() or parse_from_uri() of the connection. Each >operator >> > knows what connection type it needs. >> > >> > I checked it quickly and it seems the "get by connection type" or >> > parse_from_uri functionality is almost exclusively used by tests. >> > >> > Again - maybe I am completely wrong but maybe we should rethink the >whole >> > "connection" class and get rid of the need of "registering" new >> connection >> > types at all (or move the relevant stuff to be UI only). >> > >> > >> > >> > On Fri, Apr 12, 2019 at 6:07 PM Daniel Imberman < >> daniel.imber...@gmail.com >> > > >> > wrote: >> > >> > > I think that's the important distinction. This wouldn't be >removing >> > custom >> > > hooks and operators, this is just removing the python magic that >has >> you >> > > placing them in a plugins folder. I can see the value in having >the >> DAGs >> > > being real-time parsed as users might want to make small changes >to >> > those, >> > > but operators/hooks should be fairly static once created. >> > > >> > > On Fri, Apr 12, 2019 at 8:53 AM Ash Berlin-Taylor ><a...@apache.org> >> > wrote: >> > > >> > > > Can you give an example? >> > > > >> > > > We can still have custom hooks/operators per install, I'm just >saying >> > > they >> > > > don't need to be implemented/"registered" with airflow.plugin >-- all >> > that >> > > > provides is another way of having that be imported. >> > > > >> > > > -ash >> > > > >> > > > > On 12 Apr 2019, at 16:21, Chen Tong <cix...@gmail.com> wrote: >> > > > > >> > > > > IMHO, it's worth to have such dynamic hooks adding ability. I >met >> > > issues >> > > > > before that the current hook cannot satisfy the needs. I have >to >> > write >> > > my >> > > > > own hook and hacked Connections. If hook is not coupled >tightly >> with >> > > > > Connections, it will be easier by switching to another python >> > package. >> > > > > >> > > > > On Fri, Apr 12, 2019 at 11:14 AM Daniel Imberman < >> > > > daniel.imber...@gmail.com> >> > > > > wrote: >> > > > > >> > > > >> I am all for this. The only complexity is ensuring the >> operator/hook >> > > > >> exists in the workers too, but overall highly for. >> > > > >> On Apr 12, 2019, 7:37 AM -0700, James Meickle < >> > > jmeic...@quantopian.com >> > > > .invalid>, >> > > > >> wrote: >> > > > >>> YES - I strongly agree with this! I first did it this way >> because I >> > > > >> wanted >> > > > >>> to follow the instructions, assuming there was some Airflow >> magic, >> > > and >> > > > >>> later found it really frustrating to maintain. We should be >clear >> > > that >> > > > >>> standard Python packaging is the way to go. >> > > > >>> >> > > > >>> That being said, what if Airflow had an official >Cookiecutter >> > > template >> > > > >> for >> > > > >>> Airflow operator packages, and suggested that as a way to >manage >> > your >> > > > >>> operators in the docs? That would probably help people who >aren't >> > as >> > > > >>> familiar with Python, and over time might increase quality >of >> third >> > > > party >> > > > >>> operators (if it ships by default with linting/etc.) >> > > > >>> >> > > > >>> On Fri, Apr 12, 2019 at 3:54 AM Ash Berlin-Taylor < >> a...@apache.org> >> > > > >> wrote: >> > > > >>> >> > > > >>>> This is something I've said a number of times (but >possibly >> never >> > on >> > > > >> the >> > > > >>>> list): >> > > > >>>> >> > > > >>>> I think we should deprecate adding Operators and Hooks via >the >> > > Airflow >> > > > >>>> plugin mechanism. >> > > > >>>> >> > > > >>>> I think plugins should be reserved for any mechanism that >a >> > plain-ol >> > > > >>>> python module import won't work for (which is basically >anything >> > > that >> > > > >> needs >> > > > >>>> to tie deeply in to the Webserver or Scheduler process). >> > > > >>>> >> > > > >>>> To that endI think we should deprecate adding operators >via >> > plugins: >> > > > >>>> >> > > > >>>> from airflow.operators.my_plugin import MyOperator >> > > > >>>> >> > > > >>>> can become >> > > > >>>> >> > > > >>>> from my_plugin import MyOperator >> > > > >>>> >> > > > >>>> with no impact on functionality. >> > > > >>>> >> > > > >>>> The same could be said for hooks, with one possible >feature >> > > addition: >> > > > >>>> Right now the list of connection types in the Connections >screen >> > is >> > > > >>>> hard-coded. Is it possible worth making that dynamic >somehow >> based >> > > on >> > > > >> the >> > > > >>>> Hook classes that are loaded? i.e. adding the ability for >hooks >> > > added >> > > > >> in >> > > > >>>> plguins to add items to that list? >> > > > >>>> >> > > > >>>> -ash >> > > > >> >> > > > >> > > > >> > > >> > >> > >> > -- >> > >> > Jarek Potiuk >> > Polidea <https://www.polidea.com/> | Principal Software Engineer >> > >> > M: +48 660 796 129 <+48660796129> >> > E: jarek.pot...@polidea.com >> > >> > > >-- > >Jarek Potiuk >Polidea <https://www.polidea.com/> | Principal Software Engineer > >M: +48 660 796 129 <+48660796129> >E: jarek.pot...@polidea.com
