I, unfortunatly, couldn't make it to the Meetup. What was discussed in regards to plugins?
On 2019/02/18 12:27:46, David Sabater Dinter <[email protected]> wrote: > Hi all, > Just following up from this recommendation and the discussion we had during > our last Airflow London Meetup @Google > <https://www.meetup.com/London-Apache-Airflow-Meetup/events/257090138/>. > Are there any plans to make that official in anyway? > It would be great to have some direction/steer so the community can > implement custom hooks/operators in the right way. > > > Kind regards. > > On Mon, 14 Jan 2019 at 13:50, Björn Pollex > <[email protected]> wrote: > > > This is what we’ve been doing internally, and if it becomes the “official” > > way of doing things, then I’m more than happy. > > > > Cheers, > > > > Björn > > > > > On 14. Jan 2019, at 13:38, Ash Berlin-Taylor <[email protected]> wrote: > > > > > > Operators and hooks are just normal python modules, and don't actually > > need to be defined under `airflow.` to work. > > > > > > My advice (and something I've been meaning to update the docs to say) is > > to just make your custom hooks/ops a normal class and do > > > > > > from myco.airflow.operators.my_op import MyOp > > > > > > So long as the "myco" package is installed on the scheduler and worker > > nodes this will work fine (this is what we do at work) > > > > > > The plugin mechanism is I think still needed to get Flask > > views/menu_items integrated (as the need to be registered with the app at > > creation time)? > > > > > > So: not to namespace packages as I don't think we need it. > > > > > > -ash > > > > > >> On 14 Jan 2019, at 11:58, Björn Pollex > > >> <[email protected]> > > wrote: > > >> > > >> With the recent (awesome) new feature to [load plugins from entry > > points][1], I’m wondering if this could be taken further, to make handling > > plugins simpler and less magic. One pain point for me personally has been > > that, since plugins are loaded by Airflow magic, my IDE cannot give me any > > help (e.g. syntax completion) when using operators, hooks or similar that > > are loaded from plugins, because it can’t find the source files based on > > the package name. This could be fixed by changing the packages `airflow`, > > `airflow.hooks`, `airflow.operators` etc. into [namespace packages][2]. > > With such a change, one wouldn’t even need to declare a plugin in order to > > distribute a new hook or operator, a plain Python package is enough. > > >> > > >> Furthermore, I wonder if we need the `AirflowPlugin` class at all. > > Instead, we could have dedicated entry points for views, custom menus, > > etc., and each package would register relevant classes with those entry > > points directly. This would remove a lot of magic from Airflow, and instead > > use established tools to handle the complexity. > > >> > > >> I believe it would be possible to implement both of these without > > breaking existing code, so a migration path is available. The main problem > > is that we would need to move all code that currently resides in the > > `__init__.py` files of the prospective namespace packages elsewhere (the > > constituent parts of namespace packages are loaded in a non-deterministic > > order, so one can’t rely on code any specific `__init__.py` if a namespace > > package to be executed first, or, in fact, at all), but as far as I can > > see, everything in there can neatly fit somewhere else. > > >> > > >> I would love to start working on this, if there is a chance that at > > least parts of this would find acceptance. What does the Airflow dev > > community think about this? > > >> > > >> Cheers, > > >> > > >> Björn Pollex > > >> > > >> [1]: https://github.com/apache/airflow/pull/4412 > > >> [2]: https://packaging.python.org/guides/packaging-namespace-packages/ > > >> > > > > > > > > > -- > > > David Sabater > > Google Cloud | Cluster Customer Engineer > > [email protected] > > Belgrave House, 76 Buckingham Palace Road, London SW1W 9TQ > <https://maps.google.com/?q=76+Buckingham+Palace+Road,+London+SW1W+9TQ&entry=gmail&source=g> > > Tel +44 (0)7826 532190 > > > This email may be confidential and privileged. If you received this > communication by mistake, please don't forward it to anyone else, please > erase all copies and attachments, and please let me know that it has gone > to the wrong person. > > The above terms reflect a potential business arrangement, are provided > solely as a basis for further discussion, and are not intended to be and do > not constitute a legally binding obligation. No legally binding obligations > will be created, implied, or inferred until an agreement in final form is > executed in writing by all parties involved. > > Google UK Limited > Registered Office: Belgrave House, 76 Buckingham Palace Road, London SW1W > 9TQ > Registered in England Number: 3977902 >
