Operators and hooks are just normal python modules, and don't actually need to
be defined under `airflow.` to work.
My advice (and something I've been meaning to update the docs to say) is to
just make your custom hooks/ops a normal class and do
from myco.airflow.operators.my_op import MyOp
So long as the "myco" package is installed on the scheduler and worker nodes
this will work fine (this is what we do at work)
The plugin mechanism is I think still needed to get Flask views/menu_items
integrated (as the need to be registered with the app at creation time)?
So: not to namespace packages as I don't think we need it.
-ash
> On 14 Jan 2019, at 11:58, Björn Pollex <[email protected]>
> wrote:
>
> With the recent (awesome) new feature to [load plugins from entry points][1],
> I’m wondering if this could be taken further, to make handling plugins
> simpler and less magic. One pain point for me personally has been that, since
> plugins are loaded by Airflow magic, my IDE cannot give me any help (e.g.
> syntax completion) when using operators, hooks or similar that are loaded
> from plugins, because it can’t find the source files based on the package
> name. This could be fixed by changing the packages `airflow`,
> `airflow.hooks`, `airflow.operators` etc. into [namespace packages][2]. With
> such a change, one wouldn’t even need to declare a plugin in order to
> distribute a new hook or operator, a plain Python package is enough.
>
> Furthermore, I wonder if we need the `AirflowPlugin` class at all. Instead,
> we could have dedicated entry points for views, custom menus, etc., and each
> package would register relevant classes with those entry points directly.
> This would remove a lot of magic from Airflow, and instead use established
> tools to handle the complexity.
>
> I believe it would be possible to implement both of these without breaking
> existing code, so a migration path is available. The main problem is that we
> would need to move all code that currently resides in the `__init__.py` files
> of the prospective namespace packages elsewhere (the constituent parts of
> namespace packages are loaded in a non-deterministic order, so one can’t rely
> on code any specific `__init__.py` if a namespace package to be executed
> first, or, in fact, at all), but as far as I can see, everything in there can
> neatly fit somewhere else.
>
> I would love to start working on this, if there is a chance that at least
> parts of this would find acceptance. What does the Airflow dev community
> think about this?
>
> Cheers,
>
> Björn Pollex
>
> [1]: https://github.com/apache/airflow/pull/4412
> [2]: https://packaging.python.org/guides/packaging-namespace-packages/
>
