With the recent (awesome) new feature to [load plugins from entry points][1],
I’m wondering if this could be taken further, to make handling plugins simpler
and less magic. One pain point for me personally has been that, since plugins
are loaded by Airflow magic, my IDE cannot give me any help (e.g. syntax
completion) when using operators, hooks or similar that are loaded from
plugins, because it can’t find the source files based on the package name. This
could be fixed by changing the packages `airflow`, `airflow.hooks`,
`airflow.operators` etc. into [namespace packages][2]. With such a change, one
wouldn’t even need to declare a plugin in order to distribute a new hook or
operator, a plain Python package is enough.
Furthermore, I wonder if we need the `AirflowPlugin` class at all. Instead, we
could have dedicated entry points for views, custom menus, etc., and each
package would register relevant classes with those entry points directly. This
would remove a lot of magic from Airflow, and instead use established tools to
handle the complexity.
I believe it would be possible to implement both of these without breaking
existing code, so a migration path is available. The main problem is that we
would need to move all code that currently resides in the `__init__.py` files
of the prospective namespace packages elsewhere (the constituent parts of
namespace packages are loaded in a non-deterministic order, so one can’t rely
on code any specific `__init__.py` if a namespace package to be executed first,
or, in fact, at all), but as far as I can see, everything in there can neatly
fit somewhere else.
I would love to start working on this, if there is a chance that at least parts
of this would find acceptance. What does the Airflow dev community think about
this?
Cheers,
Björn Pollex
[1]: https://github.com/apache/airflow/pull/4412
[2]: https://packaging.python.org/guides/packaging-namespace-packages/
