I am all for this. The only complexity is ensuring the operator/hook exists in
the workers too, but overall highly for.
On Apr 12, 2019, 7:37 AM -0700, James Meickle
<jmeic...@quantopian.com.invalid>, wrote:
> YES - I strongly agree with this! I first did it this way because I wanted
> to follow the instructions, assuming there was some Airflow magic, and
> later found it really frustrating to maintain. We should be clear that
> standard Python packaging is the way to go.
>
> That being said, what if Airflow had an official Cookiecutter template for
> Airflow operator packages, and suggested that as a way to manage your
> operators in the docs? That would probably help people who aren't as
> familiar with Python, and over time might increase quality of third party
> operators (if it ships by default with linting/etc.)
>
> On Fri, Apr 12, 2019 at 3:54 AM Ash Berlin-Taylor <a...@apache.org> wrote:
>
> > This is something I've said a number of times (but possibly never on the
> > list):
> >
> > I think we should deprecate adding Operators and Hooks via the Airflow
> > plugin mechanism.
> >
> > I think plugins should be reserved for any mechanism that a plain-ol
> > python module import won't work for (which is basically anything that needs
> > to tie deeply in to the Webserver or Scheduler process).
> >
> > To that endI think we should deprecate adding operators via plugins:
> >
> > from airflow.operators.my_plugin import MyOperator
> >
> > can become
> >
> > from my_plugin import MyOperator
> >
> > with no impact on functionality.
> >
> > The same could be said for hooks, with one possible feature addition:
> > Right now the list of connection types in the Connections screen is
> > hard-coded. Is it possible worth making that dynamic somehow based on the
> > Hook classes that are loaded? i.e. adding the ability for hooks added in
> > plguins to add items to that list?
> >
> > -ash
