IMHO, it's worth to have such dynamic hooks adding ability. I met issues before that the current hook cannot satisfy the needs. I have to write my own hook and hacked Connections. If hook is not coupled tightly with Connections, it will be easier by switching to another python package.
On Fri, Apr 12, 2019 at 11:14 AM Daniel Imberman <daniel.imber...@gmail.com> wrote: > I am all for this. The only complexity is ensuring the operator/hook > exists in the workers too, but overall highly for. > On Apr 12, 2019, 7:37 AM -0700, James Meickle > <jmeic...@quantopian.com.invalid>, > wrote: > > YES - I strongly agree with this! I first did it this way because I > wanted > > to follow the instructions, assuming there was some Airflow magic, and > > later found it really frustrating to maintain. We should be clear that > > standard Python packaging is the way to go. > > > > That being said, what if Airflow had an official Cookiecutter template > for > > Airflow operator packages, and suggested that as a way to manage your > > operators in the docs? That would probably help people who aren't as > > familiar with Python, and over time might increase quality of third party > > operators (if it ships by default with linting/etc.) > > > > On Fri, Apr 12, 2019 at 3:54 AM Ash Berlin-Taylor <a...@apache.org> > wrote: > > > > > This is something I've said a number of times (but possibly never on > the > > > list): > > > > > > I think we should deprecate adding Operators and Hooks via the Airflow > > > plugin mechanism. > > > > > > I think plugins should be reserved for any mechanism that a plain-ol > > > python module import won't work for (which is basically anything that > needs > > > to tie deeply in to the Webserver or Scheduler process). > > > > > > To that endI think we should deprecate adding operators via plugins: > > > > > > from airflow.operators.my_plugin import MyOperator > > > > > > can become > > > > > > from my_plugin import MyOperator > > > > > > with no impact on functionality. > > > > > > The same could be said for hooks, with one possible feature addition: > > > Right now the list of connection types in the Connections screen is > > > hard-coded. Is it possible worth making that dynamic somehow based on > the > > > Hook classes that are loaded? i.e. adding the ability for hooks added > in > > > plguins to add items to that list? > > > > > > -ash >
