Can you give an example? We can still have custom hooks/operators per install, I'm just saying they don't need to be implemented/"registered" with airflow.plugin -- all that provides is another way of having that be imported.
-ash > On 12 Apr 2019, at 16:21, Chen Tong <cix...@gmail.com> wrote: > > IMHO, it's worth to have such dynamic hooks adding ability. I met issues > before that the current hook cannot satisfy the needs. I have to write my > own hook and hacked Connections. If hook is not coupled tightly with > Connections, it will be easier by switching to another python package. > > On Fri, Apr 12, 2019 at 11:14 AM Daniel Imberman <daniel.imber...@gmail.com> > wrote: > >> I am all for this. The only complexity is ensuring the operator/hook >> exists in the workers too, but overall highly for. >> On Apr 12, 2019, 7:37 AM -0700, James Meickle >> <jmeic...@quantopian.com.invalid>, >> wrote: >>> YES - I strongly agree with this! I first did it this way because I >> wanted >>> to follow the instructions, assuming there was some Airflow magic, and >>> later found it really frustrating to maintain. We should be clear that >>> standard Python packaging is the way to go. >>> >>> That being said, what if Airflow had an official Cookiecutter template >> for >>> Airflow operator packages, and suggested that as a way to manage your >>> operators in the docs? That would probably help people who aren't as >>> familiar with Python, and over time might increase quality of third party >>> operators (if it ships by default with linting/etc.) >>> >>> On Fri, Apr 12, 2019 at 3:54 AM Ash Berlin-Taylor <a...@apache.org> >> wrote: >>> >>>> This is something I've said a number of times (but possibly never on >> the >>>> list): >>>> >>>> I think we should deprecate adding Operators and Hooks via the Airflow >>>> plugin mechanism. >>>> >>>> I think plugins should be reserved for any mechanism that a plain-ol >>>> python module import won't work for (which is basically anything that >> needs >>>> to tie deeply in to the Webserver or Scheduler process). >>>> >>>> To that endI think we should deprecate adding operators via plugins: >>>> >>>> from airflow.operators.my_plugin import MyOperator >>>> >>>> can become >>>> >>>> from my_plugin import MyOperator >>>> >>>> with no impact on functionality. >>>> >>>> The same could be said for hooks, with one possible feature addition: >>>> Right now the list of connection types in the Connections screen is >>>> hard-coded. Is it possible worth making that dynamic somehow based on >> the >>>> Hook classes that are loaded? i.e. adding the ability for hooks added >> in >>>> plguins to add items to that list? >>>> >>>> -ash >>
