I think that's the important distinction. This wouldn't be removing custom hooks and operators, this is just removing the python magic that has you placing them in a plugins folder. I can see the value in having the DAGs being real-time parsed as users might want to make small changes to those, but operators/hooks should be fairly static once created.
On Fri, Apr 12, 2019 at 8:53 AM Ash Berlin-Taylor <a...@apache.org> wrote: > Can you give an example? > > We can still have custom hooks/operators per install, I'm just saying they > don't need to be implemented/"registered" with airflow.plugin -- all that > provides is another way of having that be imported. > > -ash > > > On 12 Apr 2019, at 16:21, Chen Tong <cix...@gmail.com> wrote: > > > > IMHO, it's worth to have such dynamic hooks adding ability. I met issues > > before that the current hook cannot satisfy the needs. I have to write my > > own hook and hacked Connections. If hook is not coupled tightly with > > Connections, it will be easier by switching to another python package. > > > > On Fri, Apr 12, 2019 at 11:14 AM Daniel Imberman < > daniel.imber...@gmail.com> > > wrote: > > > >> I am all for this. The only complexity is ensuring the operator/hook > >> exists in the workers too, but overall highly for. > >> On Apr 12, 2019, 7:37 AM -0700, James Meickle <jmeic...@quantopian.com > .invalid>, > >> wrote: > >>> YES - I strongly agree with this! I first did it this way because I > >> wanted > >>> to follow the instructions, assuming there was some Airflow magic, and > >>> later found it really frustrating to maintain. We should be clear that > >>> standard Python packaging is the way to go. > >>> > >>> That being said, what if Airflow had an official Cookiecutter template > >> for > >>> Airflow operator packages, and suggested that as a way to manage your > >>> operators in the docs? That would probably help people who aren't as > >>> familiar with Python, and over time might increase quality of third > party > >>> operators (if it ships by default with linting/etc.) > >>> > >>> On Fri, Apr 12, 2019 at 3:54 AM Ash Berlin-Taylor <a...@apache.org> > >> wrote: > >>> > >>>> This is something I've said a number of times (but possibly never on > >> the > >>>> list): > >>>> > >>>> I think we should deprecate adding Operators and Hooks via the Airflow > >>>> plugin mechanism. > >>>> > >>>> I think plugins should be reserved for any mechanism that a plain-ol > >>>> python module import won't work for (which is basically anything that > >> needs > >>>> to tie deeply in to the Webserver or Scheduler process). > >>>> > >>>> To that endI think we should deprecate adding operators via plugins: > >>>> > >>>> from airflow.operators.my_plugin import MyOperator > >>>> > >>>> can become > >>>> > >>>> from my_plugin import MyOperator > >>>> > >>>> with no impact on functionality. > >>>> > >>>> The same could be said for hooks, with one possible feature addition: > >>>> Right now the list of connection types in the Connections screen is > >>>> hard-coded. Is it possible worth making that dynamic somehow based on > >> the > >>>> Hook classes that are loaded? i.e. adding the ability for hooks added > >> in > >>>> plguins to add items to that list? > >>>> > >>>> -ash > >> > >
