Somewhat off topic from the topic of just moving the well tested parts out of contrib/ but okay. I thought I had somewhere but I can't find where I did it before.
Quickly: - How do we go about testing changes to core against hook? - Who/what/when releases the split-out hooks. Are they ASF releases or community. (If community then this can happen right now. Someone can just create a package, publish on pypi and we can start pointing at that) - What about CI? How do we test changes to core against operators? When? How often? - Some hooks need tighter integration (GCP,S3, Azure, ElasticSearch) as they can be used by loggers. How do we do that? - A "operator" package should probably depend upon Airflow to get "BaseOperator" class etc, but that means that installing the `apache-airflow[extra]` can't work as that would be a loop. Maybe we need an `apache-airflow-core` instead. In short: I like "mono" repo as it has lots of advantages :) -ash > On 12 Apr 2019, at 16:10, Chen Tong <cix...@gmail.com> wrote: > > Hi Ash, > > Could you state the questions you are concerned about? We may discuss it in > this thread. > > Thank you! > > On Fri, Apr 12, 2019 at 11:06 AM Ash Berlin-Taylor <a...@apache.org> wrote: > >> Speaking as a release manager AIP-8 also had some large unanswered >> workflow questions before I'd be happy about to "accept" it. >> >> -ash >> >>> On 12 Apr 2019, at 16:04, Chen Tong <cix...@gmail.com> wrote: >>> >>> AIP-8 is intended to achieve this by separate operators/hooks to >> individual >>> packages. But the author hasn't been actively working on this recently. >>> >>> On Fri, Apr 12, 2019 at 9:29 AM Julian De Ruiter < >>> julianderui...@godatadriven.com> wrote: >>> >>>> Isn’t this in contradiction with AIP-8, which is aimed at removing >>>> operators/hooks from the core Airflow package? >>>> >>>> Personally I would rather remove hooks/operators from Airflow than add >>>> even more to the Airflow core. This counts double for the contrib stuff, >>>> which is often poorly designed and/or tested. >>>> >>>> Best, >>>> Julian >>>> >>>>> On 12 Apr 2019, at 10:23, Bolke de Bruin <bdbr...@gmail.com> wrote: >>>>> >>>>> That’s perfectly fine to me. >>>>> >>>>> Verstuurd vanaf mijn iPad >>>>> >>>>>> Op 12 apr. 2019 om 10:20 heeft Kaxil Naik <kaxiln...@gmail.com> het >>>> volgende geschreven: >>>>>> >>>>>> Ok. How about moving the properly tested and maintained hooks/ops from >>>>>> contrib to core? >>>>>> >>>>>>> On Fri, Apr 12, 2019, 09:13 Bolke de Bruin <bdbr...@gmail.com> >> wrote: >>>>>>> >>>>>>> I disagree. Core signals “properly tested” and maintained. Ie. A kind >>>> of >>>>>>> quality. I don’t think contrib has that. >>>>>>> >>>>>>> Verstuurd vanaf mijn iPad >>>>>>> >>>>>>>> Op 12 apr. 2019 om 10:03 heeft Kaxil Naik <kaxiln...@gmail.com> het >>>>>>> volgende geschreven: >>>>>>>> >>>>>>>> Contrib folder was used when it was used at Airbnb. Currently, it >>>> doesn't >>>>>>>> make any sense and we have equal responsibility to maintain all the >>>>>>> hooks, >>>>>>>> operators, sensors in contrib folder as we do for core. >>>>>>>> >>>>>>>> I would suggest to remove contrib folder and move all hooks, ops, >> and >>>>>>>> sensors to the core folder. >>>>>>>> >>>>>>>> Or reorganize the folder structure similar to what was discussed in >> a >>>>>>>> mailing thread few months ago. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Kaxil >>>>>>> >>>> >>>> >> >>
