Re: [2.0 spring cleaning] Require unique conn_id

It can get more confusing because airflow allow to create two connection with same conn_id but different conn_type https://issues.apache.org/jira/browse/AIRFLOW-2784 Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Sunday, April 14, 2019 12:22 AM, Maxime Beauchemin wrote

Re: [2.0 spring cleaning] Deprecate adding Operators and Hooks via plugins?

+1 for using more entrypoints as discovery mechanism. Maybe we should also start to use the 'console_scripts' entrypoint group for "airflow" binary instead of copying the binary airflow script ( https://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-po

Re: Is `airflow backfill` disfunctional?

+1, backfilling, and related "subdag surgeries" are core to a data engineer's job, and great tooling around this is super important. Backfill needs more TLC! Max On Fri, Apr 12, 2019 at 11:48 PM Chao-Han Tsai wrote: > +1 on improving backfill. > > - The terminal interface was uselessly verbose.

Re: [2.0 spring cleaning] Require unique conn_id

People may rely on this feature for [poor man's] load balancing though, I forgot what the exact use case was but used this at Airbnb at some point. Maybe the solution is to make the UI/UX/log output much more clear around this. Making the CLI log more clear should be really easy to do, web server

Re: [2.0 spring cleaning] Deprecate adding Operators and Hooks via plugins?

I'd say it could be great to deprecate the whole plugin system and use Python's "entry points" instead. I just didn't know that was an option and the standard way to do this when I originally wrote it... The current plugin system is a minefield for circular dependencies... On Sat, Apr 13, 2019 at

Re: Airflow 1.10.3 has been released!

I am impressed by your hard work. I know that cherry-pick my doc changes was not a simple process. :-) Thanks, Ash :-) On Sat, Apr 13, 2019 at 12:03 PM Felix Uellendall wrote: > Thanks Ash :) > > Am 13/04/2019 um 01:33 schrieb Maxime Beauchemin: > > Infinite kudos! Thanks Ash! > > > > On Fri, A

Re: [2.0 spring cleaning] Deprecate adding Operators and Hooks via plugins?

One comment here - The current connections.py file screams to get dynamic discovery similar to what sqlalchemy does for dialects (for example). But on the other hand, I am not sure if this "registration" is really needed. Maybe I am totally wrong, but somehow I don't find it particularly useful to

Re: [VOTE] AIP-10: Multi-layered and multi-stage official Airflow image

Cool! Glad it's approved :). I see that you merged already 1st out of 3 PRs and the latest image is now built using multi-layered Dockerfile: https://hub.docker.com/r/apache/airflow/dockerfile :D I updated the AIP-10 label to accepted and will follow-up according to the proposed schedule

Re: [AIP-19] Making the webserver stateless

Hi Julian, Thanks for adding that exhaustive list of changes that are needed for each view. Assuming we went with option 2b for obtaining information about the edges: > Adding the current state of the DAG in the database, so that edges reflect the most recent version of DAG as it was parsed. Do

Re: [VOTE] AIP-10: Multi-layered and multi-stage official Airflow image

You missed to count my vote 😁 On Sat, Apr 13, 2019, 14:00 Driesprong, Fokko wrote: > Hi all, > > I'm happy to announce that the has passed. > > Binding +1: > Fokko Driesprong > Maxime Beauchemin > Tao Feng > Daniel Imberman > > Non-binding +1: > Jarek Potiuk > Flo Rance > Kamil Bregula > Chao-Ha

Re: [VOTE] AIP-10: Multi-layered and multi-stage official Airflow image

Hi all, I'm happy to announce that the has passed. Binding +1: Fokko Driesprong Maxime Beauchemin Tao Feng Daniel Imberman Non-binding +1: Jarek Potiuk Flo Rance Kamil Bregula Chao-Han Tsai Jajie Zhong Felix Uellendall Cheers, Fokko Op za 13 apr. 2019 om 11:51 schreef Felix Uellendall < felix.

Re: Longer term Airflow planning

Hello Everyone, Few thoughts which I had after proposing a few changes/PRs/AIPs. I think we should not only look at it from the system point of view but also from human psychology and emotions point of view :) * I personally felt pretty demotivated (that's the emotion part) by seeing 200 opened P

Re: [PLEASE PARTICIPATE] Brainstorming ideas for Season of Docs

One more idea - providing that we will implement AIP-7 Simplified Development Workflow ( https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow) - maybe it would be worthwhile to take a look at updates in the "Contributors" document and "Breeze" documentation. Tha

Re: [2.0 spring cleaning] Deprecate contrib folder?

I think there are quite a few contrib parts that are at least on-par with regards to code quality, testing and especially documentation. And yes, among those are GCP operators we developed which are not only unit but also system-tested and we put quite an effort into making documentation really u

Re: [DISCUSS]: Remove Mesos Executor from Airflow 2.0.0?

+1 (non-binding) Am 12/04/2019 um 23:42 schrieb Tao Feng: +1 on removing mesos executor. On Fri, Apr 12, 2019 at 2:25 PM Daniel Imberman wrote: Since we are now doing version 2.0 spring cleaning can we revisit this? On Wed, Feb 27, 2019 at 8:38 AM Maxime Beauchemin < maximebeauche...@gmail.

Re: Airflow 1.10.3 has been released!

Thanks Ash :) Am 13/04/2019 um 01:33 schrieb Maxime Beauchemin: Infinite kudos! Thanks Ash! On Fri, Apr 12, 2019 at 1:33 PM Kaxil Naik wrote: Thanks Ash and all the contributors. On Fri, Apr 12, 2019, 20:38 Robin Edwards wrote: Yes thanks everyone for their hard work :-) R On Fri, 12 A

Re: [2.0 spring cleaning] Deprecate contrib folder?

+1 on deprecating the contrib folder. Bolk de Bruin the reason the core hooks and operators are properly tested because, for example I added some more tests to it and I am "only" a contributor. So do you really want to split up contributors work and core committers work? I personally think this

Re: [VOTE] AIP-10: Multi-layered and multi-stage official Airflow image

+1 (non-binding) Am 13/04/2019 um 02:55 schrieb Daniel Imberman: +1 (binding) On Fri, Apr 12, 2019, 4:58 PM Tao Feng wrote: +1 binding On Mon, Apr 8, 2019 at 2:06 PM Kaxil Naik wrote: +1 binding On Mon, Apr 8, 2019, 21:50 Maxime Beauchemin +1 binding Max On Sun, Apr 7, 2019 at 5:23 P

Re: [2.0 spring cleaning] Deprecate subdags

-1 on deprecating subdags, because of the extra level of abstraction some of you already mentioned. We also use subdags in production. For example in cases where we get json data from an API but since we mostly need it to be in csv format we have a subdag like /specific_ap//i_specific_endpoint/_

Re: [VOTE] AIP-6 Apply Pylint to Airflow

+1 (non-binding) -feluelle Am 13/04/2019 um 07:53 schrieb Chao-Han Tsai: +1 (non-binding) On Fri, Apr 12, 2019 at 8:05 AM Chen Tong wrote: +1 (non-binding). On Fri, Apr 12, 2019 at 9:28 AM Julian De Ruiter < julianderui...@godatadriven.com> wrote: I also wouldn’t mind including black for