[Meeting Notes] Airflow 2.0 Dev Call #2 - 24 Aug 2020

Hi all, I have created a document to summarize the discussion from our second dev call for Airflow 2.0. Thank you all who joined the call. *Doc Link*: https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020 To all those who attended, can you please double-che

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 (binding) On Tue, Aug 25, 2020, 23:35 Daniel Imberman wrote: > +1 binding > > via Newton Mail > [ > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2 > ] > On Tue, Aug 25, 2020 at 8:45 AM, Gerard Casas Saez > wrote: > + 1 (non-binding) > > Gerard Casas Saez

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 binding via Newton Mail [https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.50&pv=10.15.5&source=email_footer_2] On Tue, Aug 25, 2020 at 8:45 AM, Gerard Casas Saez wrote: + 1 (non-binding) Gerard Casas Saez Twitter | Cortex | @casassaez On Tue, Aug 25, 2020

[ANNOUNCE] Apache Airflow version 1.10.12 Released

Dear Airflow community, I am pleased to announce that *Airflow 1.10.12* has just been released. Apache Airflow - A platform to programmatically author, schedule, and monitor workflows The official source release: https://dist.apache.org/repos/dist/release/airflow/1.10.12 The PyPI package: https

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

Well it became a long list eh and it was used for bootstrapping the community. So I think it's value has changed. You might want to consider categorizing it or make it into the "one million dollar" page (placing the logos on 1mio squared pixels) or a kind of overview page. Say as a potential

[RESULT][VOTE] Release Apache Airflow 1.10.12 based on 1.10.12rc4

Hello Airflow Community, The vote to release Apache Airflow version 1.10.12 based on 1.10.12rc4 is now closed. The vote PASSED with 4 binding "+1", 0 non-binding "+1" and 0 "-1" votes: "+1" Binding votes: - Kamil Breguła - Tomasz Urbaszek - Jarek Potiuk - Kaxil Naik I'll continue with the relea

Re: [Discuss] Improving AIP process documentation

I also prefer Github Issues than either CWiki or PR. I think there is little contention and usually, one person at a time leads AIP description and updates the doc but others can still edit it directly for minor changes. Github issues has a history of changes (the ... at the top) and this is enoug

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

I am fine with the "INTHEWILD" approach. On Tue, Aug 25, 2020, 17:59 Ry Walker wrote: > I like this idea, Max - best of both worlds, simplifies README while > keeping the list in the repo as an encouragement to get a "first commit" > > I'm happy to do PR and draft the communication, unless there

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

I like this idea, Max - best of both worlds, simplifies README while keeping the list in the repo as an encouragement to get a "first commit" I'm happy to do PR and draft the communication, unless there is disagreement with this plan. -Ry On Tue, Aug 25, 2020 at 11:49 AM Maxime Beauchemin < maxi

Re: [Discuss] Improving AIP process documentation

I am actually against the idea of having to PRs for editing AIPs. I think the current separation works better. Confluence is fine for me as it is because of its integrations. Don't have anything against Github issue but definitely not worth having it in Airflow docs which needs to be PR'd On Tue,

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

INTHEWILD.md is as close to a standard as it gets for this. I'd advocate to move the list there and have a link to it in the README. Here's the Superset one for reference: https://github.com/apache/incubator-superset/blob/master/INTHEWILD.md Personally I like a short clean README with links to key

Re: [Discuss] Improving AIP process documentation

In that direction, I would prefer having markdown docs and PRs instead actually, similarly to how github.com/tensorflow/community does it. Thoughts? Will do a second pass to the later today :D Gerard Casas Saez Twitter | Cortex | @casassaez On Tue, Aug 25, 2020 at

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+ 1 (non-binding) Gerard Casas Saez Twitter | Cortex | @casassaez On Tue, Aug 25, 2020 at 7:18 AM Felix Uellendall wrote: > +1 (binding) > > Felix > > On Tue, Aug 25, 2020 at 15:15, James Coder wrote: > > > +1 (non-binding) > > > > James Coder > > > >> On Aug 25

Re: Airflow + Hacktoberfest 2020

Yes, I'm in as well, sorry for slow response! We can probably bring a few others from Astronomer to join in too (for example, Pete DeJoy from our team is eager to get involved here). -Ry On Tue, Aug 25, 2020 at 3:53 AM Jarek Potiuk wrote: > Cool! QP I will let you know of the plans later this

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

I think that‘s a good idea to move it to the official website of Airflow instead of the code repository. A while ago I also thought about moving it somewhere else into a different file of the repo - but the Airflow website in my opinion is the best place for it which will also probably reach mo

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 (binding) Felix On Tue, Aug 25, 2020 at 15:15, James Coder wrote: > +1 (non-binding) > > James Coder > >> On Aug 25, 2020, at 9:08 AM, Tomasz Urbaszek wrote: >> >> +1 (binding) >> >> T. >> >> >>> On Tue, Aug 25, 2020 at 3:02 PM Jarek Potiuk >>> wrote: >>> >>> +1 (binding) >>> On Tue

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

On Tue, Aug 25, 2020 at 2:47 PM Ry Walker wrote: > > Keep in mind that if it gets moved, it won’t get updated nearly as often > (less of an incentive) so I’d say it should be a keep or delete decision. I am not so sure about it. I think it's quite the opposite. We can keep link to our website, a

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 (non-binding) James Coder > On Aug 25, 2020, at 9:08 AM, Tomasz Urbaszek wrote: > > +1 (binding) > > T. > > >> On Tue, Aug 25, 2020 at 3:02 PM Jarek Potiuk >> wrote: >> >> +1 (binding) >> >>> On Tue, Aug 25, 2020 at 2:56 PM Kaxil Naik wrote: >>> >>> +1 (binding) >>> On Tue, A

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 (binding) T. On Tue, Aug 25, 2020 at 3:02 PM Jarek Potiuk wrote: > > +1 (binding) > > On Tue, Aug 25, 2020 at 2:56 PM Kaxil Naik wrote: > > > +1 (binding) > > > > On Tue, Aug 25, 2020 at 1:55 PM Yu Qian wrote: > > > > > Hello everyone! > > > > > > This email calls for a vote on the design

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 (binding) On Tue, Aug 25, 2020 at 2:56 PM Kaxil Naik wrote: > +1 (binding) > > On Tue, Aug 25, 2020 at 1:55 PM Yu Qian wrote: > > > Hello everyone! > > > > This email calls for a vote on the design proposed in AIP-34: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskG

Re: [Discuss] Improving AIP process documentation

Yeah I’d like for AIPs to be GitHub issues, more likely to get better attention since we’re all in GitHub way more than cwiki. On Tue, Aug 25, 2020 at 4:01 AM Jarek Potiuk wrote: > Looks great Gerard! I think maybe one last pass with Superset's docs. > > > > Side comment. I really like that SIP'

Re: [VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

+1 (binding) On Tue, Aug 25, 2020 at 1:55 PM Yu Qian wrote: > Hello everyone! > > This email calls for a vote on the design proposed in AIP-34: > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator > > Discussio

[VOTE] AIP-34 TaskGroup: A UI task grouping concept as an alternative to SubDagOperator

Hello everyone! This email calls for a vote on the design proposed in AIP-34: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator Discussion threads: https://lists.apache.org/thread.html/r2e62854d1fdcee2627dcfb724c

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

Keep in mind that if it gets moved, it won’t get updated nearly as often (less of an incentive) so I’d say it should be a keep or delete decision. -Ry On Tue, Aug 25, 2020 at 6:05 AM Deng Xiaodong wrote: > Personally I think it does not fit the scope of README very well, and does > > make the

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

Personally I think it does not fit the scope of README very well, and does make the file unnecessarily long. So I would suggest to move it somewhere else too. XD On Tue, Aug 25, 2020 at 10:43 Tomasz Urbaszek wrote: > I agree with Sumit. Regarding opening a PR - Airflow website is also an > >

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

I agree with Sumit. Regarding opening a PR - Airflow website is also an open project :) Tomek On Tue, Aug 25, 2020 at 10:16 AM Sumit Maheshwari wrote: > Not sure, if I've seen such a list with any other open-source project. So I > feel it's a bit awkward to list 100s of different companies and

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

Not sure, if I've seen such a list with any other open-source project. So I feel it's a bit awkward to list 100s of different companies and people names on the main page of an Apache project. Hence, +1 from my side to move the list somewhere else. On Tue, Aug 25, 2020 at 1:35 PM Jarek Potiuk wr

Re: Move "Who uses Airflow?" (to Ecosystem? or new "users" page)?

Yeah - there are some nice properties of that indeed. As I said - no strong opinion, but I wonder what others think about it :) On Tue, Aug 25, 2020 at 2:03 AM Ry Walker wrote: > I do think it’s one of the fun and impressive things about airflow, and an > easy first commit that can lead to more.

Re: [Discuss] Improving AIP process documentation

Looks great Gerard! I think maybe one last pass with Superset's docs. Side comment. I really like that SIP's of superset end up as Github Issues. They even have template for it. Maybe we should do the same? Seems that we are gravitating towards "Github heads-on" recently - Github Actions, Github

Re: Airflow + Hacktoberfest 2020

Cool! QP I will let you know of the plans later this week :) Anyone else willing to help :)? Anyone who would like to help carry forward the "Airflow Summit" torch? J. On Fri, Aug 21, 2020 at 6:48 AM QP Hou wrote: > On Thu, Aug 20, 2020 at 9:44 PM Jarek Potiuk > wrote: > > > > @QP - for the