Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-04-01 Thread Kaxil Naik
> > We should maybe move this conversation to slack if there’s anything > specific in the AIP that I did not cover. Otherwise, I will submit it for > vote and start breaking down the work in issues (I guess now GitHub > issues?). Yes, now that the Vote has passed, please feel free to start creati

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-03-25 Thread Tomasz Urbaszek
Hi all, Should we proceed with the vote? T. On Fri, Mar 20, 2020 at 1:37 AM Gerard Casas Saez wrote: > Hi everyone, > > Sorry for leaving this unfinished. I was commenting an extension that > Tomasz suggested in the AIP doc, to add fields in the operator that serve > as templated args. This sh

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-03-19 Thread Gerard Casas Saez
Hi everyone, Sorry for leaving this unfinished. I was commenting an extension that Tomasz suggested in the AIP doc, to add fields in the operator that serve as templated args. This should be pretty easy to do if extending templated args. Not sure if doing it extending templated_fields or add a

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-03-03 Thread Evgeny Shulman
Hey Everybody I am happy to contribute to this discussion! It's something that everybody is missing at Airflow. If I am part of a discussion around orchestration framework comparison, "functional DAGs” arguments are the most used ones. We are building a similar approach and would be happy to bas

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-02-25 Thread Gerard Casas Saez
Short update on status after receiving some feedback on Slack and Confluence comments: • Updated description of PythonFunctionalOperator • Added a few clarifying comments on how each component change plays along each other Reached out to Bolke to see if I can get his thoughts on how Lineage and

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-02-24 Thread Jarek Potiuk
Ah yeah... I totally forgot about that :) (shame on me) ... But it does seem appropriate if I came to the same conclusion again looking from another angle :D J. On Mon, Feb 24, 2020 at 6:25 PM Gerard Casas Saez wrote: > Agree, I initially pitched the idea on the lineage thread and was > e

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-02-24 Thread Gerard Casas Saez
Agree, I initially pitched the idea on the lineage thread and was encouraged to pitch it separately. I would love to help figure out how to align this 2 projects better. Bolke - want to set up a call or how should we discuss this better? Would love to hear feedback on my proposal. Gerard Casas

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-02-23 Thread Jarek Potiuk
I like the idea a lot. Good direction. I know we have a few people who are better in functional thinking than me so I think I would love those people to work it out. Happy to listen to the discussions :) One thing that struck me however. I am not sure if that falls in the same camp, but I have a

Re: [DISCUSS] AIP-31: Airflow functional DAG API

2020-02-21 Thread Dan Davydov
Here is the link to the AIP for folk's convenience: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-31%3A+Airflow+functional+DAG+API The proposal and this all looks really good to me :)! I do want to call out to others that it's important we get the interface 95%+ right from the get-go sin

[DISCUSS] AIP-31: Airflow functional DAG API

2020-02-21 Thread Gerard Casas Saez
Hi everyone, Sending a new message to everyone to gather feedback on the AIP-31 about Airflow functional DAG API. This was initially discussed and proposed in   [DISCUSS] Airflow functional DAGs. After leaving open a small doc to iterate on the proposal for a couple weeks, I decided to move forw