A bit of an update: discussing some design aspects w Tomasz in the doc. My plan is to transition this into a Confluence AIP next week (Thursday) and send a separate note to the mailing list. This should allow people to chip in with their thoughts before is more established and iterate on it meanwhile. And also allow for more visibility next week once published on Confluence.
Let me know if this seems appropriate or not :D Gerard Casas Saez Twitter | Cortex | @casassaez On Feb 5, 2020, 2:46 PM -0700, Gerard Casas Saez <gcasass...@twitter.com>, wrote: > Wrote this small proposal > https://docs.google.com/document/d/1ammZ2iGmGuoXNCtbpLdnj2D23nEYrCaLMbETjim3C0Q/edit?usp=sharing > I can add it to Confluence as well if pointed to where should it live (sorry > not really familiar enough w Airflow Confluence). > > Working to add some more context on existing frameworks that inspired the > proposal. > > Gerard Casas Saez > Twitter | Cortex | @casassaez > On Feb 5, 2020, 2:36 PM -0700, Dan Davydov <ddavy...@twitter.com.invalid>, > wrote: > > Ok let's try it, don't know if we're violating some Apache process here but > > I guess we'll find out :). > > > > On Wed, Feb 5, 2020 at 4:22 PM Tomasz Urbaszek <tomasz.urbas...@polidea.com> > > wrote: > > > > > Google docs is good to work out final version that can be published on > > > confluence. But that’s only my opinion. > > > > > > T. > > > > > > On Wed, 5 Feb 2020 at 22:12, Dan Davydov <ddavy...@twitter.com.invalid> > > > wrote: > > > > > > > Traditionally we've done this in confluence within the AIP although I > > > think > > > > I would prefer google docs at some point in the future maybe :). I would > > > > use confluence though for this. > > > > > > > > On Wed, Feb 5, 2020 at 3:52 PM Gerard Casas Saez > > > > <gcasass...@twitter.com.invalid> wrote: > > > > > > > > > Happy to drive this. What would be a good place to put this design > > > > > doc? > > > > > Guessing confluence, not sure under what directory though. > > > > > > > > > > Gerard Casas Saez > > > > > Twitter | Cortex | @casassaez > > > > > On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk <jarek.pot...@polidea.com > > > > , > > > > > wrote: > > > > > > +1 short design doc would be cool. > > > > > > > > > > > > wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek < > > > > > > tomasz.urbas...@polidea.com> napisał: > > > > > > > > > > > > > Do you think we should start with some design doc for that? In > > > > > > > this > > > > > > > way, we can work out the best solution and allow other to add 2 > > > > cents? > > > > > > > > > > > > > > T. > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman > > > > > > > <daniel.imber...@gmail.com> wrote: > > > > > > > > > > > > > > > > I think if we’re not breaking any other operators (which I doubt > > > we > > > > > are) > > > > > > > it’s a great 2.0 feature. It would also look great in a “What’s > > > > > > > New > > > > in > > > > > > > Airflow 2.0” announcement ;). > > > > > > > > > > > > > > > > Docs are always a challenge, but we could set up a google doc > > > > > > > > and > > > > > hack > > > > > > > it out in a day or two. > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > via Newton Mail [ > > > > > > > > > > > > > > > > > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 > > > > > > > ] > > > > > > > > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk < > > > > > jarek.pot...@polidea.com> > > > > > > > wrote: > > > > > > > > I like the idea, especially the backwards compatibility. > > > > > > > > > > > > > > > > I would love to understand more about whether it will work (it > > > > looks > > > > > like > > > > > > > > it will) without modifying the 100s of operators we already > > > > > > > > have. > > > > If > > > > > so, > > > > > > > > this looks like a nice addition to the current way how we define > > > > > Dags and > > > > > > > > even allows for incremental migration from the "traditional" to > > > > > > > > "functional" Dag definition pattern. It does not enforce it but > > > it > > > > > opens > > > > > > > up > > > > > > > > new possibilities without changing basic paradigms of Airflow. > > > > > > > > > > > > > > > > It looks like we could even make it available in 2.0 as there > > > > > > > > are > > > > > hardly > > > > > > > > any dependencies and very low risk with introducing such change. > > > I > > > > > think > > > > > > > > the biggest challenge will be to write good documentation and > > > > making > > > > > sure > > > > > > > > that examples are there - but maybe we could even somewhat > > > automate > > > > > it > > > > > > > and > > > > > > > > generate some part of the "functional variants" for the examples > > > we > > > > > have? > > > > > > > > > > > > > > > > WDYT Dan, others ? > > > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Tomasz Urbaszek > > > > > > > Polidea | Software Engineer > > > > > > > > > > > > > > M: +48 505 628 493 > > > > > > > E: tomasz.urbas...@polidea.com > > > > > > > > > > > > > > Unique Tech > > > > > > > Check out our projects! > > > > > > > > > > > > > > > > > > > -- > > > > > > Tomasz Urbaszek > > > Polidea <https://www.polidea.com/> | Software Engineer > > > > > > M: +48 505 628 493 <+48505628493> > > > E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> > > > > > > Unique Tech > > > Check out our projects! <https://www.polidea.com/our-work> > > >
