Wrote this small proposal https://docs.google.com/document/d/1ammZ2iGmGuoXNCtbpLdnj2D23nEYrCaLMbETjim3C0Q/edit?usp=sharing I can add it to Confluence as well if pointed to where should it live (sorry not really familiar enough w Airflow Confluence).
Working to add some more context on existing frameworks that inspired the proposal. Gerard Casas Saez Twitter | Cortex | @casassaez On Feb 5, 2020, 2:36 PM -0700, Dan Davydov <ddavy...@twitter.com.invalid>, wrote: > Ok let's try it, don't know if we're violating some Apache process here but > I guess we'll find out :). > > On Wed, Feb 5, 2020 at 4:22 PM Tomasz Urbaszek <tomasz.urbas...@polidea.com> > wrote: > > > Google docs is good to work out final version that can be published on > > confluence. But that’s only my opinion. > > > > T. > > > > On Wed, 5 Feb 2020 at 22:12, Dan Davydov <ddavy...@twitter.com.invalid> > > wrote: > > > > > Traditionally we've done this in confluence within the AIP although I > > think > > > I would prefer google docs at some point in the future maybe :). I would > > > use confluence though for this. > > > > > > On Wed, Feb 5, 2020 at 3:52 PM Gerard Casas Saez > > > <gcasass...@twitter.com.invalid> wrote: > > > > > > > Happy to drive this. What would be a good place to put this design doc? > > > > Guessing confluence, not sure under what directory though. > > > > > > > > Gerard Casas Saez > > > > Twitter | Cortex | @casassaez > > > > On Feb 4, 2020, 1:18 PM -0700, Jarek Potiuk <jarek.pot...@polidea.com > > > , > > > > wrote: > > > > > +1 short design doc would be cool. > > > > > > > > > > wt., 4 lut 2020, 21:16 użytkownik Tomasz Urbaszek < > > > > > tomasz.urbas...@polidea.com> napisał: > > > > > > > > > > > Do you think we should start with some design doc for that? In this > > > > > > way, we can work out the best solution and allow other to add 2 > > > cents? > > > > > > > > > > > > T. > > > > > > > > > > > > > > > > > > On Tue, Feb 4, 2020 at 8:37 PM Daniel Imberman > > > > > > <daniel.imber...@gmail.com> wrote: > > > > > > > > > > > > > > I think if we’re not breaking any other operators (which I doubt > > we > > > > are) > > > > > > it’s a great 2.0 feature. It would also look great in a “What’s New > > > in > > > > > > Airflow 2.0” announcement ;). > > > > > > > > > > > > > > Docs are always a challenge, but we could set up a google doc and > > > > hack > > > > > > it out in a day or two. > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > via Newton Mail [ > > > > > > > > > > > > > > > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.6&source=email_footer_2 > > > > > > ] > > > > > > > On Tue, Feb 4, 2020 at 2:29 AM, Jarek Potiuk < > > > > jarek.pot...@polidea.com> > > > > > > wrote: > > > > > > > I like the idea, especially the backwards compatibility. > > > > > > > > > > > > > > I would love to understand more about whether it will work (it > > > looks > > > > like > > > > > > > it will) without modifying the 100s of operators we already have. > > > If > > > > so, > > > > > > > this looks like a nice addition to the current way how we define > > > > Dags and > > > > > > > even allows for incremental migration from the "traditional" to > > > > > > > "functional" Dag definition pattern. It does not enforce it but > > it > > > > opens > > > > > > up > > > > > > > new possibilities without changing basic paradigms of Airflow. > > > > > > > > > > > > > > It looks like we could even make it available in 2.0 as there are > > > > hardly > > > > > > > any dependencies and very low risk with introducing such change. > > I > > > > think > > > > > > > the biggest challenge will be to write good documentation and > > > making > > > > sure > > > > > > > that examples are there - but maybe we could even somewhat > > automate > > > > it > > > > > > and > > > > > > > generate some part of the "functional variants" for the examples > > we > > > > have? > > > > > > > > > > > > > > WDYT Dan, others ? > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Tomasz Urbaszek > > > > > > Polidea | Software Engineer > > > > > > > > > > > > M: +48 505 628 493 > > > > > > E: tomasz.urbas...@polidea.com > > > > > > > > > > > > Unique Tech > > > > > > Check out our projects! > > > > > > > > > > > > > > > -- > > > > Tomasz Urbaszek > > Polidea <https://www.polidea.com/> | Software Engineer > > > > M: +48 505 628 493 <+48505628493> > > E: tomasz.urbas...@polidea.com <tomasz.urbasz...@polidea.com> > > > > Unique Tech > > Check out our projects! <https://www.polidea.com/our-work> > >
