I love "Dynamic Airflow Graph" +10 to that Constance. And yes. I fully agree with your "somewhat" acyclic statement. I saw people mentioning "event-driven" workflows to better describe the AI/ML, but that for me is a synonym of "completely unmanageable workflows" (similar to micro-services) - what I was alluding to was exactly this - something like curren DAG with parts of it being potentially cyclic - so we are completely aligned here :). And that's what make it "not DAG - in mathematical sense" - even more reason to depart from it,
J. On Tue, Feb 18, 2025 at 5:23 PM Constance Martineau <consta...@astronomer.io.invalid> wrote: > Hey all, > > I appreciate the discussion about how we position the concept of a DAG in > Airflow. While I agree with updating documentation to de-emphasize the > strict mathematical definition, I do not think we should change the public > interface to use "dag" instead of "DAG" everywhere. It adds > unnecessary churn and breaks consistency across the ecosystem. > > Regarding the broader discussion on whether DAGs in Airflow should remain > acyclic: I acknowledge that some workflows, particularly in ML and AI, > require cyclical behaviour (example: iterative refinement until a condition > is met). However, I think it's important to recognize that even in those > cases, the workflow itself still has an overall start and end. It's not an > unbounded loop but rather a controlled execution process that happens to > have cycles *within* it. > > That being said, I do see value in reinterpreting "DAG" to better align > with how Airflow is actually used, and love shifting away from "Directed > Acyclic Graph" in favour of something more intuitive. However "Data Airflow > Graph" makes it sound like we're defining Airflow itself rather than > describing the structure of workflows within it. > > If we want a new meaning for DAG that resonates with our base while keeping > Airflow's identity clear, I am proposing three alternatives: > > > 1. *Dynamic Airflow Graph*: Ties directly to Airflow while emphasizing > flexibility > 2. *Distributed Automation Graph*: Highlights execution across systems > and automations without tying it strictly to data > 3. *Directed Activity Graph*: Retains the useful meaning of "Directed" > while shifting away from strict acyclic constraints. (My personal > favourite) > > I think any shift in terminology should be driven by what best communicates > Airflow's role workflow automation, rather than just distancing ourselves > from the mathematical definition. > > Curious to hear more thoughts! > > On Tue, Feb 18, 2025 at 10:40 AM Jarek Potiuk <ja...@potiuk.com> wrote: > > > Sorry for the few typos ... My slight dyslexia did not help and > autocorrect > > did not help either this time : dept -> debt, routed -> rooted and a few > > other small typos :) . > > > > On Tue, Feb 18, 2025 at 4:28 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > First of all - yes, agree with Lee and Ash, I think we do not need to > > > change the public interface, I'd also be -0 on that, but docs, internal > > > strings, yes we could. > > > > > > But .... After thinking a bit and looking at the discussion here, where > > > people have concerns, I have an alternative take. And I know this might > > be > > > controversial, and some people might find it confusing, but why don't > we > > > come up with a new acronym for DAG, one that will be more airflow and > > less > > > "mathematics/algorithm". And I have, I think a good reason for it, and > I > > > even have a proposal. Bear with me, > > > > > > DAG => *Data Airflow Graph* > > > > > > As many of us mentioned, DAG and Airflow are almost 100% 1-1 now (with > > > exception of some niche products - you know which one I speak about - > > which > > > are unfortunate enough to use the "Dag" in their name - I will explain > > why > > > this is unfortunate, IMHO). > > > > > > We are lucky to have Airflow for "A" in the acronym and we can make it > > > even more "synonymous" -> Airflow <> DAG with (A) being Airflow. > > > > > > Now, why I think "Directed Acyclic Graph" is unfortunate. > > > > > > Not only because "Directed" and "Acyclic" are mostly not in the > > vocabulary > > > of most of our users - as Ryan, starting this thread explained yes, I > > agree > > > with it, but also I think there is a more important reason - I simply > > > believe in the future our DAGs might not (and likely will not) be > > "acyclic". > > > > > > I know for some it might be herezy and we have not discussed any of > that > > > yet, but more and more I keep on hearing that Machine Learning > Workflows > > of > > > the future are often somewhat cyclic. In our case there is one type of > > > repeat-in-cycle behaviour - we have "task retry on failure" - but this > is > > > only on failure and it basically mens "idempotent" retry - we want to > > redo > > > something and get single result out of that and basically discard the > > > previous result. But machine learning/AI workflows are different. I > had a > > > lot of conversations recently (I even co-organized "AI low-level > > > engineering and hacking" devroom to learn more about those) - and I > keep > > on > > > hearing that "Repeat a sequence of things several times and refine" is > > > precisely what AI / ML workflows will need - especially with Agentic > > > workflows - often we have cooperating agents which **could** be part of > > the > > > same DAG, but currently in Airflow where our DAGs are "Acyclic" we can > > > only express "repeat several time and refine" as a single task that > will > > do > > > such refining. If we would like to split such an "agent" doing their > own > > > work in "cycles" and communicating with other "agents" doing the same - > > > represented as a multi-task-entity, we have no way to express it using > > our > > > current "acyclic" semantics. > > > > > > But - other than our implementation makes that assumption currently - > > > there is nothing fundamental with Airflow being able to run such > > "partially > > > cyclic workflows". We have all the pieces, we would have to change > > > dependency calculation to allow for cycles, update UI to handle cycles > > > representation - and we could in theory quite seamlessly, support > > "cyclic" > > > workflows. > > > > > > I know, I know, that's the first time we hear it and that's huge > change - > > > of course, I do not want to start discussing any details of it, not > > before > > > Airflow 3.0 is out - but I think we should be open that at some point > of > > > time the "Acyclic" property is going to go away, And my personal > > intuition > > > tells me that a year from now we will have acyclic workflows (assuming > > that > > > all the foundational work we do on Airflow 3 will really help in > > decreasing > > > all the technical dept, and will open our minds to new ideas). > > > > > > That's why I think - the sooner we get rid of "Directed Acyclic" - the > > > better. that will - even for us, maintainer free us from being routed > in > > > the acyclic nature of Airflow. > > > > > > How about we change to "Data Airflow Graph" (or any other good acronym > > > which has (A)irflow instead of (A)cyclic). The name is just a proposal. > > > > > > Now.. I go hide somewhere as I believe that might cause a "flamewar" > :), > > > unless on request of Ryan you already sheathed your swords in the > > > discussion (which seems everyone did). > > > > > > But honestly - I would really love what you think of "let's just break > > > with Acyclic NOW" and shape our future by using a different acronym. > > > > > > J. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 18, 2025 at 10:54 AM Ephraim Anierobi < > > > ephraimanier...@gmail.com> wrote: > > > > > >> When I started contributing to Airflow, I had to read up on DAG, and > > now I > > >> know what it means. I wonder if we are about to have users who use the > > >> term > > >> Dag without knowing that there's a DAG. > > >> > > >> Slightly concerned. > > >> > > >> -ephraim > > >> > > >> On Tue, 18 Feb 2025 at 10:26, Wei Lee <weilee...@gmail.com> wrote: > > >> > > >> > For user-facing things (e.g., `from airflow import DAG`), I’m more > > like > > >> > -0. But for documentation, docstring, internal things and etc., we > > >> probably > > >> > could still change most of them? > > >> > > > >> > Best, > > >> > Wei > > >> > > > >> > > On Feb 18, 2025, at 5:10 PM, Ash Berlin-Taylor <a...@apache.org> > > >> wrote: > > >> > > > > >> > > I also don’t personally think it’s worth the pain (not to mention > > >> > backcompat workaround) to rename DAG to Dag, so I’d be -0.5 on that. > > >> > > > > >> > > -ash > > >> > > > > >> > >> On 18 Feb 2025, at 04:40, Daniel Imberman < > > daniel.imber...@gmail.com > > >> > > > >> > wrote: > > >> > >> > > >> > >> I think my biggest concern is a marketing one and not a technical > > >> one. > > >> > >> > > >> > >> As has been mentioned on the thread the terms airflow and dag are > > >> kind > > >> > of > > >> > >> synonymous and I certainly don’t want to give the impression that > > we > > >> are > > >> > >> breaking more than we are breaking. > > >> > >> > > >> > >> I wouldn’t die on this hill, but I’m slightly concerned. > > >> > >> > > >> > >> On Mon, Feb 17, 2025 at 6:00 PM Wei Lee <wei...@apache.org> > wrote: > > >> > >> > > >> > >>> I’m not sure about adding ruff rules here 🤔 I think ruff rules > > are > > >> > best > > >> > >>> suited for user-facing things but not the airflow code base > > itself. > > >> If > > >> > what > > >> > >>> we mean is adding a rule to avoid users using "DAG" *after* we > > >> rename > > >> > it, > > >> > >>> it's definitely a +1000. > > >> > >>> > > >> > >>> I just created GitHub issues for this removing "DAG" > > >> > >>> > > >> > >>> https://github.com/apache/airflow/issues/46842 > > >> > >>> https://github.com/apache/airflow/issues/46843 > > >> > >>> > > >> > >>> One thing I'm not sure about is whether we want to get rid of > > `from > > >> > >>> airflow import DAG` as well. 🤔 > > >> > >>> > > >> > >>> Best, > > >> > >>> Wei > > >> > >>> > > >> > >>> On 2025/02/17 19:50:57 Jarek Potiuk wrote: > > >> > >>>> Hard to say until it's looked at :) > > >> > >>>> > > >> > >>>> On Mon, Feb 17, 2025 at 8:45 PM Aritra Basu < > > >> aritrabasu1...@gmail.com > > >> > > > > >> > >>>> wrote: > > >> > >>>> > > >> > >>>>> I can take it up, it's mostly just a doc update right? Or are > we > > >> > doing > > >> > >>> code > > >> > >>>>> files replacement too? > > >> > >>>>> -- > > >> > >>>>> Regards, > > >> > >>>>> Aritra Basu > > >> > >>>>> > > >> > >>>>> On Tue, 18 Feb 2025, 12:55 am Jarek Potiuk, <ja...@potiuk.com > > > > >> > wrote: > > >> > >>>>> > > >> > >>>>>> Sounds like another +1000 files big PR is coming :) ? Any > > >> volunteers > > >> > >>> to > > >> > >>>>>> make it ? It's fun. > > >> > >>>>>> > > >> > >>>>>> On Mon, Feb 17, 2025 at 6:52 PM Omkar P < > > droiddev5...@gmail.com> > > >> > >>> wrote: > > >> > >>>>>> > > >> > >>>>>>> +1 for ruff rules :) > > >> > >>>>>>> > > >> > >>>>>>> Also would be nice to introduce 'Dag' to replace 'DAG' in > the > > >> > >>> Airflow > > >> > >>>>>>> docs, in line with the new UI changes and to make the > renaming > > >> > >>>>>>> consistent across user-facing pages. > > >> > >>>>>>> > > >> > >>>>>>> Regards, > > >> > >>>>>>> Omkar > > >> > >>>>>>> > > >> > >>>>>>> On Mon, Feb 17, 2025 at 8:12 AM Jarek Potiuk < > > ja...@potiuk.com> > > >> > >>> wrote: > > >> > >>>>>>> > > >> > >>>>>>>> +1 :) . Maybe we could add a ruff rule for that :) > > >> > >>>>>>>> > > >> > >>>>>>>> On Mon, Feb 17, 2025 at 8:27 AM Wei Lee <wei...@apache.org > > > > >> > >>> wrote: > > >> > >>>>>>>> > > >> > >>>>>>>>> It seems that our current conclusion is to use "dag" or > > "Dag" > > >> > >>>>> instead > > >> > >>>>>>> of > > >> > >>>>>>>>> "DAG" whenever possible. Should we replace all "DAG" in > the > > >> > >>>>> codebase > > >> > >>>>>>> with > > >> > >>>>>>>>> "dag" or "Dag"? If it's too late for that (which it might > be > > >> > >>> 🤔), > > >> > >>>>>>> should > > >> > >>>>>>>> we > > >> > >>>>>>>>> at least avoid introducing new "DAG" in the following PRs? > > >> > >>>>>>>>> > > >> > >>>>>>>>> Best, > > >> > >>>>>>>>> Wei > > >> > >>>>>>>>> > > >> > >>>>>>>>> On 2024/10/23 17:16:55 Brent Bovenzi wrote: > > >> > >>>>>>>>>> Here's a PR to use "dag" as a word in the new 3.0 UI: > > >> > >>>>>>>>>> https://github.com/apache/airflow/pull/43325 > > >> > >>>>>>>>>> Let me know if that's the direction we want to go. > > >> > >>>>>>>>>> > > >> > >>>>>>>>>> On Wed, Oct 23, 2024 at 11:06 AM Bishundeo, Rajeshwar > > >> > >>>>>>>>>> <rbish...@amazon.com.invalid> wrote: > > >> > >>>>>>>>>> > > >> > >>>>>>>>>>> I think Brent & Daniel summarized it best, "dag" is > > >> > >>> synonymous > > >> > >>>>>> with > > >> > >>>>>>>>>>> workflows in Airflow through the way we talk and explain > > >> > >>> what > > >> > >>>>>>> Airflow > > >> > >>>>>>>>> is > > >> > >>>>>>>>>>> all about. Although folks would ask, I don’t ever use > the > > >> > >>>>>>>> mathematical > > >> > >>>>>>>>>>> definition of DAG. > > >> > >>>>>>>>>>> It will be challenging and possibly confusing for many > > >> > >>> users > > >> > >>>>>> making > > >> > >>>>>>>>> such a > > >> > >>>>>>>>>>> change - I would rather direct that energy to appending > > the > > >> > >>>>>> Oxford > > >> > >>>>>>>>>>> definition of "dag" to include a reference to workflows > in > > >> > >>>>>> Airflow. > > >> > >>>>>>>> __ > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> -- Rajesh > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> On 2024-10-23, 3:37 AM, "Jarek Potiuk" < > ja...@potiuk.com > > >> > >>>>>> <mailto: > > >> > >>>>>>>>>>> ja...@potiuk.com>> wrote: > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> CAUTION: This email originated from outside of the > > >> > >>>>> organization. > > >> > >>>>>> Do > > >> > >>>>>>>> not > > >> > >>>>>>>>>>> click links or open attachments unless you can confirm > the > > >> > >>>>> sender > > >> > >>>>>>> and > > >> > >>>>>>>>> know > > >> > >>>>>>>>>>> the content is safe. > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient d’un > > >> > >>>>> expéditeur > > >> > >>>>>>>>> externe. > > >> > >>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce > jointe > > >> > >>> si > > >> > >>>>> vous > > >> > >>>>>>> ne > > >> > >>>>>>>>> pouvez > > >> > >>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si vous > n’êtes > > >> > >>> pas > > >> > >>>>>>>> certain > > >> > >>>>>>>>> que > > >> > >>>>>>>>>>> le contenu ne présente aucun risque. > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> From Guido's post: > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> "Around this time the renaming seems to have been > > renamed". > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> Naming is hard. > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> J. > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> On Wed, Oct 23, 2024 at 7:11 AM Omkar P < > > >> > >>>>> droiddev5...@gmail.com > > >> > >>>>>>>>> <mailto: > > >> > >>>>>>>>>>> droiddev5...@gmail.com>> wrote: > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>> > > >> > >>>>>>>>>>>> Yes, completely agree with above comments, dag is an > > >> > >>> Airflow > > >> > >>>>>> term > > >> > >>>>>>>> now > > >> > >>>>>>>>>>>> rather > > >> > >>>>>>>>>>>> than just a "directed acyclic graph". Believe it or > not, > > >> > >>> I've > > >> > >>>>>>>> worked > > >> > >>>>>>>>> with > > >> > >>>>>>>>>>>> work > > >> > >>>>>>>>>>>> folks who've been using dags in Airflow for years now > > >> > >>> and are > > >> > >>>>>> pro > > >> > >>>>>>>>> devs > > >> > >>>>>>>>>>> but > > >> > >>>>>>>>>>>> have trouble remembering the full form of a DAG! For > > >> > >>> them, > > >> > >>>>> dag > > >> > >>>>>> = > > >> > >>>>>>>>> Airflow. > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>>>>> workflow, pipeline, flow are surely better terms but > > >> > >>> will be > > >> > >>>>> a > > >> > >>>>>>>> major > > >> > >>>>>>>>>>>> behavior > > >> > >>>>>>>>>>>> change for the developer community, so we'll need a > solid > > >> > >>>>> plan > > >> > >>>>>> on > > >> > >>>>>>>>> how to > > >> > >>>>>>>>>>>> introduce it, when we do. > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>>>>> While we discuss this, I'd like to share about the > Great > > >> > >>>>>> Renaming > > >> > >>>>>>>> in > > >> > >>>>>>>>> core > > >> > >>>>>>>>>>>> Python started by Guido back in 2009. We could probably > > >> > >>> get > > >> > >>>>>> some > > >> > >>>>>>>>>>> learnings > > >> > >>>>>>>>>>>> from there? Who knows! > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>>>>> Guido's blog (2009): > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>> > > >> > >>>>>>> > > >> > >>>>> > > >> > >>> > > >> > > > >> > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > >> > >>>>>>>>>>> < > > >> > >>>>>>>>> > > >> > >>>>>>> > > >> > >>>>> > > >> > >>> > > >> > > > >> > > https://python-history.blogspot.com/2009/03/great-or-grand-renaming.html > > >> > >>>>>>>>> > > >> > >>>>>>>>>>>> Follow-up discussion (2024): > > >> > >>>>>>>>>>>> > > >> > >>>>>> > > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > >> > >>>>>>> < > > >> > >>>>>>>>>>> > > >> > >>>>> > https://discuss.python.org/t/finishing-the-great-renaming/54082 > > > > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>>>>> Cheers, > > >> > >>>>>>>>>>>> Omkar > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>>>>> On Wed, Oct 23, 2024 at 7:10 AM Wei Lee < > > >> > >>> weilee...@gmail.com > > >> > >>>>>>>>> <mailto: > > >> > >>>>>>>>>>> weilee...@gmail.com>> wrote: > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>>>>>> I think we should probably just accept it as an > airflow > > >> > >>>>> term. > > >> > >>>>>>> At > > >> > >>>>>>>>> least, > > >> > >>>>>>>>>>>>> that’s how I understood it when I first used Airflow. > I > > >> > >>>>> feel > > >> > >>>>>>>>> renaming > > >> > >>>>>>>>>>> it > > >> > >>>>>>>>>>>> at > > >> > >>>>>>>>>>>>> this stage would require considerable effort from > > >> > >>>>> maintainers > > >> > >>>>>>> and > > >> > >>>>>>>>>>>> existing > > >> > >>>>>>>>>>>>> users without providing equivalent benefits. > > >> > >>>>>>>>>>>>> > > >> > >>>>>>>>>>>>> Best, > > >> > >>>>>>>>>>>>> Wei > > >> > >>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>> On Oct 23, 2024, at 8:01 AM, Kaxil Naik < > > >> > >>>>>> kaxiln...@gmail.com > > >> > >>>>>>>>>>> <mailto:kaxiln...@gmail.com>> wrote: > > >> > >>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>> Same agreed with Brent & Daniel -- maybe we re-kindle > > >> > >>>>> this > > >> > >>>>>>>>> discussion > > >> > >>>>>>>>>>>> for > > >> > >>>>>>>>>>>>>> Airflow 4 :) -- but right now it will cause too much > > >> > >>>>>>> disruption > > >> > >>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>> On Tue, 22 Oct 2024 at 21:27, Constance Martineau > > >> > >>>>>>>>>>>>>> <consta...@astronomer.io.inva <mailto: > > >> > >>>>>>>>> consta...@astronomer.io.inva>lid> > > >> > >>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>> In my experience, when you ask those with Airflow > > >> > >>>>>> experience > > >> > >>>>>>>>> what a > > >> > >>>>>>>>>>>> dag > > >> > >>>>>>>>>>>>> is, > > >> > >>>>>>>>>>>>>>> they'll start talking about workflow attributes - > > >> > >>> stuff > > >> > >>>>>> like > > >> > >>>>>>>>> dags > > >> > >>>>>>>>>>>> being > > >> > >>>>>>>>>>>>> a > > >> > >>>>>>>>>>>>>>> series of steps or tasks with owners. The structure > > >> > >>>>>> doesn't > > >> > >>>>>>>>> come up. > > >> > >>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>> Echo-ing others, at this point, my vote is to > > >> > >>> embrace > > >> > >>>>> the > > >> > >>>>>>> name > > >> > >>>>>>>>> and > > >> > >>>>>>>>>>>>>>> de-emphasize the mathematical structure aspect. > > >> > >>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:47 PM Vikram Koka > > >> > >>>>>>>>>>>>> <vik...@astronomer.io.inva <mailto: > > >> > >>>>> vik...@astronomer.io.inva > > >> > >>>>>>>>> lid> > > >> > >>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>> It's an interesting discussion and I remember > > >> > >>>>> struggling > > >> > >>>>>>> with > > >> > >>>>>>>>> this > > >> > >>>>>>>>>>>>> when I > > >> > >>>>>>>>>>>>>>>> started working with Airflow. > > >> > >>>>>>>>>>>>>>>> But, I also agree with the viewpoint of it being an > > >> > >>>>>>>> established > > >> > >>>>>>>>>>>> concept > > >> > >>>>>>>>>>>>>>> now > > >> > >>>>>>>>>>>>>>>> regardless of the origin. > > >> > >>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>> I am personally leaning towards the perspective > > >> > >>> best > > >> > >>>>>>>> expressed > > >> > >>>>>>>>> by > > >> > >>>>>>>>>>>>> Daniel > > >> > >>>>>>>>>>>>>>>> Standish and Brent of using Dag as a word, rather > > >> > >>> than > > >> > >>>>>> the > > >> > >>>>>>>>> computer > > >> > >>>>>>>>>>>>>>> science > > >> > >>>>>>>>>>>>>>>> concept. > > >> > >>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>> Best regards, > > >> > >>>>>>>>>>>>>>>> Vikram > > >> > >>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 9:46 AM Oliveira, Niko > > >> > >>>>>>>>>>>>>>> <oniko...@amazon.com.inva <mailto: > > >> > >>>>>> oniko...@amazon.com.inva > > >> > >>>>>>>>> lid > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> I agree with the general sentiment of: You're > > >> > >>> right > > >> > >>>>>> Ryan, > > >> > >>>>>>>> DAG > > >> > >>>>>>>>>>> isn't > > >> > >>>>>>>>>>>>>>> great > > >> > >>>>>>>>>>>>>>>>> and I'd rather workflow, but changing it will > > >> > >>> cause > > >> > >>>>> much > > >> > >>>>>>>> more > > >> > >>>>>>>>>>>> wreckage > > >> > >>>>>>>>>>>>>>>> than > > >> > >>>>>>>>>>>>>>>>> it solves. > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> Also agree with the idea to just move away from > > >> > >>>>> defining > > >> > >>>>>>>> DAG. > > >> > >>>>>>>>> I > > >> > >>>>>>>>>>>> think > > >> > >>>>>>>>>>>>>>>>> we've been naturally doing that as a community > > >> > >>> for a > > >> > >>>>>> while > > >> > >>>>>>>> now > > >> > >>>>>>>>>>>> anyway, > > >> > >>>>>>>>>>>>>>> so > > >> > >>>>>>>>>>>>>>>>> that feels like a natural step. > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> Cheers, > > >> > >>>>>>>>>>>>>>>>> Niko > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> ________________________________ > > >> > >>>>>>>>>>>>>>>>> From: Ash Berlin-Taylor <a...@apache.org <mailto: > > >> > >>>>>>>>> a...@apache.org>> > > >> > >>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 9:06:39 AM > > >> > >>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > >> > >>>>>> dev@airflow.apache.org > > >> > >>>>>>>> > > >> > >>>>>>>>>>>>>>>>> Subject: RE: [EXT] Airflow should deprecate the > > >> > >>> term > > >> > >>>>>> "DAG" > > >> > >>>>>>>>> for end > > >> > >>>>>>>>>>>>>>> users > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> CAUTION: This email originated from outside of the > > >> > >>>>>>>>> organization. > > >> > >>>>>>>>>>> Do > > >> > >>>>>>>>>>>>> not > > >> > >>>>>>>>>>>>>>>>> click links or open attachments unless you can > > >> > >>> confirm > > >> > >>>>>> the > > >> > >>>>>>>>> sender > > >> > >>>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>> know > > >> > >>>>>>>>>>>>>>>>> the content is safe. > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> AVERTISSEMENT: Ce courrier électronique provient > > >> > >>> d’un > > >> > >>>>>>>>> expéditeur > > >> > >>>>>>>>>>>>>>> externe. > > >> > >>>>>>>>>>>>>>>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce > > >> > >>>>>> jointe > > >> > >>>>>>> si > > >> > >>>>>>>>> vous > > >> > >>>>>>>>>>> ne > > >> > >>>>>>>>>>>>>>>> pouvez > > >> > >>>>>>>>>>>>>>>>> pas confirmer l’identité de l’expéditeur et si > > >> > >>> vous > > >> > >>>>>> n’êtes > > >> > >>>>>>>> pas > > >> > >>>>>>>>>>>> certain > > >> > >>>>>>>>>>>>>>>> que > > >> > >>>>>>>>>>>>>>>>> le contenu ne présente aucun risque. > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> Best argument in favour of keeping “dags” as a > > >> > >>> term — > > >> > >>>>>>>> getting > > >> > >>>>>>>>> to > > >> > >>>>>>>>>>>>> re-use > > >> > >>>>>>>>>>>>>>>>> puns like https://i.imgflip.com/1xhtwh.jpg < > > >> > >>>>>>>>>>> https://i.imgflip.com/1xhtwh.jpg> > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> In all seriousness: I don’t mind either way, both > > >> > >>>>> sides > > >> > >>>>>>> have > > >> > >>>>>>>>> good > > >> > >>>>>>>>>>>>>>> reasons > > >> > >>>>>>>>>>>>>>>>> presented. > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>> -a > > >> > >>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 17:03, Daniel Standish > > >> > >>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > > >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>LID> wrote: > > >> > >>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>> Yeah just say, when asked where the name comes > > >> > >>> from, > > >> > >>>>>>> "well, > > >> > >>>>>>>>> no > > >> > >>>>>>>>>>> one > > >> > >>>>>>>>>>>>>>>>> actually > > >> > >>>>>>>>>>>>>>>>>> knows but..." and then make something up. > > >> > >>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 8:31 AM Jarek Potiuk < > > >> > >>>>>>>>> ja...@potiuk.com > > >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > > >> > >>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>> Just to clarify - "directed acyclic graph" is > > >> > >>> the > > >> > >>>>>>>>>>> tongue-twister, > > >> > >>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:29 PM Jarek Potiuk < > > >> > >>>>>>>>> ja...@potiuk.com > > >> > >>>>>>>>>>> <mailto:ja...@potiuk.com>> > > >> > >>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>> I like what both Daniel and Brent wrote. I > > >> > >>> would > > >> > >>>>> very > > >> > >>>>>>>> much > > >> > >>>>>>>>> want > > >> > >>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>> be > > >> > >>>>>>>>>>>>>>>>>>> able > > >> > >>>>>>>>>>>>>>>>>>>> to say just "dag" without explaining it > > >> > >>> further. > > >> > >>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>> For me every time I explain "DAG" at a talk > > >> > >>> it's a > > >> > >>>>>>>>>>>> tongue-twister, > > >> > >>>>>>>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>>> I > > >> > >>>>>>>>>>>>>>>>>>>> almost stutter on trying to recall how to > > >> > >>> pronounce > > >> > >>>>>> it > > >> > >>>>>>>>>>> properly. > > >> > >>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>> J. > > >> > >>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 5:27 PM Brent Bovenzi > > >> > >>>>>>>>>>>>>>>>>>> <br...@astronomer.io.inva <mailto: > > >> > >>>>>>>> br...@astronomer.io.inva > > >> > >>>>>>>>>> lid> > > >> > >>>>>>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>> I remember we explored renaming "DAG" when > > >> > >>>>> starting > > >> > >>>>>> on > > >> > >>>>>>>>> AIP-38 > > >> > >>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>> modernize > > >> > >>>>>>>>>>>>>>>>>>>>> the UI. Both "pipeline" or "workflow" are more > > >> > >>>>>>>>> descriptive of > > >> > >>>>>>>>>>>> what > > >> > >>>>>>>>>>>>>>>> one > > >> > >>>>>>>>>>>>>>>>>>> is > > >> > >>>>>>>>>>>>>>>>>>>>> actually doing while Directed Acyclic Graph > > >> > >>> is an > > >> > >>>>>>>>>>> implementation > > >> > >>>>>>>>>>>>>>>>> detail. > > >> > >>>>>>>>>>>>>>>>>>>>> But I agree with Daniel Standish, at this > > >> > >>> point > > >> > >>>>>> "DAG" > > >> > >>>>>>>> has > > >> > >>>>>>>>>>> become > > >> > >>>>>>>>>>>>>>>> "dag" > > >> > >>>>>>>>>>>>>>>>>>> , a > > >> > >>>>>>>>>>>>>>>>>>>>> word in its own right. > > >> > >>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>> Examples for "dag" are abound in community > > >> > >>>>>> discussion, > > >> > >>>>>>>>> Airflow > > >> > >>>>>>>>>>>>>>>> Summit > > >> > >>>>>>>>>>>>>>>>>>>>> talks, documentation and even in the UI. Let's > > >> > >>>>>> embrace > > >> > >>>>>>>>> "dag". > > >> > >>>>>>>>>>> A > > >> > >>>>>>>>>>>>>>> user > > >> > >>>>>>>>>>>>>>>>>>> just > > >> > >>>>>>>>>>>>>>>>>>>>> needs to learn one new word vs the technical > > >> > >>>>> concept > > >> > >>>>>>>>> behind > > >> > >>>>>>>>>>> that > > >> > >>>>>>>>>>>>>>>>> word. I > > >> > >>>>>>>>>>>>>>>>>>>>> think that is much less effort than > > >> > >>> refactoring so > > >> > >>>>>>> much > > >> > >>>>>>>>> code, > > >> > >>>>>>>>>>>>>>>>>>>>> documentation, blog posts, stack overflow > > >> > >>>>> questions, > > >> > >>>>>>>> etc. > > >> > >>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:51 AM Daniel > > >> > >>> Standish > > >> > >>>>>>>>>>>>>>>>>>>>> <daniel.stand...@astronomer.io.inva <mailto: > > >> > >>>>>>>>>>> daniel.stand...@astronomer.io.inva>lid> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>> I am skeptical. Seems like introducing a lot > > >> > >>> of > > >> > >>>>>> pain > > >> > >>>>>>>> for > > >> > >>>>>>>>>>>>>>>>> questionable > > >> > >>>>>>>>>>>>>>>>>>>>>> benefit. But, I am def sympathetic to the > > >> > >>> idea. I > > >> > >>>>>>> agree > > >> > >>>>>>>>> the > > >> > >>>>>>>>>>>>>>>>>>>>> association > > >> > >>>>>>>>>>>>>>>>>>>>>> with "directed acyclic graph" is not helpful. > > >> > >>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>> And along those lines, I offer here some less > > >> > >>>>>>> invasive > > >> > >>>>>>>>>>>>>>> mitigations. > > >> > >>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>> One thing we can do no matter what is to > > >> > >>>>>> de-emphasize > > >> > >>>>>>>> the > > >> > >>>>>>>>>>> math > > >> > >>>>>>>>>>>>>>> nerd > > >> > >>>>>>>>>>>>>>>>>>>>> origins > > >> > >>>>>>>>>>>>>>>>>>>>>> of the name. That is to say, in docs / > > >> > >>> website / > > >> > >>>>>> etc, > > >> > >>>>>>>>> *never > > >> > >>>>>>>>>>>>>>>> define* > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow's "dag" concept as a directed acyclic > > >> > >>>>>> graph. > > >> > >>>>>>>>> Always > > >> > >>>>>>>>>>>>>>> define > > >> > >>>>>>>>>>>>>>>>> it > > >> > >>>>>>>>>>>>>>>>>>>>> as a > > >> > >>>>>>>>>>>>>>>>>>>>>> pipeline, collection of tasks, workflow etc. > > >> > >>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>> The "directed acyclic graph" part of it is > > >> > >>> like a > > >> > >>>>>>>>> historical > > >> > >>>>>>>>>>>>>>>>> footnote, > > >> > >>>>>>>>>>>>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>>>>>>>> we could make one mention of it somewhere > > >> > >>> hidden. > > >> > >>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>> We could also start using lowercase in the > > >> > >>> docs > > >> > >>>>> in > > >> > >>>>>>>>> general > > >> > >>>>>>>>>>> e.g. > > >> > >>>>>>>>>>>>>>>>>>> writing > > >> > >>>>>>>>>>>>>>>>>>>>>> "dag" / "dags" instead of writing "DAG" / > > >> > >>> "DAGs" > > >> > >>>>>> etc. > > >> > >>>>>>>> The > > >> > >>>>>>>>>>>> upper > > >> > >>>>>>>>>>>>>>>> case > > >> > >>>>>>>>>>>>>>>>>>>>> part > > >> > >>>>>>>>>>>>>>>>>>>>>> of it makes it look like an acronym; but > > >> > >>> "dag" in > > >> > >>>>>>>>> airlfow is > > >> > >>>>>>>>>>>> just > > >> > >>>>>>>>>>>>>>>> an > > >> > >>>>>>>>>>>>>>>>>>>>>> airflow concept and the association with > > >> > >>> "DAGs" > > >> > >>>>> is > > >> > >>>>>>> not > > >> > >>>>>>>>> really > > >> > >>>>>>>>>>>>>>>>>>> unhelpful. > > >> > >>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>> In other words embrace that "dag" in airflow > > >> > >>> is > > >> > >>>>> its > > >> > >>>>>>> own > > >> > >>>>>>>>>>> thing, > > >> > >>>>>>>>>>>> is > > >> > >>>>>>>>>>>>>>>>>>>>>> *not* strictly > > >> > >>>>>>>>>>>>>>>>>>>>>> speaking a directed acyclic graph (which > > >> > >>> nobody > > >> > >>>>>> knows > > >> > >>>>>>>>> about > > >> > >>>>>>>>>>>>>>>> anyway), > > >> > >>>>>>>>>>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>>>>>>>> tell them what it is in simple terms that > > >> > >>> normal > > >> > >>>>>>> people > > >> > >>>>>>>>>>>>>>> understand. > > >> > >>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 7:27 AM Jarek Potiuk > > >> > >>> < > > >> > >>>>>>>>>>> ja...@potiuk.com <mailto:ja...@potiuk.com> > > >> > >>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>> DAG is so embedded into what we do that it > > >> > >>> will > > >> > >>>>> be > > >> > >>>>>>>>> extremely > > >> > >>>>>>>>>>>>>>>>>>>>> difficult to > > >> > >>>>>>>>>>>>>>>>>>>>>>> get rid of it completely. Also I think it > > >> > >>> will > > >> > >>>>>> make > > >> > >>>>>>> a > > >> > >>>>>>>>> lot of > > >> > >>>>>>>>>>>>>>>>>>> "google" > > >> > >>>>>>>>>>>>>>>>>>>>>>> searches and "stack overflow" searches not > > >> > >>>>> finding > > >> > >>>>>>> the > > >> > >>>>>>>>> right > > >> > >>>>>>>>>>>>>>>>>>> answers. > > >> > >>>>>>>>>>>>>>>>>>>>>> This > > >> > >>>>>>>>>>>>>>>>>>>>>>> is one of the strengths of Airflow - > > >> > >>> besides the > > >> > >>>>>>>>> community > > >> > >>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>> ideas > > >> > >>>>>>>>>>>>>>>>>>>>> that > > >> > >>>>>>>>>>>>>>>>>>>>>>> Bernd mentioned - is the vast number of > > >> > >>>>> examples, > > >> > >>>>>>>>> problems > > >> > >>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>>>>>>> solutions > > >> > >>>>>>>>>>>>>>>>>>>>>>> you can so easily find (and we have to > > >> > >>> remember > > >> > >>>>>> that > > >> > >>>>>>>>> all the > > >> > >>>>>>>>>>>> AI > > >> > >>>>>>>>>>>>>>>>>>>>> trained > > >> > >>>>>>>>>>>>>>>>>>>>>> on > > >> > >>>>>>>>>>>>>>>>>>>>>>> past data will be also rather poorly > > >> > >>> matching > > >> > >>>>>>> queries > > >> > >>>>>>>> of > > >> > >>>>>>>>>>>> people. > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>> I am not too attached to DAG. I could easily > > >> > >>>>>> switch. > > >> > >>>>>>>>> And if > > >> > >>>>>>>>>>> we > > >> > >>>>>>>>>>>>>>> do > > >> > >>>>>>>>>>>>>>>> - > > >> > >>>>>>>>>>>>>>>>>>> I > > >> > >>>>>>>>>>>>>>>>>>>>>>> would be for using workflow or pipeline > > >> > >>> instead > > >> > >>>>> of > > >> > >>>>>>>>> `dag` if > > >> > >>>>>>>>>>>> not > > >> > >>>>>>>>>>>>>>>> the > > >> > >>>>>>>>>>>>>>>>>>>>> above > > >> > >>>>>>>>>>>>>>>>>>>>>>> reason, but I think I am here with Igor > > >> > >>> that it > > >> > >>>>>>> might > > >> > >>>>>>>>> cause > > >> > >>>>>>>>>>>> more > > >> > >>>>>>>>>>>>>>>>>>>>> problems > > >> > >>>>>>>>>>>>>>>>>>>>>>> than it solves. > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>> But I am not 100% against - if others will > > >> > >>> think > > >> > >>>>>>> it's > > >> > >>>>>>>> a > > >> > >>>>>>>>> good > > >> > >>>>>>>>>>>>>>>> idea, I > > >> > >>>>>>>>>>>>>>>>>>>>> am > > >> > >>>>>>>>>>>>>>>>>>>>>> ok > > >> > >>>>>>>>>>>>>>>>>>>>>>> with it. > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>> J, > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 3:12 PM Abhishek > > >> > >>> Bhakat > > >> > >>>>>>>>>>>>>>>>>>>>>>> <abhishek.bha...@astronomer.io.inva > > >> > >>> <mailto: > > >> > >>>>>>>>>>> abhishek.bha...@astronomer.io.inva>lid> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Agreed that the word DAG makes very less > > >> > >>> sense > > >> > >>>>> to > > >> > >>>>>>>>> someone > > >> > >>>>>>>>>>> new > > >> > >>>>>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>> workflow > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestration. But it does also show the > > >> > >>> nature > > >> > >>>>>> of > > >> > >>>>>>>>> being > > >> > >>>>>>>>>>>>>>> acyclic. > > >> > >>>>>>>>>>>>>>>>>>>>> Sure, > > >> > >>>>>>>>>>>>>>>>>>>>>>> as > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Bas mentioned, there are ways to > > >> > >>> workaround it. > > >> > >>>>>>>> Still, > > >> > >>>>>>>>> in > > >> > >>>>>>>>>>> my > > >> > >>>>>>>>>>>>>>>>>>>>> opinion, > > >> > >>>>>>>>>>>>>>>>>>>>>>> there > > >> > >>>>>>>>>>>>>>>>>>>>>>>> is generally no need for cyclic behavior in > > >> > >>>>>>> workflow > > >> > >>>>>>>>>>>>>>>>>>> orchestration. > > >> > >>>>>>>>>>>>>>>>>>>>>> Most > > >> > >>>>>>>>>>>>>>>>>>>>>>>> (*if > > >> > >>>>>>>>>>>>>>>>>>>>>>>> not all*) cases can be in some way can be > > >> > >>>>> covered > > >> > >>>>>>>>> using an > > >> > >>>>>>>>>>>>>>>> acyclic > > >> > >>>>>>>>>>>>>>>>>>>>>> manner > > >> > >>>>>>>>>>>>>>>>>>>>>>>> with multiple runs. Hence, the > > >> > >>> idempotency. So > > >> > >>>>> I > > >> > >>>>>>>> would > > >> > >>>>>>>>> want > > >> > >>>>>>>>>>>> the > > >> > >>>>>>>>>>>>>>>>>>>>>> "acyclic" > > >> > >>>>>>>>>>>>>>>>>>>>>>>> word to stick. > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Regards, > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Avi > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 12:41 PM < > > >> > >>>>>>>>>>> bernd.stroe...@kosakya.de <mailto: > > >> > >>> bernd.stroe...@kosakya.de>> > > >> > >>>>>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Brilliant, I am on the way to become an > > >> > >>>>> Airflow > > >> > >>>>>>> Fan; > > >> > >>>>>>>>> so > > >> > >>>>>>>>>>> many > > >> > >>>>>>>>>>>>>>> new > > >> > >>>>>>>>>>>>>>>>>>>>>> ideas. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> The Term DAG is misleading; it should be > > >> > >>>>>> replaced > > >> > >>>>>>> by > > >> > >>>>>>>>> the > > >> > >>>>>>>>>>>> more > > >> > >>>>>>>>>>>>>>>>>>>>> general > > >> > >>>>>>>>>>>>>>>>>>>>>>>> Term > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Airflow (Workflow) Graph (AFG) or Airflow > > >> > >>>>>> (Petri) > > >> > >>>>>>>> Net > > >> > >>>>>>>>>>> (AFN) > > >> > >>>>>>>>>>>>>>>>>>> (maybe > > >> > >>>>>>>>>>>>>>>>>>>>>>>> without > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> a direction); > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> and ... these Graphs should be stored in a > > >> > >>>>> Graph > > >> > >>>>>>>>> Database. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Every Node or Sup-Graph of an Airflow > > >> > >>> Graph > > >> > >>>>>> (AFG) > > >> > >>>>>>>>> might be > > >> > >>>>>>>>>>>>>>>>>>>>> assigned > > >> > >>>>>>>>>>>>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>>> an > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> executable (Python-, Rust-, ... ) member > > >> > >>> of a > > >> > >>>>>>>> library. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> A running Graph might have a different > > >> > >>>>> structure > > >> > >>>>>>>> than > > >> > >>>>>>>>> a > > >> > >>>>>>>>>>>>>>>>>>>>> configuration > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Graph. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Forget that if you think it's bullshit. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Best Regards > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Bernd Ströhle > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> M: +49 171 5357916 > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> E: bernd.stroe...@gmail.com <mailto: > > >> > >>>>>>>>>>> bernd.stroe...@gmail.com> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> -----Original Message----- > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> From: Igor Kholopov > > >> > >>> <ikholo...@google.com.inva > > >> > >>>>>>>>> <mailto: > > >> > >>>>>>>>>>> ikholo...@google.com.inva>LID> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Sent: Tuesday, October 22, 2024 12:02 PM > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> To: dev@airflow.apache.org <mailto: > > >> > >>>>>>>>> dev@airflow.apache.org > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Subject: Re: Airflow should deprecate the > > >> > >>> term > > >> > >>>>>>> "DAG" > > >> > >>>>>>>>> for > > >> > >>>>>>>>>>> end > > >> > >>>>>>>>>>>>>>>>>>> users > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Even though the term "DAG" is clearly > > >> > >>>>>> suboptimal, > > >> > >>>>>>> it > > >> > >>>>>>>>> is > > >> > >>>>>>>>>>> part > > >> > >>>>>>>>>>>>>>> of > > >> > >>>>>>>>>>>>>>>>>>>>>> Airflow > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> DAG definition interface at so many > > >> > >>> levels, > > >> > >>>>> that > > >> > >>>>>>> any > > >> > >>>>>>>>>>> attempt > > >> > >>>>>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>> change > > >> > >>>>>>>>>>>>>>>>>>>>>>> it > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> will only introduce more chaos, not > > >> > >>> reduce it. > > >> > >>>>>> The > > >> > >>>>>>>>> only > > >> > >>>>>>>>>>>> thing > > >> > >>>>>>>>>>>>>>>>>>>>> that is > > >> > >>>>>>>>>>>>>>>>>>>>>>>> worse > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> than a poorly chosen name in the code is > > >> > >>> when > > >> > >>>>>>> there > > >> > >>>>>>>>> are > > >> > >>>>>>>>>>> two > > >> > >>>>>>>>>>>>>>> ways > > >> > >>>>>>>>>>>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>>>> define > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> the same thing. Countless articles and > > >> > >>>>> tutorials > > >> > >>>>>>>> will > > >> > >>>>>>>>>>>> suddenly > > >> > >>>>>>>>>>>>>>>>>>>>> become > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> confusing as they all refer to workflows > > >> > >>> as > > >> > >>>>>>> "DAG"s. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> We are already at risk of scaring the > > >> > >>> users > > >> > >>>>> away > > >> > >>>>>>>> with > > >> > >>>>>>>>> a > > >> > >>>>>>>>>>>> number > > >> > >>>>>>>>>>>>>>>>>>> of > > >> > >>>>>>>>>>>>>>>>>>>>>>>> breaking > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> changes in Airflow 3, promising even more > > >> > >>>>>> breaking > > >> > >>>>>>>>> changes > > >> > >>>>>>>>>>>> for > > >> > >>>>>>>>>>>>>>>>>>> the > > >> > >>>>>>>>>>>>>>>>>>>>>> most > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> basic things is not something that people > > >> > >>> are > > >> > >>>>>>>> looking > > >> > >>>>>>>>> for. > > >> > >>>>>>>>>>>>>>>>>>>>> Attempting > > >> > >>>>>>>>>>>>>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> change the fundamental terms will be > > >> > >>>>> interpreted > > >> > >>>>>>> as > > >> > >>>>>>>> an > > >> > >>>>>>>>>>> even > > >> > >>>>>>>>>>>>>>>>>>>>> stronger > > >> > >>>>>>>>>>>>>>>>>>>>>>>> signal > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> of project immaturity. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Given that, I oppose the idea of changing > > >> > >>> the > > >> > >>>>>> term > > >> > >>>>>>>> in > > >> > >>>>>>>>> the > > >> > >>>>>>>>>>>> long > > >> > >>>>>>>>>>>>>>>>>>>>> run. I > > >> > >>>>>>>>>>>>>>>>>>>>>>>> even > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> stricter oppose the idea of deprecating > > >> > >>> it in > > >> > >>>>>> the > > >> > >>>>>>>> DAG > > >> > >>>>>>>>>>>>>>> definition > > >> > >>>>>>>>>>>>>>>>>>>>>>>> interface. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> We better put our time and efforts in > > >> > >>> other > > >> > >>>>>> places > > >> > >>>>>>>> in > > >> > >>>>>>>>>>>> Airflow, > > >> > >>>>>>>>>>>>>>>>>>> of > > >> > >>>>>>>>>>>>>>>>>>>>>> which > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> there are plenty. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Kind regards, > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> Igor > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 10:36 AM Bas > > >> > >>> Harenslak > > >> > >>>>>>>>>>>>>>>>>>>>>>> <b...@astronomer.io.inva <mailto: > > >> > >>>>>>>> b...@astronomer.io.inva > > >> > >>>>>>>>>> lid > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Couple of thoughts: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 1. The boundaries/properties of “DAG” > > >> > >>> have > > >> > >>>>>>> already > > >> > >>>>>>>>> faded > > >> > >>>>>>>>>>>> over > > >> > >>>>>>>>>>>>>>>>>>>>> time, > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for example there are now several ways to > > >> > >>>>>> create > > >> > >>>>>>>>> cyclic > > >> > >>>>>>>>>>>>>>>>>>> graphs, > > >> > >>>>>>>>>>>>>>>>>>>>>> e.g. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using the @continuous schedule. I imagine > > >> > >>>>> these > > >> > >>>>>>>>>>> properties > > >> > >>>>>>>>>>>>>>>>>>>>>> vanishing > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> even more in the future, so from that > > >> > >>>>>>> perspective I > > >> > >>>>>>>>>>> support > > >> > >>>>>>>>>>>>>>>>>>>>>> changing > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> “DAG" to a more generic name. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 2. How other orchestration frameworks do > > >> > >>>>>> naming: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Dagster: pipeline > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Prefect: flow > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Flyte: workflow > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Temporal: workflow > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Kestra: flow > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> I think “workflow” is the most fitting > > >> > >>> name. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> 3. Given the large impact of this > > >> > >>> change, I > > >> > >>>>>>> suggest > > >> > >>>>>>>>>>>> defining > > >> > >>>>>>>>>>>>>>> a > > >> > >>>>>>>>>>>>>>>>>>>>>> clear > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> path forward. Would we first introduce > > >> > >>> the > > >> > >>>>>>>>> deprecation in > > >> > >>>>>>>>>>>>>>>>>>>>> Airflow > > >> > >>>>>>>>>>>>>>>>>>>>>> 3, > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> and remove “DAG” in Airflow 4? > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> Bas > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 09:22, Neil < > > >> > >>>>>>>> neil4r...@gmail.com > > >> > >>>>>>>>>>> <mailto:neil4r...@gmail.com>> > > >> > >>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see a problem with the term DAG, > > >> > >>>>>>>> especially > > >> > >>>>>>>>> when > > >> > >>>>>>>>>>>>>>>>>>> most > > >> > >>>>>>>>>>>>>>>>>>>>>> other > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> platforms embrace the term > > >> > >>> wholeheartedly. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> I don't see anything intimidating or > > >> > >>>>> confusing > > >> > >>>>>>>>> about it > > >> > >>>>>>>>>>> at > > >> > >>>>>>>>>>>>>>>>>>>>> all, > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> changing the term though would be fairly > > >> > >>>>>>> confusing > > >> > >>>>>>>>> to > > >> > >>>>>>>>>>> most > > >> > >>>>>>>>>>>>>>>>>>>>> users > > >> > >>>>>>>>>>>>>>>>>>>>>>> who > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> have been > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> using > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> the term for years. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Oct 22, 2024 at 1:18 AM Tzu-ping > > >> > >>>>> Chung > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> <t...@astronomer.io.inva <mailto: > > >> > >>>>>>>>> t...@astronomer.io.inva > > >> > >>>>>>>>>>>> lid > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I totally agree with doing away with > > >> > >>> the > > >> > >>>>> term > > >> > >>>>>>>> DAG. > > >> > >>>>>>>>> The > > >> > >>>>>>>>>>>> only > > >> > >>>>>>>>>>>>>>>>>>>>>>> problem > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> (aside > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> from actually telling people—including > > >> > >>>>>>> myself—to > > >> > >>>>>>>>> stop > > >> > >>>>>>>>>>>> using > > >> > >>>>>>>>>>>>>>>>>>>>> the > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> term) > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> is to > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> come up with a reasonable alternative. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> I can’t recall who, but someone > > >> > >>> mentioned > > >> > >>>>>>>>> “workflow” is > > >> > >>>>>>>>>>>> not > > >> > >>>>>>>>>>>>>>>>>>>>> very > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> accurate > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> for Airflow. The term “definition” was > > >> > >>>>>>> proposed, > > >> > >>>>>>>>> but > > >> > >>>>>>>>>>>> it’s a > > >> > >>>>>>>>>>>>>>>>>>>>> bit > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> broad; I tried to use it in a few > > >> > >>> places > > >> > >>>>> and > > >> > >>>>>>> kept > > >> > >>>>>>>>>>> finding > > >> > >>>>>>>>>>>>>>>>>>>>> myself > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> doubting “what definition?” and > > >> > >>> wanting to > > >> > >>>>>>>> clarify > > >> > >>>>>>>>> “DAG > > >> > >>>>>>>>>>>>>>>>>>>>>>> definition” > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> (defeating the purpose). > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> TP > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 22 Oct 2024, at 13:07, Jens > > >> > >>> Scheffler > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <j_scheff...@gmx.de.inva <mailto: > > >> > >>>>>>>>>>> j_scheff...@gmx.de.inva>LID> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi Ryan, > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for posting. I share the > > >> > >>> exactly > > >> > >>>>> same > > >> > >>>>>>>>>>>> observation, > > >> > >>>>>>>>>>>>>>>>>>>>> had a > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> short > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> laight because the DAG question is > > >> > >>> always > > >> > >>>>> an > > >> > >>>>>>>>>>> introduction > > >> > >>>>>>>>>>>>>>>>>>> if > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> someone > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> joins > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> the party. I think a global renaming > > >> > >>> makes > > >> > >>>>>>> sense. > > >> > >>>>>>>>>>>>>>>>>>> Especially > > >> > >>>>>>>>>>>>>>>>>>>>>> when > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> we > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> also > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> rename Dataset to Asset this is also a > > >> > >>>>>>> reasonable > > >> > >>>>>>>>> step. > > >> > >>>>>>>>>>>>>>>>>>>>> Concepts > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> still > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> can > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> stay the same. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> So I hope I don‘t need to join hiding > > >> > >>>>> below > > >> > >>>>>>> the > > >> > >>>>>>>>> desk > > >> > >>>>>>>>>>>> with > > >> > >>>>>>>>>>>>>>>>>>>>> you > > >> > >>>>>>>>>>>>>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> +1 > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> raising the discussion. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Technically we can still think if we > > >> > >>> keep > > >> > >>>>>>>> details > > >> > >>>>>>>>> of > > >> > >>>>>>>>>>>>>>>>>>> python > > >> > >>>>>>>>>>>>>>>>>>>>>> names > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> the > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> same because the execution is still a > > >> > >>> DAG… > > >> > >>>>>> but > > >> > >>>>>>>> user > > >> > >>>>>>>>>>>> facing > > >> > >>>>>>>>>>>>>>>>>>> it > > >> > >>>>>>>>>>>>>>>>>>>>>> is a > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> workflow. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Jens > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Sent from my Smartphone > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> On 21. Oct 2024, at 23:56, Ryan > > >> > >>> Hatter < > > >> > >>>>>>>>>>>>>>>>>>>>>>> ryan.hat...@astronomer.io <mailto: > > >> > >>>>>>>>> ryan.hat...@astronomer.io > > >> > >>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> .invalid> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> wrote: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Everyone please sheathe your > > >> > >>> swords... at > > >> > >>>>>>> least > > >> > >>>>>>>>> for > > >> > >>>>>>>>>>>> now. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" has very little > > >> > >>> meaning to > > >> > >>>>>>>> Airflow > > >> > >>>>>>>>>>>> users. > > >> > >>>>>>>>>>>>>>>>>>>>>> Indeed, > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> has > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> little meaning outside of some > > >> > >>>>>> mathematicians > > >> > >>>>>>>> and > > >> > >>>>>>>>>>>>>>>>>>> software > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> engineers > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> for > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> whom the properties of a DAG actually > > >> > >>>>>> matter. > > >> > >>>>>>>> For > > >> > >>>>>>>>>>>> someone > > >> > >>>>>>>>>>>>>>>>>>>>> new > > >> > >>>>>>>>>>>>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> data engineering or workflow > > >> > >>>>> orchestration, > > >> > >>>>>>> one > > >> > >>>>>>>>> of > > >> > >>>>>>>>>>> the > > >> > >>>>>>>>>>>>>>>>>>>>> first > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> questions they > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> will > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> likely have is, "what on earth is a > > >> > >>> DAG?" > > >> > >>>>>> The > > >> > >>>>>>>>> answer > > >> > >>>>>>>>>>> is > > >> > >>>>>>>>>>>>>>>>>>>>> almost > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> always, "It's a directed acyclic > > >> > >>> graph. > > >> > >>>>> You > > >> > >>>>>>>> don't > > >> > >>>>>>>>>>> need > > >> > >>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>> worry > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> about what > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> that > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> means; it's just a term for your > > >> > >>>>> workflow." > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The term "DAG" is problematic for at > > >> > >>>>> least > > >> > >>>>>> a > > >> > >>>>>>>>> couple > > >> > >>>>>>>>>>>>>>>>>>>>> important > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> reasons: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Complexity for New Users*: As > > >> > >>> mentioned > > >> > >>>>>>> above, > > >> > >>>>>>>>> "DAG" > > >> > >>>>>>>>>>>> is > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> unnecessarily intimidating and > > >> > >>> confusing. > > >> > >>>>>> We > > >> > >>>>>>>> want > > >> > >>>>>>>>>>>> Airflow > > >> > >>>>>>>>>>>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>> be > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> approachable, and > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> using > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> technical jargon like "DAG" right > > >> > >>> off the > > >> > >>>>>> bat > > >> > >>>>>>>>> creates > > >> > >>>>>>>>>>>> an > > >> > >>>>>>>>>>>>>>>>>>>>>> initial > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> barrier to > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> understanding. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *Disconnect Between DAG and Workflow > > >> > >>>>>>> Concepts*: > > >> > >>>>>>>>> The > > >> > >>>>>>>>>>> DAG > > >> > >>>>>>>>>>>>>>>>>>> is > > >> > >>>>>>>>>>>>>>>>>>>>>> just > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one component of an Airflow > > >> > >>> workflow. The > > >> > >>>>>>>>> workflow > > >> > >>>>>>>>>>>>>>>>>>> includes > > >> > >>>>>>>>>>>>>>>>>>>>>> its > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule, retries, timeouts, a dozen > > >> > >>>>> other > > >> > >>>>>>>>>>> parameters, > > >> > >>>>>>>>>>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>>>>>>>> other > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> metadata that > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> the > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG component doesn’t account for. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Consider the following from the > > >> > >>> Airflow > > >> > >>>>>>>> homepage > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <https://airflow.apache.org/> < > > >> > >>>>>>>>>>> https://airflow.apache.org/>>. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Apache Airflow® is a platform > > >> > >>> created by > > >> > >>>>>> the > > >> > >>>>>>>>>>> community > > >> > >>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> programmatically > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> author, schedule and monitor > > >> > >>> workflows. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Then, if we look at the "What is > > >> > >>>>> Airflow?" > > >> > >>>>>>> docs > > >> > >>>>>>>>> page > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> < > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>> > > >> > >>>>>>>>> > > >> > >>> > https://airflow.apache.org/docs/apache-airflow/stable/index.html > > < > > >> > >>>>>>>>>>> > > >> > >>>>> > > https://airflow.apache.org/docs/apache-airflow/stable/index.html > > >> > >>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> , > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>> we > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> can > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> see that the docs explain what > > >> > >>> Airflow is > > >> > >>>>>>>> without > > >> > >>>>>>>>>>> using > > >> > >>>>>>>>>>>>>>>>>>>>> "DAG." > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It's > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> only in > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the *workflow* Python code that the > > >> > >>> term > > >> > >>>>> is > > >> > >>>>>>>>>>> introduced > > >> > >>>>>>>>>>>>>>>>>>> out > > >> > >>>>>>>>>>>>>>>>>>>>> of > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nowhere > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> as a > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> comment that awkwardly tries to > > >> > >>> explain > > >> > >>>>> it: > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> # A DAG represents a workflow, a > > >> > >>>>> collection > > >> > >>>>>>> of > > >> > >>>>>>>>> tasks > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> It makes sense to not refer to DAGs > > >> > >>> in > > >> > >>>>>> these > > >> > >>>>>>>>>>>>>>>>>>> introductions > > >> > >>>>>>>>>>>>>>>>>>>>> to > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Airflow, because *Airflow doesn't > > >> > >>>>>> orchestrate > > >> > >>>>>>>>> DAGs; > > >> > >>>>>>>>>>> it > > >> > >>>>>>>>>>>>>>>>>>>>>>>> orchestrates > > >> > >>>>>>>>>>>>>>>>>>>>>>>>> workflows*. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> The > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> DAG is the model that, for reasons > > >> > >>>>>> irrelevant > > >> > >>>>>>>> to > > >> > >>>>>>>>>>> almost > > >> > >>>>>>>>>>>>>>>>>>>>> every > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> user, workflows must adhere to. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> So, I propose at least adding an > > >> > >>> alias > > >> > >>>>> for > > >> > >>>>>>> the > > >> > >>>>>>>>> term > > >> > >>>>>>>>>>>> "DAG" > > >> > >>>>>>>>>>>>>>>>>>>>> and > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> updating documentation to replace > > >> > >>> "DAG" > > >> > >>>>>> with > > >> > >>>>>>>>>>>> "workflow". > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> For example, instead of... > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @dag( > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> dagrun_timeout=timedelta(hours=1) > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Users could do... > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> @workflow( > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> schedule="@daily", > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_timeout=timedelta(hours=1) > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ) > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>>> And with that... I will start running > > >> > >>>>> away. > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>> > > >> > >>>>>>>>> > > >> > >>> > ------------------------------------------------------------------ > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> --- To unsubscribe, e-mail: > > >> > >>>>>>>>>>>>>>>>>>>>> dev-unsubscr...@airflow.apache.org <mailto: > > >> > >>>>>>>>>>> dev-unsubscr...@airflow.apache.org> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> For additional commands, e-mail: > > >> > >>>>>>>>>>>>>>>>>>>>> dev-h...@airflow.apache.org <mailto: > > >> > >>>>>>>>>>> dev-h...@airflow.apache.org> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>>>>>>>>> > > >> > >>>>>>>>>>>>>>>>>>>>> > > >> > > > > >> > > > > >> > > > > --------------------------------------------------------------------- > > >> > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> > > For additional commands, e-mail: dev-h...@airflow.apache.org > > >> > > > > >> > > > >> > > > >> > > > >> > > --------------------------------------------------------------------- > > >> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >> > For additional commands, e-mail: dev-h...@airflow.apache.org > > >> > > > >> > > > >> > > > > > >
