Also I would like to drag attention of those who were interested in the subject on two related PRs:
* https://github.com/apache/airflow/pull/40010 by jannisko (sorry if you see it - I do not know your real name :) ) -> where you can run a dag test while skipping (or actually mark as success) some tasks (for example sensors) * https://github.com/apache/airflow/pull/40205 by Vincent -> where you can run dag test using executor rather than `_run_raw_task` I think - the debug feature that Nielsen showcased on the call falls in the same pattern "make our airflow dags test` more powerful - and it would be great if we could incorporate similar pattern - where you can recreate task context from already executed dag_run - as part of the `airflow dags test` CLI command and `dag.test()` method. This is also I think a good opportunity to enhance documentation and explain all those patterns on how you can debug dag - in https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/debug.html#testing-dags-with-dag-test - for now the documentation is rather bare-bone, but it would be great if we explain some of the best practices and use cases how dag debugging might be done using the `airflow dags test` command. I think maybe other people have their own patterns of testing DAGs that they could contribute here - both as documentation update and maybe new features of our existing "airlfow dags test" command. WDYT? J. On Thu, Jun 13, 2024 at 10:29 PM Stefan Krawczyk <[email protected]> wrote: > +1 for the recording please. > > On Thu, Jun 13, 2024, 1:26 PM Jarek Potiuk <[email protected]> wrote: > > > Just summarizing the call: > > > > * we had demo from Nielsen showing their debugging feature > > * It's based on environments created in the research environment where > > users can run DAGs and debug individual tasks - basically replaying and > > debugging the tasks based on existing DAG runs but without saving any > > changes to state of the dag in the DB > > * pretty useful thing is the way how they can use existing DAG run to > > recreate the context of execution based on existing dag run > > * Nielsen team used it with Airflow 1.10 and they will look into how the > > new `dag.test()` feature from Airflow 2.5 can be plugged into it and come > > back to it > > * nice thing is that they hooked it up with VSCode plugin where they can > > easily do all that within VSCode and can debug it out-of-the-box > > * possibly they could generalise it either as a "what could be done by > > others" description or maybe even having VSCode-from-airflow > out-of-the-box > > (the latter was my brainstorming idea). > > > > I have a recording - I do not want to publish it on the public devlist, > but > > If anyone is interested - let me know and I will share. > > > > J. > > > > > > On Thu, Jun 13, 2024 at 6:49 AM Albert Okiri <[email protected]> > > wrote: > > > > > Hi Jarek, I'm interested in joining this call. > > > > > > Regards, > > > Albert. > > > > > > On Thu, 13 Jun 2024, 07:43 Poorvi Rohidekar, < > > > [email protected]> > > > wrote: > > > > > > > Hi Jarek, > > > > > > > > I'd be interested in joining this call. > > > > > > > > Regards, > > > > Poorvi > > > > > > > > On Tue, 11 Jun 2024 at 21:42, Jarek Potiuk <[email protected]> wrote: > > > > > > > > > And yes. I will check with them about recording :) > > > > > > > > > > On Tue, Jun 11, 2024 at 4:41 PM Jarek Potiuk <[email protected]> > > wrote: > > > > > > > > > > > I think I have to warn Nielsen team that we are going to have a > big > > > > crowd > > > > > > :) > > > > > > > > > > > > On Tue, Jun 11, 2024 at 4:40 PM Bishundeo, Rajeshwar > > > > > > <[email protected]> wrote: > > > > > > > > > > > >> This sounds exciting! I would like to join as well. > > > > > >> > > > > > >> -- Rajesh > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> On 2024-06-10, 11:12 PM, "Ephraim Anierobi" < > > > > [email protected] > > > > > >> <mailto:[email protected]>> wrote: > > > > > >> > > > > > >> > > > > > >> CAUTION: This email originated from outside of the organization. > > Do > > > > not > > > > > >> click links or open attachments unless you can confirm the > sender > > > and > > > > > know > > > > > >> the content is safe. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > > > externe. > > > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > vous > > ne > > > > > pouvez > > > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > certain > > > > > que > > > > > >> le contenu ne présente aucun risque. > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> Hi Jarek, > > > > > >> > > > > > >> > > > > > >> Awesome!! I’m interested to join too > > > > > >> > > > > > >> > > > > > >> Ephraim > > > > > >> > > > > > >> > > > > > >> On Mon, 10 Jun 2024 at 23:41, Mehta, Shubham > > <[email protected] > > > > > >> <mailto:[email protected]>lid> > > > > > >> wrote: > > > > > >> > > > > > >> > > > > > >> > This is great, thanks for working with them to share this with > > the > > > > > >> > community. Interested to join as well. > > > > > >> > > > > > > >> > Shubham > > > > > >> > > > > > > >> > On 2024-06-07, 11:56 PM, "Jarek Potiuk" <[email protected] > > > <mailto: > > > > > >> [email protected]> <mailto: > > > > > >> > [email protected] <mailto:[email protected]>>> wrote: > > > > > >> > > > > > > >> > > > > > > >> > CAUTION: This email originated from outside of the > organization. > > > Do > > > > > not > > > > > >> > click links or open attachments unless you can confirm the > > sender > > > > and > > > > > >> know > > > > > >> > the content is safe. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > AVERTISSEMENT: Ce courrier électronique provient d’un > expéditeur > > > > > >> externe. > > > > > >> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > ne > > > > > >> pouvez > > > > > >> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > > certain > > > > > >> que > > > > > >> > le contenu ne présente aucun risque. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > Hello here, > > > > > >> > > > > > > >> > > > > > > >> > At PyCon US I met a few people from Nielsen who had developed > > > > > internally > > > > > >> > tooling for IDE/Python debugger integrated debugging of > Airflow > > > > DAGs. > > > > > >> > > > > > > >> > > > > > > >> > They are thrilled with the opportunity of sharing what they've > > > done > > > > > and > > > > > >> > possibly maybe even bringing it to Airflow. As one of the > > Airflow > > > 3 > > > > > >> > workstreams I am particularly interested in is to "*Simplify > the > > > > > >> learning > > > > > >> > curve*" [1] - this sounds pretty interesting in this context. > > > > > >> > > > > > > >> > > > > > > >> > I will have a call with them next week - immediately after the > > > > > Airflow 3 > > > > > >> > dev call (Thu, 13th of June, 6pm CEST), where they will demo > > what > > > > they > > > > > >> have > > > > > >> > - so if you would like to join it - let me know, I will invite > > you > > > > to > > > > > >> the > > > > > >> > call. > > > > > >> > > > > > > >> > > > > > > >> > [1] > > > > > >> > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > >> > > > > > > >> > < > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > >> < > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > >> > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > J. > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > --------------------------------------------------------------------- > > > > > >> > To unsubscribe, e-mail: [email protected] > > > <mailto: > > > > > >> [email protected]> > > > > > >> > For additional commands, e-mail: [email protected] > > > > <mailto: > > > > > >> [email protected]> > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > --------------------------------------------------------------------- > > > > > >> To unsubscribe, e-mail: [email protected] > > > > > >> For additional commands, e-mail: [email protected] > > > > > >> > > > > > > > > > > > > > > > > > > > > >
