Hi Jarek, Thank you for summarizing the discussion points and linking to the two PRs that enhance the dag test feature.
We (at Nielsen) are working on a PR to extend the `airflow dag test` to be able to reconstruct the context from an existing dag run. I think allowing the option to execute only a specific task will also be helpful. The linked PR to mark tasks as success based on a pattern is handy for shortening the iteration time but expresses the same idea as "exclude all these other tasks". In cases where someone wants to debug only a specific task in a dag with many tasks, OR the task of interest is towards the end of the dag that has many branches and the task is a leaf, etc, it adds a little bit of extra work to figure out which tasks should be set to success. So specifying the task_id of interest vs specifying what all to exclude, will make the CLI more intuitive. Together, these features can be powerful when there is a production dag failure where the cause is not obvious from the logs. So the workflow would be: * production dag run failed * developer runs something like `airflow dag test <dag_id> <run_id> <task_id> --use-executor <executor of interest>` by providing the run_id of the failed dag run. Now they're able to get a debugger entrypoint into the task of interest in the executor of interest with the same context as the failed dag (who among us hasn't spent many hours trying to replicate a production failure scenario ;-) ) On Mon, Jun 17, 2024 at 1:14 AM Amogh Desai <amoghdesai....@gmail.com> wrote: > I agree with you Jarek. > > Every developer has their own way of debugging things and sharing those > with the community > including the best practices. There is always scope for improvement to the > documentation! > > Thanks & Regards, > Amogh Desai > > > On Fri, Jun 14, 2024 at 12:17 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Also I would like to drag attention of those who were interested in the >> subject on two related PRs: >> >> * https://github.com/apache/airflow/pull/40010 by jannisko (sorry if you >> see it - I do not know your real name :) ) -> where you can run a dag test >> while skipping (or actually mark as success) some tasks (for example >> sensors) >> * https://github.com/apache/airflow/pull/40205 by Vincent -> where you >> can >> run dag test using executor rather than `_run_raw_task` >> >> I think - the debug feature that Nielsen showcased on the call falls in >> the >> same pattern "make our airflow dags test` more powerful - and it would be >> great if we could incorporate similar pattern - where you can recreate >> task >> context from already executed dag_run - as part of the `airflow dags test` >> CLI command and `dag.test()` method. >> >> This is also I think a good opportunity to enhance documentation and >> explain all those patterns on how you can debug dag - in >> >> https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/debug.html#testing-dags-with-dag-test >> - for now the documentation is rather bare-bone, but it would be great if >> we explain some of the best practices and use cases how dag debugging >> might >> be done using the `airflow dags test` command. >> >> I think maybe other people have their own patterns of testing DAGs that >> they could contribute here - both as documentation update and maybe new >> features of our existing "airlfow dags test" command. >> >> WDYT? >> >> J. >> >> >> On Thu, Jun 13, 2024 at 10:29 PM Stefan Krawczyk <ste...@dagworks.io> >> wrote: >> >> > +1 for the recording please. >> > >> > On Thu, Jun 13, 2024, 1:26 PM Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> > > Just summarizing the call: >> > > >> > > * we had demo from Nielsen showing their debugging feature >> > > * It's based on environments created in the research environment where >> > > users can run DAGs and debug individual tasks - basically replaying >> and >> > > debugging the tasks based on existing DAG runs but without saving any >> > > changes to state of the dag in the DB >> > > * pretty useful thing is the way how they can use existing DAG run to >> > > recreate the context of execution based on existing dag run >> > > * Nielsen team used it with Airflow 1.10 and they will look into how >> the >> > > new `dag.test()` feature from Airflow 2.5 can be plugged into it and >> come >> > > back to it >> > > * nice thing is that they hooked it up with VSCode plugin where they >> can >> > > easily do all that within VSCode and can debug it out-of-the-box >> > > * possibly they could generalise it either as a "what could be done by >> > > others" description or maybe even having VSCode-from-airflow >> > out-of-the-box >> > > (the latter was my brainstorming idea). >> > > >> > > I have a recording - I do not want to publish it on the public >> devlist, >> > but >> > > If anyone is interested - let me know and I will share. >> > > >> > > J. >> > > >> > > >> > > On Thu, Jun 13, 2024 at 6:49 AM Albert Okiri <okirialb...@gmail.com> >> > > wrote: >> > > >> > > > Hi Jarek, I'm interested in joining this call. >> > > > >> > > > Regards, >> > > > Albert. >> > > > >> > > > On Thu, 13 Jun 2024, 07:43 Poorvi Rohidekar, < >> > > > poorvirohidekar....@gmail.com> >> > > > wrote: >> > > > >> > > > > Hi Jarek, >> > > > > >> > > > > I'd be interested in joining this call. >> > > > > >> > > > > Regards, >> > > > > Poorvi >> > > > > >> > > > > On Tue, 11 Jun 2024 at 21:42, Jarek Potiuk <ja...@potiuk.com> >> wrote: >> > > > > >> > > > > > And yes. I will check with them about recording :) >> > > > > > >> > > > > > On Tue, Jun 11, 2024 at 4:41 PM Jarek Potiuk <ja...@potiuk.com> >> > > wrote: >> > > > > > >> > > > > > > I think I have to warn Nielsen team that we are going to have >> a >> > big >> > > > > crowd >> > > > > > > :) >> > > > > > > >> > > > > > > On Tue, Jun 11, 2024 at 4:40 PM Bishundeo, Rajeshwar >> > > > > > > <rbish...@amazon.com.invalid> wrote: >> > > > > > > >> > > > > > >> This sounds exciting! I would like to join as well. >> > > > > > >> >> > > > > > >> -- Rajesh >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> On 2024-06-10, 11:12 PM, "Ephraim Anierobi" < >> > > > > ephraimanier...@gmail.com >> > > > > > >> <mailto:ephraimanier...@gmail.com>> wrote: >> > > > > > >> >> > > > > > >> >> > > > > > >> CAUTION: This email originated from outside of the >> organization. >> > > Do >> > > > > not >> > > > > > >> click links or open attachments unless you can confirm the >> > sender >> > > > and >> > > > > > know >> > > > > > >> the content is safe. >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un >> expéditeur >> > > > > > externe. >> > > > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si >> > vous >> > > ne >> > > > > > pouvez >> > > > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes >> pas >> > > > certain >> > > > > > que >> > > > > > >> le contenu ne présente aucun risque. >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> Hi Jarek, >> > > > > > >> >> > > > > > >> >> > > > > > >> Awesome!! I’m interested to join too >> > > > > > >> >> > > > > > >> >> > > > > > >> Ephraim >> > > > > > >> >> > > > > > >> >> > > > > > >> On Mon, 10 Jun 2024 at 23:41, Mehta, Shubham >> > > <shu...@amazon.com.inva >> > > > > > >> <mailto:shu...@amazon.com.inva>lid> >> > > > > > >> wrote: >> > > > > > >> >> > > > > > >> >> > > > > > >> > This is great, thanks for working with them to share this >> with >> > > the >> > > > > > >> > community. Interested to join as well. >> > > > > > >> > >> > > > > > >> > Shubham >> > > > > > >> > >> > > > > > >> > On 2024-06-07, 11:56 PM, "Jarek Potiuk" <ja...@potiuk.com >> > > > <mailto: >> > > > > > >> ja...@potiuk.com> <mailto: >> > > > > > >> > ja...@potiuk.com <mailto:ja...@potiuk.com>>> wrote: >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > CAUTION: This email originated from outside of the >> > organization. >> > > > Do >> > > > > > not >> > > > > > >> > click links or open attachments unless you can confirm the >> > > sender >> > > > > and >> > > > > > >> know >> > > > > > >> > the content is safe. >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > AVERTISSEMENT: Ce courrier électronique provient d’un >> > expéditeur >> > > > > > >> externe. >> > > > > > >> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe >> si >> > > vous >> > > > ne >> > > > > > >> pouvez >> > > > > > >> > pas confirmer l’identité de l’expéditeur et si vous n’êtes >> pas >> > > > > certain >> > > > > > >> que >> > > > > > >> > le contenu ne présente aucun risque. >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > Hello here, >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > At PyCon US I met a few people from Nielsen who had >> developed >> > > > > > internally >> > > > > > >> > tooling for IDE/Python debugger integrated debugging of >> > Airflow >> > > > > DAGs. >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > They are thrilled with the opportunity of sharing what >> they've >> > > > done >> > > > > > and >> > > > > > >> > possibly maybe even bringing it to Airflow. As one of the >> > > Airflow >> > > > 3 >> > > > > > >> > workstreams I am particularly interested in is to >> "*Simplify >> > the >> > > > > > >> learning >> > > > > > >> > curve*" [1] - this sounds pretty interesting in this >> context. >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > I will have a call with them next week - immediately after >> the >> > > > > > Airflow 3 >> > > > > > >> > dev call (Thu, 13th of June, 6pm CEST), where they will >> demo >> > > what >> > > > > they >> > > > > > >> have >> > > > > > >> > - so if you would like to join it - let me know, I will >> invite >> > > you >> > > > > to >> > > > > > >> the >> > > > > > >> > call. >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > [1] >> > > > > > >> > >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 >> > > > > > >> < >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 >> > > > > > >> > >> > > > > > >> > < >> > > > > > >> > >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 >> > > > > > >> < >> > > > > > >> >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 >> > > > > > >> > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > J. >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > >> --------------------------------------------------------------------- >> > > > > > >> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > > > <mailto: >> > > > > > >> dev-unsubscr...@airflow.apache.org> >> > > > > > >> > For additional commands, e-mail: >> dev-h...@airflow.apache.org >> > > > > <mailto: >> > > > > > >> dev-h...@airflow.apache.org> >> > > > > > >> > >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > >> --------------------------------------------------------------------- >> > > > > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> > > > > > >> For additional commands, e-mail: dev-h...@airflow.apache.org >> > > > > > >> >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > -- Regards, Meygha Machado Nielsen
