Hi Meghya, There is also “airflow task test” that can run a specific task for debugging.
On Tue, 18 Jun 2024 at 21:04, Meygha Machado <meygha.b...@nielsen.com.invalid> wrote: > Hi Jarek, > > Thank you for summarizing the discussion points and linking to the two PRs > that enhance the dag test feature. > > We (at Nielsen) are working on a PR to extend the `airflow dag test` to be > able to reconstruct the context from an existing dag run. > > I think allowing the option to execute only a specific task will also be > helpful. The linked PR to mark tasks as success based on a pattern is handy > for shortening the iteration time but expresses the same idea as "exclude > all these other tasks". In cases where someone wants to debug only a > specific task in a dag with many tasks, OR the task of interest is towards > the end of the dag that has many branches and the task is a leaf, etc, it > adds a little bit of extra work to figure out which tasks should be set to > success. So specifying the task_id of interest vs specifying what all to > exclude, will make the CLI more intuitive. > > Together, these features can be powerful when there is a production dag > failure where the cause is not obvious from the logs. So the workflow would > be: > * production dag run failed > * developer runs something like `airflow dag test <dag_id> <run_id> > <task_id> --use-executor <executor of interest>` by providing the run_id of > the failed dag run. Now they're able to get a debugger entrypoint into the > task of interest in the executor of interest with the same context as the > failed dag (who among us hasn't spent many hours trying to replicate a > production failure scenario ;-) ) > > > On Mon, Jun 17, 2024 at 1:14 AM Amogh Desai <amoghdesai....@gmail.com> > wrote: > > > I agree with you Jarek. > > > > Every developer has their own way of debugging things and sharing those > > with the community > > including the best practices. There is always scope for improvement to > the > > documentation! > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Fri, Jun 14, 2024 at 12:17 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > >> Also I would like to drag attention of those who were interested in the > >> subject on two related PRs: > >> > >> * https://github.com/apache/airflow/pull/40010 by jannisko (sorry if > you > >> see it - I do not know your real name :) ) -> where you can run a dag > test > >> while skipping (or actually mark as success) some tasks (for example > >> sensors) > >> * https://github.com/apache/airflow/pull/40205 by Vincent -> where you > >> can > >> run dag test using executor rather than `_run_raw_task` > >> > >> I think - the debug feature that Nielsen showcased on the call falls in > >> the > >> same pattern "make our airflow dags test` more powerful - and it would > be > >> great if we could incorporate similar pattern - where you can recreate > >> task > >> context from already executed dag_run - as part of the `airflow dags > test` > >> CLI command and `dag.test()` method. > >> > >> This is also I think a good opportunity to enhance documentation and > >> explain all those patterns on how you can debug dag - in > >> > >> > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/debug.html#testing-dags-with-dag-test > >> - for now the documentation is rather bare-bone, but it would be great > if > >> we explain some of the best practices and use cases how dag debugging > >> might > >> be done using the `airflow dags test` command. > >> > >> I think maybe other people have their own patterns of testing DAGs that > >> they could contribute here - both as documentation update and maybe new > >> features of our existing "airlfow dags test" command. > >> > >> WDYT? > >> > >> J. > >> > >> > >> On Thu, Jun 13, 2024 at 10:29 PM Stefan Krawczyk <ste...@dagworks.io> > >> wrote: > >> > >> > +1 for the recording please. > >> > > >> > On Thu, Jun 13, 2024, 1:26 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > > >> > > Just summarizing the call: > >> > > > >> > > * we had demo from Nielsen showing their debugging feature > >> > > * It's based on environments created in the research environment > where > >> > > users can run DAGs and debug individual tasks - basically replaying > >> and > >> > > debugging the tasks based on existing DAG runs but without saving > any > >> > > changes to state of the dag in the DB > >> > > * pretty useful thing is the way how they can use existing DAG run > to > >> > > recreate the context of execution based on existing dag run > >> > > * Nielsen team used it with Airflow 1.10 and they will look into how > >> the > >> > > new `dag.test()` feature from Airflow 2.5 can be plugged into it and > >> come > >> > > back to it > >> > > * nice thing is that they hooked it up with VSCode plugin where they > >> can > >> > > easily do all that within VSCode and can debug it out-of-the-box > >> > > * possibly they could generalise it either as a "what could be done > by > >> > > others" description or maybe even having VSCode-from-airflow > >> > out-of-the-box > >> > > (the latter was my brainstorming idea). > >> > > > >> > > I have a recording - I do not want to publish it on the public > >> devlist, > >> > but > >> > > If anyone is interested - let me know and I will share. > >> > > > >> > > J. > >> > > > >> > > > >> > > On Thu, Jun 13, 2024 at 6:49 AM Albert Okiri <okirialb...@gmail.com > > > >> > > wrote: > >> > > > >> > > > Hi Jarek, I'm interested in joining this call. > >> > > > > >> > > > Regards, > >> > > > Albert. > >> > > > > >> > > > On Thu, 13 Jun 2024, 07:43 Poorvi Rohidekar, < > >> > > > poorvirohidekar....@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > Hi Jarek, > >> > > > > > >> > > > > I'd be interested in joining this call. > >> > > > > > >> > > > > Regards, > >> > > > > Poorvi > >> > > > > > >> > > > > On Tue, 11 Jun 2024 at 21:42, Jarek Potiuk <ja...@potiuk.com> > >> wrote: > >> > > > > > >> > > > > > And yes. I will check with them about recording :) > >> > > > > > > >> > > > > > On Tue, Jun 11, 2024 at 4:41 PM Jarek Potiuk < > ja...@potiuk.com> > >> > > wrote: > >> > > > > > > >> > > > > > > I think I have to warn Nielsen team that we are going to > have > >> a > >> > big > >> > > > > crowd > >> > > > > > > :) > >> > > > > > > > >> > > > > > > On Tue, Jun 11, 2024 at 4:40 PM Bishundeo, Rajeshwar > >> > > > > > > <rbish...@amazon.com.invalid> wrote: > >> > > > > > > > >> > > > > > >> This sounds exciting! I would like to join as well. > >> > > > > > >> > >> > > > > > >> -- Rajesh > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> On 2024-06-10, 11:12 PM, "Ephraim Anierobi" < > >> > > > > ephraimanier...@gmail.com > >> > > > > > >> <mailto:ephraimanier...@gmail.com>> wrote: > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> CAUTION: This email originated from outside of the > >> organization. > >> > > Do > >> > > > > not > >> > > > > > >> click links or open attachments unless you can confirm the > >> > sender > >> > > > and > >> > > > > > know > >> > > > > > >> the content is safe. > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un > >> expéditeur > >> > > > > > externe. > >> > > > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > si > >> > vous > >> > > ne > >> > > > > > pouvez > >> > > > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes > >> pas > >> > > > certain > >> > > > > > que > >> > > > > > >> le contenu ne présente aucun risque. > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> Hi Jarek, > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> Awesome!! I’m interested to join too > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> Ephraim > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> On Mon, 10 Jun 2024 at 23:41, Mehta, Shubham > >> > > <shu...@amazon.com.inva > >> > > > > > >> <mailto:shu...@amazon.com.inva>lid> > >> > > > > > >> wrote: > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > This is great, thanks for working with them to share this > >> with > >> > > the > >> > > > > > >> > community. Interested to join as well. > >> > > > > > >> > > >> > > > > > >> > Shubham > >> > > > > > >> > > >> > > > > > >> > On 2024-06-07, 11:56 PM, "Jarek Potiuk" < > ja...@potiuk.com > >> > > > <mailto: > >> > > > > > >> ja...@potiuk.com> <mailto: > >> > > > > > >> > ja...@potiuk.com <mailto:ja...@potiuk.com>>> wrote: > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > CAUTION: This email originated from outside of the > >> > organization. > >> > > > Do > >> > > > > > not > >> > > > > > >> > click links or open attachments unless you can confirm > the > >> > > sender > >> > > > > and > >> > > > > > >> know > >> > > > > > >> > the content is safe. > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > AVERTISSEMENT: Ce courrier électronique provient d’un > >> > expéditeur > >> > > > > > >> externe. > >> > > > > > >> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe > >> si > >> > > vous > >> > > > ne > >> > > > > > >> pouvez > >> > > > > > >> > pas confirmer l’identité de l’expéditeur et si vous > n’êtes > >> pas > >> > > > > certain > >> > > > > > >> que > >> > > > > > >> > le contenu ne présente aucun risque. > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > Hello here, > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > At PyCon US I met a few people from Nielsen who had > >> developed > >> > > > > > internally > >> > > > > > >> > tooling for IDE/Python debugger integrated debugging of > >> > Airflow > >> > > > > DAGs. > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > They are thrilled with the opportunity of sharing what > >> they've > >> > > > done > >> > > > > > and > >> > > > > > >> > possibly maybe even bringing it to Airflow. As one of the > >> > > Airflow > >> > > > 3 > >> > > > > > >> > workstreams I am particularly interested in is to > >> "*Simplify > >> > the > >> > > > > > >> learning > >> > > > > > >> > curve*" [1] - this sounds pretty interesting in this > >> context. > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > I will have a call with them next week - immediately > after > >> the > >> > > > > > Airflow 3 > >> > > > > > >> > dev call (Thu, 13th of June, 6pm CEST), where they will > >> demo > >> > > what > >> > > > > they > >> > > > > > >> have > >> > > > > > >> > - so if you would like to join it - let me know, I will > >> invite > >> > > you > >> > > > > to > >> > > > > > >> the > >> > > > > > >> > call. > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > [1] > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > >> > > > > > >> < > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > >> > > > > > >> > > >> > > > > > >> > < > >> > > > > > >> > > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > >> > > > > > >> < > >> > > > > > >> > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > >> > > > > > >> > > >> > > > > > >> > > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > J. > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> > > >> > > > > > >> --------------------------------------------------------------------- > >> > > > > > >> > To unsubscribe, e-mail: > dev-unsubscr...@airflow.apache.org > >> > > > <mailto: > >> > > > > > >> dev-unsubscr...@airflow.apache.org> > >> > > > > > >> > For additional commands, e-mail: > >> dev-h...@airflow.apache.org > >> > > > > <mailto: > >> > > > > > >> dev-h...@airflow.apache.org> > >> > > > > > >> > > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > >> --------------------------------------------------------------------- > >> > > > > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >> > > > > > >> For additional commands, e-mail: > dev-h...@airflow.apache.org > >> > > > > > >> > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > -- > Regards, > Meygha Machado > Nielsen >
