I agree with you Jarek. Every developer has their own way of debugging things and sharing those with the community including the best practices. There is always scope for improvement to the documentation!
Thanks & Regards, Amogh Desai On Fri, Jun 14, 2024 at 12:17 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Also I would like to drag attention of those who were interested in the > subject on two related PRs: > > * https://github.com/apache/airflow/pull/40010 by jannisko (sorry if you > see it - I do not know your real name :) ) -> where you can run a dag test > while skipping (or actually mark as success) some tasks (for example > sensors) > * https://github.com/apache/airflow/pull/40205 by Vincent -> where you can > run dag test using executor rather than `_run_raw_task` > > I think - the debug feature that Nielsen showcased on the call falls in the > same pattern "make our airflow dags test` more powerful - and it would be > great if we could incorporate similar pattern - where you can recreate task > context from already executed dag_run - as part of the `airflow dags test` > CLI command and `dag.test()` method. > > This is also I think a good opportunity to enhance documentation and > explain all those patterns on how you can debug dag - in > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/debug.html#testing-dags-with-dag-test > - for now the documentation is rather bare-bone, but it would be great if > we explain some of the best practices and use cases how dag debugging might > be done using the `airflow dags test` command. > > I think maybe other people have their own patterns of testing DAGs that > they could contribute here - both as documentation update and maybe new > features of our existing "airlfow dags test" command. > > WDYT? > > J. > > > On Thu, Jun 13, 2024 at 10:29 PM Stefan Krawczyk <ste...@dagworks.io> > wrote: > > > +1 for the recording please. > > > > On Thu, Jun 13, 2024, 1:26 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > Just summarizing the call: > > > > > > * we had demo from Nielsen showing their debugging feature > > > * It's based on environments created in the research environment where > > > users can run DAGs and debug individual tasks - basically replaying and > > > debugging the tasks based on existing DAG runs but without saving any > > > changes to state of the dag in the DB > > > * pretty useful thing is the way how they can use existing DAG run to > > > recreate the context of execution based on existing dag run > > > * Nielsen team used it with Airflow 1.10 and they will look into how > the > > > new `dag.test()` feature from Airflow 2.5 can be plugged into it and > come > > > back to it > > > * nice thing is that they hooked it up with VSCode plugin where they > can > > > easily do all that within VSCode and can debug it out-of-the-box > > > * possibly they could generalise it either as a "what could be done by > > > others" description or maybe even having VSCode-from-airflow > > out-of-the-box > > > (the latter was my brainstorming idea). > > > > > > I have a recording - I do not want to publish it on the public devlist, > > but > > > If anyone is interested - let me know and I will share. > > > > > > J. > > > > > > > > > On Thu, Jun 13, 2024 at 6:49 AM Albert Okiri <okirialb...@gmail.com> > > > wrote: > > > > > > > Hi Jarek, I'm interested in joining this call. > > > > > > > > Regards, > > > > Albert. > > > > > > > > On Thu, 13 Jun 2024, 07:43 Poorvi Rohidekar, < > > > > poorvirohidekar....@gmail.com> > > > > wrote: > > > > > > > > > Hi Jarek, > > > > > > > > > > I'd be interested in joining this call. > > > > > > > > > > Regards, > > > > > Poorvi > > > > > > > > > > On Tue, 11 Jun 2024 at 21:42, Jarek Potiuk <ja...@potiuk.com> > wrote: > > > > > > > > > > > And yes. I will check with them about recording :) > > > > > > > > > > > > On Tue, Jun 11, 2024 at 4:41 PM Jarek Potiuk <ja...@potiuk.com> > > > wrote: > > > > > > > > > > > > > I think I have to warn Nielsen team that we are going to have a > > big > > > > > crowd > > > > > > > :) > > > > > > > > > > > > > > On Tue, Jun 11, 2024 at 4:40 PM Bishundeo, Rajeshwar > > > > > > > <rbish...@amazon.com.invalid> wrote: > > > > > > > > > > > > > >> This sounds exciting! I would like to join as well. > > > > > > >> > > > > > > >> -- Rajesh > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> On 2024-06-10, 11:12 PM, "Ephraim Anierobi" < > > > > > ephraimanier...@gmail.com > > > > > > >> <mailto:ephraimanier...@gmail.com>> wrote: > > > > > > >> > > > > > > >> > > > > > > >> CAUTION: This email originated from outside of the > organization. > > > Do > > > > > not > > > > > > >> click links or open attachments unless you can confirm the > > sender > > > > and > > > > > > know > > > > > > >> the content is safe. > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un > expéditeur > > > > > > externe. > > > > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > vous > > > ne > > > > > > pouvez > > > > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > > > certain > > > > > > que > > > > > > >> le contenu ne présente aucun risque. > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> Hi Jarek, > > > > > > >> > > > > > > >> > > > > > > >> Awesome!! I’m interested to join too > > > > > > >> > > > > > > >> > > > > > > >> Ephraim > > > > > > >> > > > > > > >> > > > > > > >> On Mon, 10 Jun 2024 at 23:41, Mehta, Shubham > > > <shu...@amazon.com.inva > > > > > > >> <mailto:shu...@amazon.com.inva>lid> > > > > > > >> wrote: > > > > > > >> > > > > > > >> > > > > > > >> > This is great, thanks for working with them to share this > with > > > the > > > > > > >> > community. Interested to join as well. > > > > > > >> > > > > > > > >> > Shubham > > > > > > >> > > > > > > > >> > On 2024-06-07, 11:56 PM, "Jarek Potiuk" <ja...@potiuk.com > > > > <mailto: > > > > > > >> ja...@potiuk.com> <mailto: > > > > > > >> > ja...@potiuk.com <mailto:ja...@potiuk.com>>> wrote: > > > > > > >> > > > > > > > >> > > > > > > > >> > CAUTION: This email originated from outside of the > > organization. > > > > Do > > > > > > not > > > > > > >> > click links or open attachments unless you can confirm the > > > sender > > > > > and > > > > > > >> know > > > > > > >> > the content is safe. > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > AVERTISSEMENT: Ce courrier électronique provient d’un > > expéditeur > > > > > > >> externe. > > > > > > >> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si > > > vous > > > > ne > > > > > > >> pouvez > > > > > > >> > pas confirmer l’identité de l’expéditeur et si vous n’êtes > pas > > > > > certain > > > > > > >> que > > > > > > >> > le contenu ne présente aucun risque. > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > Hello here, > > > > > > >> > > > > > > > >> > > > > > > > >> > At PyCon US I met a few people from Nielsen who had > developed > > > > > > internally > > > > > > >> > tooling for IDE/Python debugger integrated debugging of > > Airflow > > > > > DAGs. > > > > > > >> > > > > > > > >> > > > > > > > >> > They are thrilled with the opportunity of sharing what > they've > > > > done > > > > > > and > > > > > > >> > possibly maybe even bringing it to Airflow. As one of the > > > Airflow > > > > 3 > > > > > > >> > workstreams I am particularly interested in is to "*Simplify > > the > > > > > > >> learning > > > > > > >> > curve*" [1] - this sounds pretty interesting in this > context. > > > > > > >> > > > > > > > >> > > > > > > > >> > I will have a call with them next week - immediately after > the > > > > > > Airflow 3 > > > > > > >> > dev call (Thu, 13th of June, 6pm CEST), where they will demo > > > what > > > > > they > > > > > > >> have > > > > > > >> > - so if you would like to join it - let me know, I will > invite > > > you > > > > > to > > > > > > >> the > > > > > > >> > call. > > > > > > >> > > > > > > > >> > > > > > > > >> > [1] > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > > >> < > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > > >> > > > > > > > >> > < > > > > > > >> > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > > >> < > > > > > > >> > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > > > > >> > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > J. > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > --------------------------------------------------------------------- > > > > > > >> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > <mailto: > > > > > > >> dev-unsubscr...@airflow.apache.org> > > > > > > >> > For additional commands, e-mail: > dev-h...@airflow.apache.org > > > > > <mailto: > > > > > > >> dev-h...@airflow.apache.org> > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > --------------------------------------------------------------------- > > > > > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > > > > >> For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > >
