Just summarizing the call: * we had demo from Nielsen showing their debugging feature * It's based on environments created in the research environment where users can run DAGs and debug individual tasks - basically replaying and debugging the tasks based on existing DAG runs but without saving any changes to state of the dag in the DB * pretty useful thing is the way how they can use existing DAG run to recreate the context of execution based on existing dag run * Nielsen team used it with Airflow 1.10 and they will look into how the new `dag.test()` feature from Airflow 2.5 can be plugged into it and come back to it * nice thing is that they hooked it up with VSCode plugin where they can easily do all that within VSCode and can debug it out-of-the-box * possibly they could generalise it either as a "what could be done by others" description or maybe even having VSCode-from-airflow out-of-the-box (the latter was my brainstorming idea).
I have a recording - I do not want to publish it on the public devlist, but If anyone is interested - let me know and I will share. J. On Thu, Jun 13, 2024 at 6:49 AM Albert Okiri <okirialb...@gmail.com> wrote: > Hi Jarek, I'm interested in joining this call. > > Regards, > Albert. > > On Thu, 13 Jun 2024, 07:43 Poorvi Rohidekar, < > poorvirohidekar....@gmail.com> > wrote: > > > Hi Jarek, > > > > I'd be interested in joining this call. > > > > Regards, > > Poorvi > > > > On Tue, 11 Jun 2024 at 21:42, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > And yes. I will check with them about recording :) > > > > > > On Tue, Jun 11, 2024 at 4:41 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > I think I have to warn Nielsen team that we are going to have a big > > crowd > > > > :) > > > > > > > > On Tue, Jun 11, 2024 at 4:40 PM Bishundeo, Rajeshwar > > > > <rbish...@amazon.com.invalid> wrote: > > > > > > > >> This sounds exciting! I would like to join as well. > > > >> > > > >> -- Rajesh > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> On 2024-06-10, 11:12 PM, "Ephraim Anierobi" < > > ephraimanier...@gmail.com > > > >> <mailto:ephraimanier...@gmail.com>> wrote: > > > >> > > > >> > > > >> CAUTION: This email originated from outside of the organization. Do > > not > > > >> click links or open attachments unless you can confirm the sender > and > > > know > > > >> the content is safe. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > externe. > > > >> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > > pouvez > > > >> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain > > > que > > > >> le contenu ne présente aucun risque. > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> > > > >> Hi Jarek, > > > >> > > > >> > > > >> Awesome!! I’m interested to join too > > > >> > > > >> > > > >> Ephraim > > > >> > > > >> > > > >> On Mon, 10 Jun 2024 at 23:41, Mehta, Shubham <shu...@amazon.com.inva > > > >> <mailto:shu...@amazon.com.inva>lid> > > > >> wrote: > > > >> > > > >> > > > >> > This is great, thanks for working with them to share this with the > > > >> > community. Interested to join as well. > > > >> > > > > >> > Shubham > > > >> > > > > >> > On 2024-06-07, 11:56 PM, "Jarek Potiuk" <ja...@potiuk.com > <mailto: > > > >> ja...@potiuk.com> <mailto: > > > >> > ja...@potiuk.com <mailto:ja...@potiuk.com>>> wrote: > > > >> > > > > >> > > > > >> > CAUTION: This email originated from outside of the organization. > Do > > > not > > > >> > click links or open attachments unless you can confirm the sender > > and > > > >> know > > > >> > the content is safe. > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > > >> externe. > > > >> > Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous > ne > > > >> pouvez > > > >> > pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > > certain > > > >> que > > > >> > le contenu ne présente aucun risque. > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > Hello here, > > > >> > > > > >> > > > > >> > At PyCon US I met a few people from Nielsen who had developed > > > internally > > > >> > tooling for IDE/Python debugger integrated debugging of Airflow > > DAGs. > > > >> > > > > >> > > > > >> > They are thrilled with the opportunity of sharing what they've > done > > > and > > > >> > possibly maybe even bringing it to Airflow. As one of the Airflow > 3 > > > >> > workstreams I am particularly interested in is to "*Simplify the > > > >> learning > > > >> > curve*" [1] - this sounds pretty interesting in this context. > > > >> > > > > >> > > > > >> > I will have a call with them next week - immediately after the > > > Airflow 3 > > > >> > dev call (Thu, 13th of June, 6pm CEST), where they will demo what > > they > > > >> have > > > >> > - so if you would like to join it - let me know, I will invite you > > to > > > >> the > > > >> > call. > > > >> > > > > >> > > > > >> > [1] > > > >> > > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > >> < > > > >> > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > >> > > > > >> > < > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > >> < > > > >> > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-4June2024 > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > J. > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > --------------------------------------------------------------------- > > > >> > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > <mailto: > > > >> dev-unsubscr...@airflow.apache.org> > > > >> > For additional commands, e-mail: dev-h...@airflow.apache.org > > <mailto: > > > >> dev-h...@airflow.apache.org> > > > >> > > > > >> > > > >> > > > >> > > > >> > > > >> > --------------------------------------------------------------------- > > > >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > > >> For additional commands, e-mail: dev-h...@airflow.apache.org > > > >> > > > > > > > > > >
