I wonder if we could generalize to summarize what network activity there was on dag parse more generally. e.g. if there were API calls or whatnot.
Quick google search yields this https://thepythoncode.com/article/make-a-network-usage-monitor-in-python but not sure it could be used for this. Intuitively it seems possible. On Fri, Jun 14, 2024 at 1:45 PM Jarek Potiuk <[email protected]> wrote: > Agree with Kaxil. Last run is.good. Actually if we are looking for Airflow > 3, the target for 'readu for migration' will be just '0 everywhere' . > Having last run and getting totals and seeing it goes to 0 steadily should > be the 'ready for migration' target for anyone running Airflow 3. > > pt., 14 cze 2024, 22:36 użytkownik Kaxil Naik <[email protected]> > napisał: > > > Yeah, it is 30s by default; storing it in DB would be useless, apart from > > the most recent run, in my opinion. > > > > On Fri, 14 Jun 2024 at 15:24, Constance Martineau > > <[email protected]> wrote: > > > > > I love the idea. If we were to store it in the DB, would we keep a > > history, > > > or only the latest stats from the most recent dag parsing loop? DAG > > parsing > > > by default is every 30s right? > > > > > > On Fri, Jun 14, 2024 at 6:53 AM Jarek Potiuk <[email protected]> wrote: > > > > > > > > I think we still need to enable the ability for DAGs at parse time > to > > > > access Variables. > > > > > > > > It's actually DISCOURAGED ;) to access Variables at parse time - > though > > > we > > > > have an experimental feature to make it efficient and we discussed > > > whether > > > > to treat it as "good practice" and there was strong opposition to > that > > > .... > > > > So I am not actually sure what will be our position on it. > > > > > > > > On Fri, Jun 14, 2024 at 12:50 PM Ash Berlin-Taylor <[email protected]> > > > wrote: > > > > > > > > > > > > > > > > > > > > On 14 Jun 2024, at 10:22, Jarek Potiuk <[email protected]> wrote: > > > > > > > > > > > > think in the future of Airflow 3 where > > > > > > we will have task isolation, having `0` for all the DAGs will be > a > > > > > > prerequisite for switching to "task isolation" mode and this > could > > be > > > > > > actually verified in a migration tool. > > > > > > > > > > I think we still need to enable the ability for DAGs at parse time > to > > > > > access Variables. > > > > > > > > > > Or at least I am not proposing we remove that ability. (I wouldn’t > be > > > > > against it, but I was planning on continuing to support that for > now) > > > > > > > > > > -ash > > > > > > > > > >
