Hi Vikram, Thanks for pointing that out, 'task latency',
"we define task latency as the time it takes for a task to begin executing > once its dependencies have been met." It will be great if you can elaborate more about "begin executing" and how you calculate "its dependencies have been met.". If the 'begin executing' means the state of ti becomes running, then the 'Scheduling Delay' metric focuses on the overhead introduced by the scheduler. In our prod and stress test, we use the `task_instance_audit` table ( a new row is created whenever there is state change in task_instance table) to compute the time of a ti should be scheduled. Thanks, Ping On Wed, Jun 8, 2022 at 11:25 AM Vikram Koka <vik...@astronomer.io.invalid> wrote: > Ping, > > I am quite interested in this topic and trying to understand the > difference between the "scheduling delay" metric articulated as compared to > the "task latency" aka "task lag" metric which we have been using before. > > As you may recall, we have been using two specific metrics to > benchmark Scheduler performance, specifically "task latency" and "task > throughput" since Airflow 2.0. > These were described in the 2.0 Scheduler blog post > <https://www.astronomer.io/blog/airflow-2-scheduler/> > Specifically, within that we defined task tatency as the time it takes for > the task to begin executing once it's dependencies are all met. > > Thanks, > Vikram > > > > > On Wed, Jun 8, 2022 at 10:25 AM Ping Zhang <pin...@umich.edu> wrote: > >> Hi Airflow Community, >> >> Airflow is a scheduling platform for data pipelines, however there is no >> good metric to measure the scheduling delay in the production and also the >> stress test environment. This makes it hard to catch regressions in the >> scheduler during the stress test stage. >> >> I would like to propose an airflow scheduling delay metric >> definition. Here is the detailed design of the metric and its >> implementation: >> >> >> https://docs.google.com/document/d/1NhO26kgWkIZJEe50M60yh_jgROaU84dRJ5qGFqbkNbU/edit?usp=sharing >> >> Please take a look and any feedback is welcome. >> >> Thanks, >> >> Ping >> >>
