Asquator commented on issue #45636: URL: https://github.com/apache/airflow/issues/45636#issuecomment-3101412322
This is how I imagine testing this proposal: After we have a fully-working query, with the relevant fields in the DB, we should reproduce the case nevcohen described in the issue (the one we really faced in production) and similar cases, with different limits starved anywhere and measure the following metrics: (1) scheduled TI's per CS loop - max_tis - # starved TIs ( = max_tis with new logic) (2) # scheduled TI's per minute (3) # scheduler loops per minute (4) average scheduler throughput - (2) / (3) For the control group, we can schedule tasks without any limits and check how much the new logic slows down the fetching, with different max_tis values. In the end, I think we need a variety of test cases with different workload styles, to mimic the average impact on performance. I think we could use a slightly modified performance DAG from AIP-59 to build a varied benchmark workflow. The main point is to prove that under reasonable assumptions (like some limits on concurrency, even the default ones): In a busy workload, the new logic boosts the scheduler's throughput. In a light workload, the nested windowing doesn't significantly slow down the computation. Cross-posted: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
