GitHub user potiuk added a comment to the discussion: Scheduler performance with large number of mapped task instances
Interesting discussion. I converted issue to discussion due to that. One of the aims of Airlfow 3 is to heavily limit overhead for running many small(ish) tasks. I personally call it a "stream-ish" experience. I suggest you save your tests and repeat it when Airflow 3 rc will be out (in 2-3 months}. Alternatively I invite you to look and comment and review changes that are coming as part of https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-72+Task+Execution+Interface+aka+Task+SDK. that mainly aims to achieve that goal. Any input, feedback, PR review, comment on details of it are welcome - look for https://github.com/orgs/apache/projects/405 project where the work is tracked. GitHub link: https://github.com/apache/airflow/discussions/46044#discussioncomment-11954529 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
