ashb commented on issue #27449: URL: https://github.com/apache/airflow/issues/27449#issuecomment-1303620967
Such a change shouldn't be made without measuring the performance before and after. Yes removing it would make things simpler, but I think a tiny bit of complexity is worth it for the speed boost. (I can't remember the figures off the top of my head, but I know we measured it before adding this, and it did make a difference to the "inter-task lag") The place where this change really helps is when you have a large number (500-1000+) of DAGs, as this mini scheduler only locks the _current_ dag, leaving the scheduler on to do other DAGs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
