kaxil commented on PR #74188: URL: https://github.com/apache/airflow/pull/74188#issuecomment-5999849036
Closing in favour of #74283. The cause you found is the right one. The problem is `get_flat_relatives` runs for every task in every Dag, so each task below any `SkipMixin` task does its own XCom query on each scheduling pass. I measured a plain unmapped Dag (a short-circuit and 1999 tasks after it): 1999 queries per pass instead of 1, 0.64 s instead of 0.029 s on SQLite. #74283 only reads skip decisions for tasks inside a mapped task group, with one query per pass, and your branch + join case is one of its tests. If it doesn't land we can reopen this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
