uranusjr commented on PR #75404: URL: https://github.com/apache/airflow/pull/75404#issuecomment-6359883233
An alternative we could do is skip the reference check for already-active assets. `asset_active` cascades from `asset` via its fk, and the orphan DELETE runs first in the same transaction, so whatever is left in `asset_active` is already referenced. The active set can be a plain SELECT on `asset_active`. The inactive query can do the NOT EXISTS against `asset_active` first, so the four reference EXISTS checks only run on the few rows left. That also gets rid of the CTEs entirely. I’m perfectly not sure if this worth doing; the query would boot the performance a lot against inactive assets (those that used to exist but no longer in Dag files), but would incur a little compensation against orphan and active ones, so whether it’s worthwhile very much depends on how we feel Airflow deployments tend to be like in the wild. It would also rely on the ordering of operations (DELETE before activation) and the fk. Both should hold going forward, and we can add a comment to document this. If we want to do this, it should probably be a separate follow-up PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
