I like the proposal of having a way to invalidate the cache manually when changes need to be propagated quickly. I think this is something we can ship without in the first iteration, provided we make the cache disabled by default. Then we could add it as an option in the Airflow CLI, and eventually in the UI, but this should be done in incremental commits in my opinion to keep the change size manageable.
There is an open PR here https://github.com/apache/airflow/pull/30259 I guess we can continue the conversation there ?
