potiuk commented on PR #30259: URL: https://github.com/apache/airflow/pull/30259#issuecomment-1481717401
> multiprocessing.Manager() also supports remote calls over the network, but then it's kind of moot because we replace a network call with an other. Actually if we could have some sort of caching work _only_ at dag parsing time, I think that'd already be pretty helpful for the many users who don't follow the best practices. Possibly - but it very much depends on the DAGs they have. If you have big dag with many tasks that are small-ish, the whole DAG is parsed every time the task is being executed (actually in certain situations it happens twice- for example when there is impersonation involved). Once/Twice for every single task execution. If you have dynamically mapped tasks - it happens every time for every instance of mapped task. So the advice 'don't do anything heavy in DAG` IS actually a helpful (and serious) advice. BTW. When it comes to savings on secrets, the Lookup patterns by @vincbeck are a very good solution https://github.com/apache/airflow/pull/29580 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
