potiuk commented on PR #30259:
URL: https://github.com/apache/airflow/pull/30259#issuecomment-1481717401

   > multiprocessing.Manager() also supports remote calls over the network, but 
then it's kind of moot because we replace a network call with an other. 
Actually if we could have some sort of caching work _only_ at dag parsing time, 
I think that'd already be pretty helpful for the many users who don't follow 
the best practices.
   
   Possibly - but it very much depends on the DAGs they have. If you have big 
dag with many tasks that are small-ish, the whole DAG is parsed every time the 
task is being executed (actually in certain situations it happens twice- for 
example when there is impersonation involved). Once/Twice for every single task 
execution. If you have dynamically mapped tasks - it happens every time for 
every instance of mapped task.
   
   So the advice 'don't do anything heavy in DAG` IS actually a helpful (and 
serious) advice. 
   
   BTW. When it comes to savings on secrets, the Lookup patterns by @vincbeck 
are a very good solution https://github.com/apache/airflow/pull/29580
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to