nevcohen commented on PR #46237:
URL: https://github.com/apache/airflow/pull/46237#issuecomment-2623665005

   > > Hi, looks good!
   > > 
   > > I would love to understand why it makes a difference if it runs every 15 
minutes or every hour (for example)?
   > > 
   > > How does this affect the worker performance?
   > 
   > Thanks for taking a look!
   > 
   > I personally wouldn't think it'd affect worker performance at all, since 
all they do is write to that file system. If anything it could see it improving 
because the volumes are not busy listing files all the time. But then again, 
I'm not super familiar with all the different types of setups Airflow can be 
run in.
   > 
   > In our case, extensive pruning on that shared volume was a) a major 
performance hog - we're dealing with Azure Blob Storage as backend, which is 
fairly slow to do a recursive `find` looking for last modified times and b) a 
major cost factor (`ListBlob` transactions get pricey). And our Airflow setup 
is not that large, maybe ~100 DAGs or so. 
   > 
   > I broke one of the helm chart unit tests in the process here it seems. 
Will be attempting to deal with that this morning, but I need a working local 
setup first :)
   
   Got it, thanks!  I would love to do CR for you :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to