olegkachur-e commented on PR #49633: URL: https://github.com/apache/airflow/pull/49633#issuecomment-2854338838
Hi @razeghi71! Thank you for submitting the PR, I have a couple of doubts regarding the solution provided: 1) Are we sure that the library code called in ThreadPoolExecutor is thread safe? 2) We multiply requests per second by number of workers, at some point it can trigger the rate limiter on the GCS service side, and make the whole clean-up even longer. There is an [delete_blobs](https://github.com/googleapis/python-storage/blob/8f56cc812bfb81e7b483eb205bddc1cb1d42cc2f/google/cloud/storage/bucket.py#L1812) method in the GCS client that meant to be run for multiple objects, but it calls `delete_blob` internally, still handles some validation. To address the initial problem of speeding up the deletion process we can take a look at [bulk deletion](https://cloud.google.com/storage/docs/deleting-objects#delete-objects-in-bulk). I think the current implementation of the operator is not optimal for a large number of objects, so there is a good chance to improve it. As we have a way to delete in bulk, we might not need ThreadPoolExecutor run anymore. @razeghi71 WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
