potiuk opened a new issue, #50429: URL: https://github.com/apache/airflow/issues/50429
### Body Hey @gopidesupavan - I think we can remove all the parallel stuff from s3/gh syncing. From what I understand `aws s3 cp` and `aws s3 sync` already support parallel uploads and downloads and we can even control the parallelism there - so I think we can significantly simplify our syncing code - by not spawning multiple processes in pools. I think what we have now even causes some contentions and occasional errors when we run the cli commands in parallel - see https://github.com/apache/airflow-site-archive/actions/runs/14936817797/job/41966102337#step:11:402 and https://github.com/apache/airflow-site-archive/actions/runs/14936549582/job/41965209960 for example. WDYT? Also it woudl be nice to enable progress - I think both `cp` and `sync` can show progress while doing stuff. ### Committer - [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
