elaye-canopy commented on issue #34995: URL: https://github.com/apache/airflow/issues/34995#issuecomment-1880016235
@he-gong-sardine Great to see this is worked on, we independently solved it on our side and I put a comment in the code about that. Your solution does seem to work but seems to have led to significant slowdown when we tested it on files > 60gb. I put a comment with a snippet of the code we used to solve it, using a Paramiko maintained function (getfo) which uses Paramiko's prefetch to solve the performence issues. We have a dag downloading 4 files simultaneously (10GB, 16GB, 60GB, and 166GB) so a slowdown would have meant we would keep doing it all day long. The file is in a private repo currently but if you want my version of the fix, to make it PR ready, feel free to contact me and I'll get it to you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
