elaye-canopy commented on issue #34995:
URL: https://github.com/apache/airflow/issues/34995#issuecomment-1880016235

   @he-gong-sardine 
   Great to see this is worked on, we independently solved it on our side and I 
put a comment in the code about that.
   
   Your solution does seem to work but seems to have led to significant 
slowdown when we tested it on files > 60gb.
   I put a comment with a snippet of the code we used to solve it, using a 
Paramiko maintained function (getfo) which uses Paramiko's prefetch to solve 
the performence issues.
   We have a dag downloading 4 files simultaneously (10GB, 16GB, 60GB, and 
166GB) so a slowdown would have meant we would keep doing it all day long.
   
   The file is in a private repo currently but if you want my version of the 
fix, to make it PR ready, feel free to contact me and I'll get it to you. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to