Wow, top-notch work, Brad. I'll happily submit this as a ticket as you suggested and give full credit to you for the sleuthing, unless you'd like to submit it.
On Mon, May 13, 2013 at 6:47 AM, bradleyland <[email protected]> wrote: > Have a look at this Gist: > > https://gist.github.com/bradland/5568217 > > That's the test script I've been using, along with stack traces after the > interrupt. The `when :get` case statement writes progress, so we can see > exactly how many bytes were transferred, plus we get some insight in to the > read loop activity level. Observing the script, we can see the Net::SFTP > read loop stop updating when trying to read 4294049792 bytes. > Operations::Download#on_read updates progress every time a read finishes, > so when it stops, we can assume that something has hung internally. > > In that same Gist, you can see the stack trace from the script when given > SIGINT after it hangs. The resulting stack trace gives further clues that > the problem is actually inside Net::SSH. I tried both Session#download! and > Session#download. Both show the hang > at net-ssh-2.6.7/lib/net/ssh/ruby_compat.rb:30 when given the interrupt. > There's a call to IO#select here, which makes a system call to select(2) > (we've arrived at the threshold of Ruby's C implementations) and returns an > array. If we wanted to keep debugging, we'd want to attach a debugger to > the process and have a look at the stack, focusing on arguments passed to > IO#select. Given the file offset range we're in, a datatype boundary issue > is likely. > > And at this point, you're really traveling down the rabbit hole. I'm an > amateur Rubyist, so we're down to the point where my knowledge withers and > I start grabbing at straws. > > This would make a good ticket for Net::SSH (and maybe Net::SFTP). I'm sure > the authors of those libraries would be interested in seeing what we've > discovered here. If you're not up for it, I'd be happy to take it to them. > If you do open a ticket, be sure to post a link here so we can follow the > progress. > > In the interim, it looks like your best bet for a workaround is shelling > out. > > On Monday, May 13, 2013 12:18:04 AM UTC-4, Chris McCann wrote: > >> I haven't tried the #download method though I could easily do that. I >> was contemplating setting up a local FTP server on my dev machine to try to >> debug this, under the assumption that the file transfer speed would be very >> fast running locally. >> >> I also haven't looked into Net::SSH but I sort of discounted it because >> it seemed that a timeout or other issue there would cause the file transfer >> to not always fail on a 4GB boundary. It fails at exactly the same number >> of bytes each time. Still, there may be something to glean by looking >> there, too. >> >> On Sun, May 12, 2013 at 9:05 PM, bradleyland <[email protected]> wrote: >> >>> Yeah, Ruby Fixnum normally automatically casts to Bignum if you exceed >>> Fixnum boundaries, so it would be an odd case that a reference counter >>> overflow would hang this up. This is really puzzling, and I'm wondering if >>> the bug isn't further down the stack, like Net::SSH or something. >>> >>> I'm curious, have you tried Session#download instead of >>> Session#download!. The former is non-blocking, but won't really have any >>> effect on your operation unless you specifically need it to block until >>> finished. >>> >>> This really sucks to debug! Waiting for a 4 GB file transfer is like >>> waiting for paint to dry. >>> >>> >>> On Sunday, May 12, 2013 11:43:04 PM UTC-4, Chris McCann wrote: >>> >>>> Thanks very much, Brad. I looked through the source to see if I could >>>> find a 32-bit integer somewhere that might be the problem but didn't find >>>> any smoking guns. Let me know if you do. >>>> >>>> On May 12, 2013, at 8:24 PM, bradleyland <[email protected]> wrote: >>>> >>>> A lot of libraries (Ruby and elsewhere) handle large files rather >>>> naively. Fortunately, Net::SFTP looks like it has some smarts for this. The >>>> docs for Session#download! say: "If local [the second argument] is omitted, >>>> downloads the file to an in-memory buffer and returns the result as a >>>> string; otherwise, returns the Net::SFTP::Operations::**Downloa**d >>>> instance." The next place to look is at Download: http://net-ssh.** >>>> githu**b.io/net-sftp/classes/**Net/**SFTP/Operations/Download.**html<http://net-ssh.github.io/net-sftp/classes/Net/SFTP/Operations/Download.html>. >>>> It looks like you've already been there though, because you're using the >>>> event handler. The file is written as the download progresses, so it >>>> doesn't look like it's cramming the whole thing in memory. I also watched >>>> memory usage while downloading a large file, and the ruby process stays >>>> around 30 MB, so I don't think it's an issue related to in-memory >>>> buffering. >>>> >>>> I'm still digging to see if I can find a reason why it would hang at 4 >>>> GB. >>>> >>>> -- >>>> -- >>>> SD Ruby mailing list >>>> [email protected] >>>> >>>> http://groups.google.com/**group**/sdruby<http://groups.google.com/group/sdruby> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "SD Ruby" group. >>>> To unsubscribe from this topic, visit https://groups.google.com/d/**to* >>>> *pic/sdruby/4OhfZ4hklIA/**unsubsc**ribe?hl=en<https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en> >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> sdruby+un...@**googlegroups.com. >>>> >>>> For more options, visit >>>> https://groups.google.com/**grou**ps/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>>> >>>> >>>> -- >>> -- >>> SD Ruby mailing list >>> [email protected] >>> http://groups.google.com/**group/sdruby<http://groups.google.com/group/sdruby> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "SD Ruby" group. >>> To unsubscribe from this topic, visit https://groups.google.com/d/** >>> topic/sdruby/4OhfZ4hklIA/**unsubscribe?hl=en<https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en> >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> sdruby+un...@**googlegroups.com. >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >>> >>> >> >> -- > -- > SD Ruby mailing list > [email protected] > http://groups.google.com/group/sdruby > --- > You received this message because you are subscribed to a topic in the > Google Groups "SD Ruby" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/sdruby/4OhfZ4hklIA/unsubscribe?hl=en. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
