This (rekey problem) is how it is being in solved in various libraries. I wanted to know what logic apache-mina uses ?
1. Dropbear SSH server --> It disallows sending of packets during kex rekeying. During rekeying, it enqueues packets to a linked list. No limit is enforced (except by available memory). 2. OpenSSH --> The OpenSSH implementation is similar. packet.c enqueues outbound packets in packet_send2() when active_state->rekeying == 1. 3. Paramiko --> Without disabling transmission during re-keying and without the need for a complex (and possibly error-prone) time-based algorithm, would be to initiate the re-keying really early (at 500 MB, for example) and then raise the exception after 1 GB has been received like we do now. Regards K.Basaveswar ----- Original Message ----- From: Basaveswar Kureti <[email protected]> To: "[email protected]" <[email protected]> Cc: Sent: Friday, 26 April 2013 6:41 PM Subject: Re: SFTP File transfer problem at 1 GB transfer (huge file) Thanks for the info Emmanuel Lecharny. This is the issue which I am talking about - https://github.com/paramiko/paramiko/issues/49 Some SFTP libraries do this out of box. At 1GB of data transferred over sftp, paramiko library initiates a ssh rekey request and then sets a limit on the number of packets exchanged before the remote side answers the rekey request. This limit is hard-coded to 20. There is a suggestion on how to tune this number. 1. Determine your bandwidth between end points. Run scp with a large files (>1G) to get a good average. 2. Determine your "round trip time" (rtt). Run ping from one end point to the other. It'll provide info on this. 3. Note that most of the "packets" paramiko is using are 24k. 4. Estimate the number of packets that go out from this side in the time it takes one rekey packet to come back from the other side. Note this is a lower bound since it assumes that the remote side instantly puts a rekey packet out on the wire the moment it receives one. num_packets = (bandwidth_MB * 1024) / 24 * rtt_msec * 1000 / 2 The factor of two gets it down from a round trip to a one-way trip time. The 1024 converts from MB (which bandwidth is in) to KB, which the packet size (24) is in. The 1000 converts from milliseconds (which ping reports for rtt) into second, which everything else is in. 5. Take num_packets and multiply it by some safety margin, at least 2 or 3x. I believe the downside here is that paramiko will buffer more packets the larger the number is, but it's been a while since I read the code. Unless you're heavily memory constrained, you can get away with a large number here. I wanted to know how this is being handled in apache-mina. In one of our developer setups, we consistently see that apache-mina is closing the socket at 1 GB of transfer. In some dev setups this issue is sporadic. This was the exception thrown by apache-mina. Does anyone seen this problem ? Let me know, how to debug this issue. ===== java.io.IOException: An established connection was aborted by the software in your host machine at sun.nio.ch.SocketDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(Unknown Source) at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source) at sun.nio.ch.IOUtil.write(Unknown Source) at sun.nio.ch.SocketChannelImpl.write(Unknown Source) at org.apache.mina.transport.socket.nio.NioProcessor.write(NioProcessor.java:287) at org.apache.mina.transport.socket.nio.NioProcessor.write(NioProcessor.java:44) at org.apache.mina.core.polling.AbstractPollingIoProcessor.writeBuffer(AbstractPollingIoProcessor.java:950) at org.apache.mina.core.polling.AbstractPollingIoProcessor.flushNow(AbstractPollingIoProcessor.java:874) at org.apache.mina.core.polling.AbstractPollingIoProcessor.flush(AbstractPollingIoProcessor.java:799) at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$700(AbstractPollingIoProcessor.java:68) at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1146) at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ===== Regards K.Basaveswar ----- Original Message ----- From: Emmanuel Lecharny <[email protected]> To: Basaveswar Kureti <[email protected]>; [email protected] Cc: Sent: Friday, 26 April 2013 3:47 PM Subject: Re: SFTP File transfer problem at 1 GB transfer (huge file) this is the right mailing list. I guess nobody had any experience with the transfert of 1G files, thus you got no answer. Le 26 avr. 2013 10:11, "Basaveswar Kureti" <[email protected]> a écrit : > Is this the right mailing list for my question ? or Shall I ask in dev > mailing list ? > > > > ----- Original Message ----- > From: Basaveswar Kureti <[email protected]> > To: "[email protected]" <[email protected]> > Cc: > Sent: Thursday, 25 April 2013 8:12 PM > Subject: SFTP File transfer problem at 1 GB transfer (huge file) > > Hi, > > I am using apache mina 2.0.4 version. While I was transfering a huge file > ( 1.5 GB ) from SFTP client to apache-mina server, it was failing at 1 GB > transfer. > I came to know that “1 GB is the point at which the SSH specification > requires the client and the server to perform key re-exchange.” > > Can someone tell me that how the key re-exchange happens in apache-mina ? > Is there any solution to avoid this problem ? > > Regards > K.Basaveswar >
