Thanks for the info Emmanuel Lecharny.
This is the issue which I am talking about -
https://github.com/paramiko/paramiko/issues/49
Some SFTP libraries do this out of box.
At 1GB of data transferred over sftp, paramiko library initiates a ssh rekey
request and then sets a limit on the number of packets exchanged before
the remote side answers the rekey request. This limit is hard-coded to
20.
There is a suggestion on how to tune this number.
1. Determine your bandwidth between end points. Run scp with a large
files (>1G) to get a good average.
2. Determine your "round trip time" (rtt). Run ping from one end point
to the other. It'll provide info on this.
3. Note that most of the "packets" paramiko is using are 24k.
4. Estimate the number of packets that go out from this side in the
time it takes one rekey packet to come back from the other side. Note
this is a lower bound since it assumes that the remote side instantly
puts a rekey packet out on the wire the moment it receives one.
num_packets = (bandwidth_MB * 1024) / 24 * rtt_msec * 1000 / 2
The factor of two gets it down from a round trip to a one-way trip time.
The 1024 converts from MB (which bandwidth is in) to KB, which the
packet size (24) is in. The 1000 converts from milliseconds (which ping
reports for rtt) into second, which everything else is in.
5. Take num_packets and multiply it by some safety margin, at least 2
or 3x. I believe the downside here is that paramiko will buffer more
packets the larger the number is, but it's been a while since I read the code.
Unless you're heavily memory constrained, you can get away with a large number
here.
I wanted to know how this is being handled in apache-mina. In one of our
developer setups, we consistently see that apache-mina is closing the socket at
1 GB of transfer. In some dev setups this issue is sporadic.
This was the exception thrown by apache-mina. Does anyone seen this problem ?
Let me know, how to debug this issue.
=====
java.io.IOException: An established connection was aborted by the software in
your host machine
at sun.nio.ch.SocketDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(Unknown Source)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.write(Unknown Source)
at sun.nio.ch.SocketChannelImpl.write(Unknown Source)
at
org.apache.mina.transport.socket.nio.NioProcessor.write(NioProcessor.java:287)
at
org.apache.mina.transport.socket.nio.NioProcessor.write(NioProcessor.java:44)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.writeBuffer(AbstractPollingIoProcessor.java:950)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.flushNow(AbstractPollingIoProcessor.java:874)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.flush(AbstractPollingIoProcessor.java:799)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor.access$700(AbstractPollingIoProcessor.java:68)
at
org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1146)
at
org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
=====
Regards
K.Basaveswar
----- Original Message -----
From: Emmanuel Lecharny <[email protected]>
To: Basaveswar Kureti <[email protected]>; [email protected]
Cc:
Sent: Friday, 26 April 2013 3:47 PM
Subject: Re: SFTP File transfer problem at 1 GB transfer (huge file)
this is the right mailing list.
I guess nobody had any experience with the transfert of 1G files, thus you
got no answer.
Le 26 avr. 2013 10:11, "Basaveswar Kureti" <[email protected]> a écrit :
> Is this the right mailing list for my question ? or Shall I ask in dev
> mailing list ?
>
>
>
> ----- Original Message -----
> From: Basaveswar Kureti <[email protected]>
> To: "[email protected]" <[email protected]>
> Cc:
> Sent: Thursday, 25 April 2013 8:12 PM
> Subject: SFTP File transfer problem at 1 GB transfer (huge file)
>
> Hi,
>
> I am using apache mina 2.0.4 version. While I was transfering a huge file
> ( 1.5 GB ) from SFTP client to apache-mina server, it was failing at 1 GB
> transfer.
> I came to know that “1 GB is the point at which the SSH specification
> requires the client and the server to perform key re-exchange.”
>
> Can someone tell me that how the key re-exchange happens in apache-mina ?
> Is there any solution to avoid this problem ?
>
> Regards
> K.Basaveswar
>
