Sorry for the double-post, I forgot to answer your questions.
Arnaud
Le 09/02/2010 17:31, Arnaud Brand a écrit :
Hi James,
Sorry to bother you again, I think I found the problem but need
confirmation/advice.
It appears that A wants to send through SSH more data to B that B can
accept.
In fact, B is in the process of committing the zfs recv of a big
snapshot, but A still has some snaps to send (zfs send -R).
After exactly 6 minutes, B sends a RST to A and both close their
connections.
Sadly the receive operation goes away with the ssh session.
Whats looks strange is that B doesn't reduce its window, it just keeps
ACKing the last byte SSH could eat and leaves the window at 64436.
I guess it's related to SSH channel multiplexing features : it can't
reduce the window or other channels won't make progress either, am I
right here ?
I did some ndd lookups in /dev/tcp and /dev/ip to see if I can find a
timeout to raise, but found nothing matching the 6 minutes.
The docs related to tcp/ip tunables I found on sun's website didn't
bring me much further, I found nothing that seemed applicable.
I agree my case is perhaps a bit overstretched and I'm going generate
the replication stream in a local file and send it over "by hand".
Later I shouldn't have snapshots that are that big and I wish zfs recv
wouldn't block for so long either, but still I'm asking myself if the
behavior I'm observing is correct or if it's the sign of some
misconfiguration on my part (or perhaps I've once more forgotten how
things work).
Sorry for my bad english, I hope you understood what I meant.
Just in case I've attached the tcpdump output of the ssh session
starting at the very last packet that is accepted and acked by B.
A is 192.0.2.2 and B is 192.0.2.1.
If you could shed some light on this case I'd be very grateful, but I
don't want to bother you.
Thanks,
Arnaud
Le 09/02/2010 14:39, James Carlson a écrit :
Arnaud Brand wrote:
Le 08/02/10 23:18, James Carlson a écrit :
Causes for RST include:
- peer application is intentionally setting the linger time to
zero
and issuing close(2), which results in TCP RST generation.
Might be possible, but I can't see why the receiving end would do that.
No idea, but a debugger on that side might be able to detect something.
- bugs in one or both peers (often related to TCP keepalive; key
signature of such a problem is an apparent two-hour time limit).
That could be it, but I doubt it since disconnections appeared anywhere
randomly in the range 10 minutes to 13 hours.
It should be noted that the node sending the RST keeps the connection
open (netstat -a shows its still established).
To be honest that puzzles me.
That sounds horrible. There's no way a node that still has state for
the connection should be sending RST. Normal procedure is to generate
RST when you do _not_ have state for the connection or (if you're
intentionally aborting the connection) to discard the state at the same
time you send RST.
That points to either a bug in the peer's TCP/IP implementation or one
of the causes that you've dismissed (particularly either a duplicate IP
address or a firewall/NAT problem).
It was with netcat, not ssh.
I was indeed very surprised to see the connection still "established"
instead of "listening"
You (at least) have to analyze the packet sequences to determine
what is
going wrong. Depending on the nature of the problem, it may also take
in-depth kernel debugging on one or both peers to locate the cause.
I relaunched another transfer and I'm tcpdumping both servers in the
hope that I find something.
In the mean time I've received a beta bios from tyan which provides
support for IKVM over tagged VLANs.
Until now the intel chips (on which the IKVM/IPMI card is piggy-backed)
are working better than before.
I can't tell if it's related or not, I'm crossing fingers.
That could EASILY be related. That was key information to include.
IPMI, as I recall, hijacks the node's Ethernet controller to provide
low-level node control service. From what I remember out of ARC
reviews, the architecture is pretty brutal^W"clever".
I wouldn't be surprised in the least if this is the problem. I like the
idea of remote management, but that's the sort of thing I'd never enable
on my systems ...
Sadly, I can't disable it.
The VLAN setting didn't change anything.
Regarding kernel debugging I though I would look for dtrace scripts,
and
found some, but nothing that seemed relevant in my case.
As I a complete beginner (read: copy-paste) in dtrace I couldn't yet
figure out how to write one myself.
Is the remote end that's generating RST also OpenSolaris?
Yes, they were both snv_132.
I reproduced the disconnection with snv_131.
I think I still have a snv_130 be, so I could give it a try too.
_______________________________________________
networking-discuss mailing list
[email protected]
