Dan:
Looks similar to the tcp_rput_data() trace.
0 -> ip_input lport 23 dport 56618
seq 3146400406 ack 1429020323 flags 12
SYN-ACK.
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
This ACK is not for SYN-ACK, else the ack would have been 3146400407.
BTW, why do we always see 5 SYN packets?
-venu
On Thu, 3 Aug 2006, Dan Celidonio wrote:
Venu,
Sorry I had a copy-and-paste error on the last
email. I didn't show the correct ip_input()
probe output. It's now below.
Dan
ip_input()
==========
CPU FUNCTION
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_input lport 23 dport 56618
seq 3146400406 ack 1429020323 flags 12
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
ip_tcp_input()
==============
CPU FUNCTION
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
-----Original Message-----
From: Dan Celidonio <[EMAIL PROTECTED]>
Sent: Aug 3, 2006 7:15 PM
To: venugopal iyer <[EMAIL PROTECTED]>
Cc: [email protected]
Subject: Re: [networking-discuss] problem with GLDV3 bge interface
Venu,
I ran dtrace probes at ip_input() and ip_tcp_input()
I do see the ACK for port 23 (the telnet port).
From the previous probe at tcp_rput_data() this
ACK is not seen.
Dan
ip_input()
==========
CPU FUNCTION
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
ip_tcp_input()
==============
CPU FUNCTION
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020322 ack 0 flags 2
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020347 ack 3487148167 flags 10
0 -> ip_tcp_input lport 56618 dport 23
seq 1429020323 ack 3487148167 flags 18
-----Original Message-----
From: venugopal iyer <[EMAIL PROTECTED]>
Sent: Aug 3, 2006 6:28 PM
To: Dan Celidonio <[EMAIL PROTECTED]>
Cc: [email protected]
Subject: Re: [networking-discuss] problem with GLDV3 bge interface
Dan:
On Thu, 3 Aug 2006, Dan Celidonio wrote:
Hi Venu,
Thanks for your response.
I tried your dtrace scripts for both tcp_conn_request()
and tcp_rput_data() twice on the server node. The first
time I del'ed out of the tcp_rput_data dtrace before the
client terminated the connection. The secind time I let
it go until the connection was terminated by the client.
This test is just a simple telnet.
I see this on the client:
Connected to xxx.xxx.xxx.xx. [actual IP not shown]
Escape character is '^]'.
Connection closed by foreign host.
Looking at the below results it seems that tcp_rput_data()
never sees the ACK sent by the client. However, I do see the
ACK come in from the client in our Streams module before
we send it up the IP stack. We send it up to IP via a
call to ip_input(). I'll need to do more digging.
I'd check ip_tcp_input().
Thanks,
Dan
1st try
=======
[There was just one print in tcp_conn_request()]
That's right, only the SYN comes in here, rest should be in
tcp_rput_data().
CPU FUNCTION
0 -> tcp_conn_request TCP 30002e13600 state -3 suna 0 snxt 0 rnxt 0
lport 52203 dport 23
seq 2476877873 ack 0 flags 2
[There were many prints in tcp_rput_data()]
I removed all prints which didn't have port 23 printed.
You can make the dtrace probe conditional on destination port 23, i.e.
using my prev example, something like:
fbt::tcp_conn_request:entry
/ *(uint16_t *)(((tcph_t *)((uchar_t *)(ipha_t *)((mblk_t *)arg1)->b_rptr +
20))->th_fport) == 23 /
{
....
}
I terminated the dtrace before the client terminated
the connection.
CPU FUNCTION
0 -> tcp_rput_data TCP 30002dfe9c0 state -1 suna
614917337 snxt 614917338 rnxt 2476877874
lport 52203 dport 23
seq 2476877873 ack 0 flags 2
0 -> tcp_rput_data TCP 30002dfe9c0 state -1 suna
614917337 snxt 614917338 rnxt 2476877874
lport 52203 dport 23
seq 2476877873 ack 0 flags 2
0 -> tcp_rput_data TCP 30002dfe9c0 state -1 suna
614917337 snxt 614917338 rnxt 2476877874
lport 52203 dport 23
seq 2476877873 ack 0 flags 2
0 -> tcp_rput_data TCP 30002dfe9c0 state -1 suna
614917337 snxt 614917338 rnxt 2476877874
lport 52203 dport 23
seq 2476877873 ack 0 flags 2
From the above looks like we keep getting SYN (flags = 2). Do you see these
SYN in the snoop as well? Does the other side get the SYN-ACK (and send the
ACK)?
2nd try
=======
[There was just one print in tcp_conn_request()]
CPU FUNCTION
0 -> tcp_conn_request TCP 30002e13600 state -3 suna 0 snxt 0 rnxt
lport 52463 dport 23
seq 2667176845 ack 0 flags 2
[There were many prints in tcp_rput_data()]
I removed all prints which didn't have port 23 printed.
I let it run until the telnet connection was closed by
the client.
CPU FUNCTION
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176845 ack 0 flags 2
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176845 ack 0 flags 2
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176845 ack 0 flags 2
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176845 ack 0 flags 2
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176870 ack 4250440928 flags 10
this probably causes bad_ack... since the ack is greater than tcp_snxt.
(Look at the comments in tcp_rput_data, LL12779-12790).
-venu
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176846 ack 4250440928 flags 18
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176870 ack 4250440928 flags 10
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176846 ack 4250440928 flags 18
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176870 ack 4250440928 flags 10
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176846 ack 4250440928 flags 18
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176870 ack 4250440928 flags 10
0 -> tcp_rput_data TCP 30002dfd100 state -1 suna
774266961 snxt 774266962 rnxt 2667176846
lport 52463 dport 23
seq 2667176846 ack 4250440928 flags 18
-----Original Message-----
From: venugopal iyer <[EMAIL PROTECTED]>
Sent: Aug 3, 2006 2:35 PM
To: Dan Celidonio <[EMAIL PROTECTED]>
Cc: [email protected]
Subject: Re: [networking-discuss] problem with GLDV3 bge interface
[...]
Some of the problems seen on the server using the
newly converted load balancer driver for bge GLDV3
interface are below. We set tcp_trace and tcp_debug and use
strace to see these errors on the server node.
1. The SYN packet coming from the client into the
traffic cop node and forwarded to the server node
is getting an "unacceptable sequence number gap"
error from the TCP layer on the server.
2. TCP Duplicate segment counter increments
from the TCP layer on the server.
3. TCP Duplicate Ack counter increments
from the TCP layer on the server.
4. "bad_ack" in SYN_RCVD state.
To get a sense for what's happening in TCP, I'd use dtrace in
tcp_conn_request() and tcp_rput_data() and print out the tcp_t
(get this from the conn_t in arg0) along with its state, snxt,
suna and rnxt. Additionally, also print out the TCP header from
the incoming packet (arg1), the seq, ack and flags, something
like:
fbt::tcp_conn_request:entry
{
connp = (conn_t *)arg0;
tcp = (tcp_t *)connp->conn_tcp;
ipha = (ipha_t *)((mblk_t *)arg1)->b_rptr;
/* 20 - assuming no IP options etc. */
tcph = (tcph_t *)((uchar_t *)ipha + 20);
printf("\tTCP %p state %d suna %u snxt %u rnxt %u\n",
tcp, tcp->tcp_state, tcp->tcp_suna, tcp->tcp_snxt,
tcp->tcp_rnxt);
printf("\tlport %u dport %u\n", *(uint16_t *)tcph->th_lport,
*(uint16_t *)tcph->th_fport);
printf("\tseq %u ack %u flags %x\n",
*(uint32_t *)tcph->th_seq, *(uint32_t *)tcph->th_ack,
*tcph->th_flags);
}
similarly for tcp_rput_data.
(on an x86 machine you would need to print out the TCP header content
taking endianness into consideration).
I suppose you could restrict the dtrace probes to the interested port.
Ordinarily, I'd expect the SYN to come into tcp_conn_request(),
which will result in sending the SYN-ACK and transitioning to
SYN-RCVD. The ACK from the client will be processed in tcp_rput_data.
You can check the code in tcp_rput_data() that result in the
messages/conditions that you list above.
Look at:
http://cvs.opensolaris.org/source/xref/on/usr/src/uts/common/inet/tcp.h
for the TCP structure (tcp_t) and TCP header structure (tcph_t)
Look at:
http://cvs.opensolaris.org/source/xref/on/usr/src/uts/common/inet/tcp/tcp.c
for tcp_conn_request() and tcp_rput_data() - L12764-12789, L12903-12961 etc.
-venu
Thanks,
Dan
This message posted from opensolaris.org
_______________________________________________
networking-discuss mailing list
[email protected]
_______________________________________________
networking-discuss mailing list
[email protected]
_______________________________________________
networking-discuss mailing list
[email protected]
