On Mon, Feb 27, 2017 at 5:01 PM, Sun Paul <paul...@gmail.com> wrote: > Hi, can I confirm that the problem is on the Linux router itself or on > both server and client? > try with a new kernel in router, like build and install a upstream kernel on your route machine.
git://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git https://kernelnewbies.org/KernelBuild : part "Setting up your kernel configuration" and part "Building the kernel" > On Thu, Feb 23, 2017 at 2:07 PM, Xin Long <lucien....@gmail.com> wrote: >> On Thu, Feb 23, 2017 at 1:30 PM, Sun Paul <paul...@gmail.com> wrote: >>> does this fixed in RHEL7? >> yes, I think so. >> >>> >>> On Wed, Feb 22, 2017 at 11:03 AM, Xin Long <lucien....@gmail.com> wrote: >>>> On Wed, Feb 22, 2017 at 10:29 AM, Sun Paul <paul...@gmail.com> wrote: >>>>> Hi Xin >>>>> >>>>> do you mean we need to patch the kernel? >>>> Yups, pls comment on that bz if it's really needed for your env. >>>> A z-stream kernel may be available for that issue soon. >>>> >>>> Thanks. >>>> >>>>> >>>>> >>>>> >>>>> On Wed, Feb 22, 2017 at 10:00 AM, Xin Long <lucien....@gmail.com> wrote: >>>>>> On Wed, Feb 22, 2017 at 9:12 AM, Sun Paul <paul...@gmail.com> wrote: >>>>>>> Hi >>>>>>> >>>>>>> the router is actually is a linux running on RHEL6.8 >>>>>>> (2.6.32-642.4.2.el6.x86_64). it uses iptables to do SNAT aand DNAT >>>>>>> forward. >>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1412038 >>>>>> >>>>>> sctp_manip_pkt->sctp_compute_cksum: >>>>>> struct sctphdr *sh = sctp_hdr(sub); >>>>>> >>>>>> But in rhel6, skb->transport_header is not yet set at that time. >>>>>> This patch should be backported into rhel6. >>>>>> >>>>>> commit 21d1196a35f5686c4323e42a62fdb4b23b0ab4a3 >>>>>> Author: Eric Dumazet <eduma...@google.com> >>>>>> Date: Mon Jul 15 20:03:19 2013 -0700 >>>>>> >>>>>> ipv4: set transport header earlier >>>>>> >>>>>> >>>>>>> >>>>>>> On Tue, Feb 21, 2017 at 11:53 PM, Xin Long <lucien....@gmail.com> wrote: >>>>>>>> On Tue, Feb 21, 2017 at 12:26 PM, Sun Paul <paul...@gmail.com> wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> sorry to get back late, the platform is running on KVM. and this >>>>>>>>> problem is resolved by moving to VMware environment, however, a new >>>>>>>>> problem is coming out, we noticed that the HB REQ is being ABORT from >>>>>>>>> client. >>>>>>>>> >>>>>>>>> >>>>>>>>> 03:32:35.233572 IP 10.165.250.22.3868 > 192.168.2.13.40001: sctp (1) >>>>>>>>> [HB REQ] (from server to sctp router) >>>>>>>>> 03:32:35.233603 IP 192.168.2.14.3868 > 192.168.2.13.40001: sctp (1) >>>>>>>>> [HB REQ] (from sctp router to client) >>>>>>>>> 03:32:35.233852 IP 192.168.2.13.40001 > 192.168.2.14.3868: sctp (1) >>>>>>>>> [ABORT] (from client to sctp router) >>>>>>>>> >>>>>>>>> 03:32:37.928679 IP 10.165.250.22.3868 > 192.168.2.13.40001: sctp (1) >>>>>>>>> [HB REQ] >>>>>>>>> 03:32:37.928717 IP 192.168.2.14.3868 > 192.168.2.13.40001: sctp (1) >>>>>>>>> [HB REQ] >>>>>>>>> 03:32:37.929247 IP 192.168.2.13.40001 > 192.168.2.14.3868: sctp (1) >>>>>>>>> [ABORT] >>>>>>>>> >>>>>>>>> For the above packet flow, 10.165.250.22 is the server and >>>>>>>>> 192.168.2.13 is the client, the server 10.165.250.22 sends HB REQ to >>>>>>>>> client 192.168.2.13 through 192.168.2.14 (the SCTP router), and the >>>>>>>>> SCTP router change the src address before forward the HB REQ to the >>>>>>>>> client. >>>>>>>>> >>>>>>>>> But somehow the client is ABORT the HB REQ, any idea? is it related to >>>>>>>>> the HEARTBEAT information? or the checksum again>? >>>>>>>> The incorrect checksum won't cause ABORT, but the abnormal HB REQ >>>>>>>> could be, if HB information was modified when calculating the checksum >>>>>>>> on router, the ABORT may be caused in client process. >>>>>>>> >>>>>>>> is your SCTP router linux ? if yes, what's the kernel version ? >>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jan 13, 2017 at 9:19 PM, Michael Tuexen >>>>>>>>> <michael.tue...@lurchi.franken.de> wrote: >>>>>>>>>>> On 13 Jan 2017, at 10:43, Michael Tuexen >>>>>>>>>>> <michael.tue...@lurchi.franken.de> wrote: >>>>>>>>>>> >>>>>>>>>>> Your router does NOT change any field in the SCTP packet, but the >>>>>>>>>>> SCTP checksum was modified from >>>>>>>>>>> Checksum: 0xbaea49e5 (not verified) >>>>>>>>>>> to >>>>>>>>>>> Checksum: 0xa9a86d3f (not verified) >>>>>>>>>>> At least one of these is wrong. Read the tracefiles in wireshark and >>>>>>>>>>> enable checksum validation and wireshark will tell you which one is >>>>>>>>>>> correct. (That is why I asked for .pcap file instead of a .txt). >>>>>>>>>>> >>>>>>>>>>> My guess is that the initial checksum is correct and your box >>>>>>>>>>> middlebox >>>>>>>>>>> not only changes the destination address and ttl field and header >>>>>>>>>>> checksum in the IP-header (which is expected) but also incorrectly >>>>>>>>>>> the >>>>>>>>>>> SCTP checksum. Since no field in the SCTP packet has changed, the >>>>>>>>>>> checksum >>>>>>>>>>> must be the same. >>>>>>>>>> At the server have a look at the SNMP counters: >>>>>>>>>> cat /proc/net/sctp/snmp >>>>>>>>>> You should find a line staring with >>>>>>>>>> SctpChecksumErrors >>>>>>>>>> If the number reported there is positive, the node received packets >>>>>>>>>> with checksum errors. >>>>>>>>>> >>>>>>>>>> Best regards >>>>>>>>>> Michael >>>>>>>>>>> >>>>>>>>>>> Best regards >>>>>>>>>>> Michael >>>>>>>>>>>> On 13 Jan 2017, at 04:29, Sun Paul <paul...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Frame 2: 98 bytes on wire (784 bits), 98 bytes captured (784 bits) >>>>>>>>>>>> Encapsulation type: Ethernet (1) >>>>>>>>>>>> Arrival Time: Jan 6, 2017 16:52:49.662321000 Malay Peninsula >>>>>>>>>>>> Standard Time >>>>>>>>>>>> [Time shift for this packet: 0.000000000 seconds] >>>>>>>>>>>> Epoch Time: 1483692769.662321000 seconds >>>>>>>>>>>> [Time delta from previous captured frame: 0.000179000 seconds] >>>>>>>>>>>> [Time delta from previous displayed frame: 0.000179000 seconds] >>>>>>>>>>>> [Time since reference or first frame: 0.000179000 seconds] >>>>>>>>>>>> Frame Number: 2 >>>>>>>>>>>> Frame Length: 98 bytes (784 bits) >>>>>>>>>>>> Capture Length: 98 bytes (784 bits) >>>>>>>>>>>> [Frame is marked: False] >>>>>>>>>>>> [Frame is ignored: False] >>>>>>>>>>>> [Protocols in frame: eth:ethertype:ip:sctp] >>>>>>>>>>>> Ethernet II, Src: Vmware_81:41:6b (00:50:56:81:41:6b), Dst: >>>>>>>>>>>> Vmware_81:a6:a3 (00:50:56:81:a6:a3) >>>>>>>>>>>> Destination: Vmware_81:a6:a3 (00:50:56:81:a6:a3) >>>>>>>>>>>> Address: Vmware_81:a6:a3 (00:50:56:81:a6:a3) >>>>>>>>>>>> .... ..0. .... .... .... .... = LG bit: Globally unique >>>>>>>>>>>> address (factory default) >>>>>>>>>>>> .... ...0 .... .... .... .... = IG bit: Individual address >>>>>>>>>>>> (unicast) >>>>>>>>>>>> Source: Vmware_81:41:6b (00:50:56:81:41:6b) >>>>>>>>>>>> Address: Vmware_81:41:6b (00:50:56:81:41:6b) >>>>>>>>>>>> .... ..0. .... .... .... .... = LG bit: Globally unique >>>>>>>>>>>> address (factory default) >>>>>>>>>>>> .... ...0 .... .... .... .... = IG bit: Individual address >>>>>>>>>>>> (unicast) >>>>>>>>>>>> Type: IPv4 (0x0800) >>>>>>>>>>>> Internet Protocol Version 4, Src: 192.168.206.83, Dst: >>>>>>>>>>>> 192.168.206.66 >>>>>>>>>>>> 0100 .... = Version: 4 >>>>>>>>>>>> .... 0101 = Header Length: 20 bytes (5) >>>>>>>>>>>> Differentiated Services Field: 0x02 (DSCP: CS0, ECN: ECT(0)) >>>>>>>>>>>> 0000 00.. = Differentiated Services Codepoint: Default (0) >>>>>>>>>>>> .... ..10 = Explicit Congestion Notification: ECN-Capable >>>>>>>>>>>> Transport codepoint '10' (2) >>>>>>>>>>>> Total Length: 84 >>>>>>>>>>>> Identification: 0x0000 (0) >>>>>>>>>>>> Flags: 0x02 (Don't Fragment) >>>>>>>>>>>> 0... .... = Reserved bit: Not set >>>>>>>>>>>> .1.. .... = Don't fragment: Set >>>>>>>>>>>> ..0. .... = More fragments: Not set >>>>>>>>>>>> Fragment offset: 0 >>>>>>>>>>>> Time to live: 63 >>>>>>>>>>>> Protocol: SCTP (132) >>>>>>>>>>>> Header checksum: 0x1d3d [validation disabled] >>>>>>>>>>>> [Good: False] >>>>>>>>>>>> [Bad: False] >>>>>>>>>>>> Source: 192.168.206.83 >>>>>>>>>>>> Destination: 192.168.206.66 >>>>>>>>>>>> [Source GeoIP: Unknown] >>>>>>>>>>>> [Destination GeoIP: Unknown] >>>>>>>>>>>> Stream Control Transmission Protocol, Src Port: 50001 (50001), Dst >>>>>>>>>>>> Port: 3868 (3868) >>>>>>>>>>>> Source port: 50001 >>>>>>>>>>>> Destination port: 3868 >>>>>>>>>>>> Verification tag: 0x00000000 >>>>>>>>>>>> [Assocation index: 0] >>>>>>>>>>>> Checksum: 0xa9a86d3f (not verified) >>>>>>>>>>>> >>>>>>>>>>>> INIT chunk (Outbound streams: 3000, inbound streams: 3000) >>>>>>>>>>>> Chunk type: INIT (1) >>>>>>>>>>>> 0... .... = Bit: Stop processing of the packet >>>>>>>>>>>> .0.. .... = Bit: Do not report >>>>>>>>>>>> Chunk flags: 0x00 >>>>>>>>>>>> Chunk length: 52 >>>>>>>>>>>> Initiate tag: 0xe79f40cb >>>>>>>>>>>> Advertised receiver window credit (a_rwnd): 62464 >>>>>>>>>>>> Number of outbound streams: 3000 >>>>>>>>>>>> Number of inbound streams: 3000 >>>>>>>>>>>> Initial TSN: 176990880 >>>>>>>>>>>> IPv4 address parameter (Address: 192.168.206.83) >>>>>>>>>>>> Parameter type: IPv4 address (0x0005) >>>>>>>>>>>> 0... .... .... .... = Bit: Stop processing of chunk >>>>>>>>>>>> .0.. .... .... .... = Bit: Do not report >>>>>>>>>>>> Parameter length: 8 >>>>>>>>>>>> IP Version 4 address: 192.168.206.83 >>>>>>>>>>>> IPv4 address parameter (Address: 192.168.1.83) >>>>>>>>>>>> Parameter type: IPv4 address (0x0005) >>>>>>>>>>>> 0... .... .... .... = Bit: Stop processing of chunk >>>>>>>>>>>> .0.. .... .... .... = Bit: Do not report >>>>>>>>>>>> Parameter length: 8 >>>>>>>>>>>> IP Version 4 address: 192.168.1.83 >>>>>>>>>>>> Supported address types parameter (Supported types: IPv6, >>>>>>>>>>>> IPv4) >>>>>>>>>>>> Parameter type: Supported address types (0x000c) >>>>>>>>>>>> 0... .... .... .... = Bit: Stop processing of chunk >>>>>>>>>>>> .0.. .... .... .... = Bit: Do not report >>>>>>>>>>>> Parameter length: 8 >>>>>>>>>>>> Supported address type: IPv6 address (6) >>>>>>>>>>>> Supported address type: IPv4 address (5) >>>>>>>>>>>> ECN parameter >>>>>>>>>>>> Parameter type: ECN (0x8000) >>>>>>>>>>>> 1... .... .... .... = Bit: Skip parameter and >>>>>>>>>>>> continue >>>>>>>>>>>> processing of the chunk >>>>>>>>>>>> .0.. .... .... .... = Bit: Do not report >>>>>>>>>>>> Parameter length: 4 >>>>>>>>>>>> Forward TSN supported parameter >>>>>>>>>>>> Parameter type: Forward TSN supported (0xc000) >>>>>>>>>>>> 1... .... .... .... = Bit: Skip parameter and >>>>>>>>>>>> continue >>>>>>>>>>>> processing of the chunk >>>>>>>>>>>> .1.. .... .... .... = Bit: Do report >>>>>>>>>>>> Parameter length: 4 >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> To unsubscribe from this list: send the line "unsubscribe >>>>>>>>>>> linux-sctp" in >>>>>>>>>>> the body of a message to majord...@vger.kernel.org >>>>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-sctp" >>>>>>>>> in >>>>>>>>> the body of a message to majord...@vger.kernel.org >>>>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html