OK, I have a tcpdump of some LVS packets. Immediately after receiving
this, the backup goes to 100% S.I.
A few notes that may help:
172.26.64.76 is the external machine I am testing from
192.168.148.2 is the VIP
10.17.192.19 is the private-side address of master
10.17.192.20 is the private-side address of backup
I notice that the packet dump contains multiple references to the
*same* connection. Is that normal?
This problem doesn't happen with HTTP with this small number of
connections. I suspect that may be because my HTTP tests have a lot
less packets per connection.
I triggered this with four simultaneous connections - but only on my
third attempt.
On 13 September 2010 16:47, JL <[email protected]> wrote:
> On 13 September 2010 16:25, Simon Horman <[email protected]> wrote:
>> On Mon, Sep 13, 2010 at 03:52:29PM +0100, JL wrote:
>>> OK, More information:
>>>
>>> I hope someone who is up on ipvs kernel side is listening!
>>
>> I am listening, sorry for not responding earlier.
> Thanks. I was getting nervous :)
>
>>> If a backup machines receives an IPVS state update packet (the ones
>>> sent to 224.0.0.81) with a certain number of connections in it
>>> (somewhere between two and eight, inclusive, will trigger it) then SI
>>> goes to 100% on the backup immediately.
>>>
>>> Firewalling 224.0.0.81 insulates you from the problem (although, of
>>> course, is unsuitable for a live deployment).
>>
>> Presumably turning off connection synchronisation
>> has the same effect.
>
>
>>> Feeding in only one connection at a time (slowly enough that the each
>>> have their own IPVS packet) doesn't trigger the problem.
>>
>> So it occurs if the number of synchronised connections in
>> a single packet is between 2 and 8. So 1 is ok, and so is 9?
> No, one is ok, but somewhere between 2 and 8 this problem begins, and
> anything higher has the problem. I just haven't been able to narrow
> down the number any tighter than that.
>
> However, some more testing indicates that it is not that straight-forward.
>
> If I trigger it by pressing reload in the browser (which kicks off
> about 9 HTTPS connections) I get the problem - If I put those same
> gets into a bash script, and get them all at once, then it doesn't.
>
> I'm still trying to simplify the problem down to a simple script I can run.
>
> This is a two-node LVS/RS system. I have found that if none the
> connections in the state packet are to backup machine, then it doesn't
> trigger this problem.
>
>>> This happens with linux 2.6.35.4, but not 2.6.27.45.
>>
>> That is a fairly wide number of kernel versions.
>> But if it is easy to reproduce then it should be fairly easy to track down.
> That was the intent of coming up with a simple test - that I could try
> a number of different kernel versions, and see where the problem
> appears.
>
>
> Investigation is ongoing...
>
> Thanks,
> --
> Jarrod Lowe
>
--
Jarrod Lowe
No. Time Source Destination Protocol Info
5 55.000000 10.17.192.19 224.0.0.81 IPVS
Frame 5 (358 bytes on wire, 358 bytes captured)
Arrival Time: Sep 13, 2010 17:31:30.854317000
[Time delta from previous captured frame: 19.000045000 seconds]
[Time delta from previous displayed frame: 19.000045000 seconds]
[Time since reference or first frame: 55.000000000 seconds]
Frame Number: 5
Frame Length: 358 bytes
Capture Length: 358 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:udp:ipvs]
[Coloring Rule Name: UDP]
[Coloring Rule String: udp]
Ethernet II, Src: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb), Dst:
IPv4mcast_00:00:51 (01:00:5e:00:00:51)
Destination: IPv4mcast_00:00:51 (01:00:5e:00:00:51)
Address: IPv4mcast_00:00:51 (01:00:5e:00:00:51)
.... ...1 .... .... .... .... = IG bit: Group address
(multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb)
Address: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..1. .... .... .... .... = LG bit: Locally administered address
(this is NOT the factory default)
Type: IP (0x0800)
Internet Protocol, Src: 10.17.192.19 (10.17.192.19), Dst: 224.0.0.81
(224.0.0.81)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 344
Identification: 0x458d (17805)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 1
Protocol: UDP (0x11)
Header checksum: 0x8892 [correct]
[Good: True]
[Bad : False]
Source: 10.17.192.19 (10.17.192.19)
Destination: 224.0.0.81 (224.0.0.81)
User Datagram Protocol, Src Port: 35032 (35032), Dst Port: 8848 (8848)
Source port: 35032 (35032)
Destination port: 8848 (8848)
Length: 324
Checksum: 0xabcb [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
IP Virtual Services Sync Daemon
Connection Count: 13
Synchronization ID: 101
Size: 316
Connection #1
Reserved: 0x00
Protocol: TCP (0x06)
Client Port: 52272
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0083
Connection Type: Direct Routing
No Output Packets
State: Unknown (0x0001)
Connection #2
Reserved: 0x00
Protocol: TCP (0x06)
Client Port: 52275
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0083
Connection Type: Direct Routing
No Output Packets
State: Unknown (0x0001)
Connection #3
Reserved: 0x00
Protocol: TCP (0x06)
Client Port: 52276
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0083
Connection Type: Direct Routing
No Output Packets
State: Unknown (0x0001)
Connection #4
Reserved: 0x81
Protocol: TCP (0x06)
Client Port: 52273
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0083
Connection Type: Direct Routing
No Output Packets
State: Unknown (0x0001)
Connection #5
Reserved: 0x01
Protocol: TCP (0x06)
Client Port: 52275
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Output (0x0004)
Connection #6
Reserved: 0xff
Protocol: TCP (0x06)
Client Port: 52275
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Unknown (0x0006)
Connection #7
Reserved: 0x11
Protocol: TCP (0x06)
Client Port: 52277
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0083
Connection Type: Direct Routing
No Output Packets
State: Unknown (0x0001)
Connection #8
Reserved: 0x48
Protocol: TCP (0x06)
Client Port: 52272
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Output (0x0004)
Connection #9
Reserved: 0x31
Protocol: TCP (0x06)
Client Port: 52272
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Unknown (0x0006)
Connection #10
Reserved: 0x48
Protocol: TCP (0x06)
Client Port: 52278
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0083
Connection Type: Direct Routing
No Output Packets
State: Unknown (0x0001)
Connection #11
Reserved: 0x89
Protocol: TCP (0x06)
Client Port: 52276
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Output (0x0004)
Connection #12
Reserved: 0x2a
Protocol: TCP (0x06)
Client Port: 52276
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Unknown (0x0006)
Connection #13
Reserved: 0xf5
Protocol: TCP (0x06)
Client Port: 52279
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0083
Connection Type: Direct Routing
No Output Packets
State: Unknown (0x0001)
No. Time Source Destination Protocol Info
6 55.999885 10.17.192.19 224.0.0.81 IPVS
Frame 6 (142 bytes on wire, 142 bytes captured)
Arrival Time: Sep 13, 2010 17:31:31.854202000
[Time delta from previous captured frame: 0.999885000 seconds]
[Time delta from previous displayed frame: 0.999885000 seconds]
[Time since reference or first frame: 55.999885000 seconds]
Frame Number: 6
Frame Length: 142 bytes
Capture Length: 142 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:udp:ipvs]
[Coloring Rule Name: UDP]
[Coloring Rule String: udp]
Ethernet II, Src: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb), Dst:
IPv4mcast_00:00:51 (01:00:5e:00:00:51)
Destination: IPv4mcast_00:00:51 (01:00:5e:00:00:51)
Address: IPv4mcast_00:00:51 (01:00:5e:00:00:51)
.... ...1 .... .... .... .... = IG bit: Group address
(multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb)
Address: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..1. .... .... .... .... = LG bit: Locally administered address
(this is NOT the factory default)
Type: IP (0x0800)
Internet Protocol, Src: 10.17.192.19 (10.17.192.19), Dst: 224.0.0.81
(224.0.0.81)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 128
Identification: 0x458e (17806)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 1
Protocol: UDP (0x11)
Header checksum: 0x8969 [correct]
[Good: True]
[Bad : False]
Source: 10.17.192.19 (10.17.192.19)
Destination: 224.0.0.81 (224.0.0.81)
User Datagram Protocol, Src Port: 35032 (35032), Dst Port: 8848 (8848)
Source port: 35032 (35032)
Destination port: 8848 (8848)
Length: 108
Checksum: 0xaaf3 [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
IP Virtual Services Sync Daemon
Connection Count: 4
Synchronization ID: 101
Size: 100
Connection #1
Reserved: 0x00
Protocol: TCP (0x06)
Client Port: 52273
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Output (0x0004)
Connection #2
Reserved: 0x00
Protocol: TCP (0x06)
Client Port: 52280
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0083
Connection Type: Direct Routing
No Output Packets
State: Unknown (0x0001)
Connection #3
Reserved: 0x00
Protocol: TCP (0x06)
Client Port: 52273
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Unknown (0x0006)
Connection #4
Reserved: 0x50
Protocol: TCP (0x06)
Client Port: 52277
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Output (0x0004)
No. Time Source Destination Protocol Info
7 56.999908 10.17.192.19 224.0.0.81 IPVS
Frame 7 (70 bytes on wire, 70 bytes captured)
Arrival Time: Sep 13, 2010 17:31:32.854225000
[Time delta from previous captured frame: 1.000023000 seconds]
[Time delta from previous displayed frame: 1.000023000 seconds]
[Time since reference or first frame: 56.999908000 seconds]
Frame Number: 7
Frame Length: 70 bytes
Capture Length: 70 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:udp:ipvs]
[Coloring Rule Name: UDP]
[Coloring Rule String: udp]
Ethernet II, Src: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb), Dst:
IPv4mcast_00:00:51 (01:00:5e:00:00:51)
Destination: IPv4mcast_00:00:51 (01:00:5e:00:00:51)
Address: IPv4mcast_00:00:51 (01:00:5e:00:00:51)
.... ...1 .... .... .... .... = IG bit: Group address
(multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address
(factory default)
Source: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb)
Address: fa:d1:dd:70:50:eb (fa:d1:dd:70:50:eb)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..1. .... .... .... .... = LG bit: Locally administered address
(this is NOT the factory default)
Type: IP (0x0800)
Internet Protocol, Src: 10.17.192.19 (10.17.192.19), Dst: 224.0.0.81
(224.0.0.81)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 56
Identification: 0x458f (17807)
Flags: 0x04 (Don't Fragment)
0... = Reserved bit: Not set
.1.. = Don't fragment: Set
..0. = More fragments: Not set
Fragment offset: 0
Time to live: 1
Protocol: UDP (0x11)
Header checksum: 0x89b0 [correct]
[Good: True]
[Bad : False]
Source: 10.17.192.19 (10.17.192.19)
Destination: 224.0.0.81 (224.0.0.81)
User Datagram Protocol, Src Port: 35032 (35032), Dst Port: 8848 (8848)
Source port: 35032 (35032)
Destination port: 8848 (8848)
Length: 36
Checksum: 0xaaab [validation disabled]
[Good Checksum: False]
[Bad Checksum: False]
IP Virtual Services Sync Daemon
Connection Count: 1
Synchronization ID: 101
Size: 28
Connection #1
Reserved: 0x00
Protocol: TCP (0x06)
Client Port: 52277
Virtual Port: 443
Destination Port: 443
Client Address: 172.16.64.76 (172.16.64.76)
Virtual Address: 192.168.148.2 (192.168.148.2)
Destination Address: 10.17.192.20 (10.17.192.20)
Flags: 0x0183
Connection Type: Direct Routing
No Output Packets
Connection Not Established
State: Unknown (0x0006)
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - [email protected]
Send requests to [email protected]
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
