[Kernel-packages] [Bug 1681847] Re: IPVS incorrectly reverse-NATs traffic to LVS host
This was reported as an upstream kernel bug: http://archive.linuxvirtualserver.org/html/lvs-devel/2017-04/msg00014.html An IPVS kernel developer has responded to the issue and a patch has been tested. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Triaged Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-mod
[Kernel-packages] [Bug 1681847] Re: IPVS incorrectly reverse-NATs traffic to LVS host
** Tags added: kernel-bug-reported-upstream -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Triaged Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules-4.4.0-72-generic N/A linux-firmware1.127.23 RfKill: Error: [Errno 2] No
[Kernel-packages] [Bug 1681847] Re: IPVS incorrectly reverse-NATs traffic to LVS host
I've attempted to patch this out by adding checks where snat_handler and dnat_handler are called (ip_vs_core.c and ip_vs_xmit.c), with no success. I have to surmise that either: - My patches aren't being built correctly - My checks don't work - This isn't the code that's mangling the packets - The connection table entries are being marked as MASQ connections even though they're DROUTE The extra checks take the form (referring to the instance in the previous comment): if (pp->snat_handler && (IP_VS_FWD_METHOD(cp) == IP_VS_CONN_F_MASQ) && !pp->snat_handler(skb, pp, cp, iph)) goto drop; Similar guards were put on the dnat_handler calls. This should cause the NAT handler to be ignored unless the connection information (cp) is for a MASQ connection. I should also note, in case it is useful, that in include/uapi/linux/ip_vs.h MASQ connections are marked with a value of 0 in the relevant bits of cp->flags. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2]
[Kernel-packages] [Bug 1681847] Re: IPVS incorrectly reverse-NATs traffic to LVS host
I think I may have tracked this down, but I haven't had a go at patching it yet. In net/netfilter/ipvs/ip_vs_core.c - in handle_response(): /* mangle the packet */ if (pp->snat_handler && !pp->snat_handler(skb, pp, cp, iph)) goto drop; This calls the protocol-specific SNAT handler - however, it doesn't first check that (IP_VS_FWD_METHOD(cp) == IP_VS_CONN_F_MASQ), as is done in a couple of the other netfilter modules. I think this means SNAT is *always* done on response packets, even if the connection information was for a non-NAT connection. I'll update the issue once I've had a go at patching this. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integ
[Kernel-packages] [Bug 1681847] Re: IPVS incorrectly reverse-NATs traffic to LVS host
I've tested this with the latest upstream kernel (4.11.0-041100rc6), and the problem is still present. ** Tags added: kernel-bug-exists-upstream -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules-4
[Kernel-packages] [Bug 1681847] Re: IPVS incorrectly reverse-NATs traffic to LVS host
It's hard to tell whether this was the result of an upgrade - I think this behaviour has always been present in the 14.04 stock kernels, but we noticed it less until recently. I'll look into testing the latest upstream kernel and get back to you. If I get a chance I'll also try to look through the kernel source where we know the problem exists - I expect it relates to the code that handles the return path for NAT (masq) services, in relation to how it treats entries in the IPVS connection table. Thanks -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320
[Kernel-packages] [Bug 1681847] ProcEnviron.txt
apport information ** Attachment added: "ProcEnviron.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860195/+files/ProcEnviron.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules-4
[Kernel-packages] [Bug 1681847] UdevDb.txt
apport information ** Attachment added: "UdevDb.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860198/+files/UdevDb.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules-4.4.0-72-gen
[Kernel-packages] [Bug 1681847] ProcCpuinfo.txt
apport information ** Attachment added: "ProcCpuinfo.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860194/+files/ProcCpuinfo.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules-4
[Kernel-packages] [Bug 1681847] ProcInterrupts.txt
apport information ** Attachment added: "ProcInterrupts.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860196/+files/ProcInterrupts.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-mod
[Kernel-packages] [Bug 1681847] ProcModules.txt
apport information ** Attachment added: "ProcModules.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860197/+files/ProcModules.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules-4
[Kernel-packages] [Bug 1681847] WifiSyslog.txt
apport information ** Attachment added: "WifiSyslog.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860200/+files/WifiSyslog.txt ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-
[Kernel-packages] [Bug 1681847] CurrentDmesg.txt
apport information ** Attachment added: "CurrentDmesg.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860192/+files/CurrentDmesg.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules
[Kernel-packages] [Bug 1681847] UdevLog.txt
apport information ** Attachment added: "UdevLog.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860199/+files/UdevLog.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules-4.4.0-72-
[Kernel-packages] [Bug 1681847] Lspci.txt
apport information ** Attachment added: "Lspci.txt" https://bugs.launchpad.net/bugs/1681847/+attachment/4860193/+files/Lspci.txt -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Confirmed Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 --- AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Apr 11 13:18 seq crw-rw 1 root audio 116, 33 Apr 11 13:18 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.14.1-0ubuntu3.23 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: Error: [Errno 2] No such file or directory DistroRelease: Ubuntu 14.04 HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap IwConfig: Error: [Errno 2] No such file or directory Lsusb: Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Dell Inc. PowerEdge R320 Package: linux (not installed) PciMultimedia: ProcFB: 0 VESA VGA ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 RelatedPackageVersions: linux-restricted-modules-4.4.0-72-generic N/A linux-backports-modules-4.4.0-72-gener
[Kernel-packages] [Bug 1681847] Re: IPVS incorrectly reverse-NATs traffic to LVS host
apport information ** Tags added: apport-collected trusty ** Description changed: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 + --- + AlsaDevices: + total 0 + crw-rw 1 root audio 116, 1 Apr 11 13:18 seq + crw-rw 1 root audio 116, 33 Apr 11 13:18 timer + AplayDevices: Error: [Errno 2] No such file or directory + ApportVersion: 2.14.1-0ubuntu3.23 + Architecture: amd64 + ArecordDevices: Error: [Errno 2] No such file or directory + AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: + CRDA: Error: [Errno 2] No such file or directory + DistroRelease: Ubuntu 14.04 + HibernationDevice: RESUME=/dev/mapper/lvs5--vg-swap + IwConfig: Error: [Errno 2] No such file or directory + Lsusb: + Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub + Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub + Bus 001 Device 003: ID 413c:a001 Dell Computer Corp. Hub + Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub + Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub + MachineType: Dell Inc. PowerEdge R320 + Package: linux (not installed) + PciMultimedia: + + ProcFB: 0 VESA VGA + ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-72-generic root=/dev/mapper/hostname--vg-root ro quiet splash vt.handoff=7 + ProcVersionSignature: Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 + RelatedPackageVersions: + linux-restricted-modules-4.4.0-72-generic N/A + linux-backports-modules-4.4.0-72-generic N/A + linux-firmware1.127.23 + RfKill: Error: [Errno 2] No such file or directory + Tags: trusty + Uname: Linux 4.4.0-72-generic x86_64 + UpgradeStatus: No upgrade log present (probably fresh install) + UserGroups: btcats daemon dba named oinstall yimsora + _MarkForUpload: True + dmi.bios
[Kernel-packages] [Bug 1681847] Re: IPVS incorrectly reverse-NATs traffic to LVS host
** Description changed: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) - for example, (lvsip:5 -> dnsip:53) + - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The - application layer then sees the response from (lvsip:53), which is + application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): - Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 - Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 + Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 + Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Incomplete Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keye
[Kernel-packages] [Bug 1681847] [NEW] IPVS incorrectly reverse-NATs traffic to LVS host
Public bug reported: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from dnsip:53 and there is a connection table entry, IPVS seems to assume NAT is in use, and translates it using the entry (lvsip:5 -> dnsip:53). The application layer then sees the response from (dnsvip:53), which is incorrect. /proc/version_signature (from both nodes): Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39 Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49 IPVS Configuration: root@lvs5:~# ipvsadm -Sn -A -t 144.32.128.183:25 -s wlc -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10 -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10 -A -t 144.32.128.183:465 -s wlc -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10 -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10 -A -t 144.32.128.183:587 -s wlc -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10 -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10 -A -t 144.32.128.240:53 -s rr -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -t 144.32.128.242:53 -s rr -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 -A -t 144.32.129.39:25 -s wlc -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10 -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10 -A -t 144.32.129.39:465 -s wlc -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10 -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10 -A -t 144.32.129.39:587 -s wlc -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10 -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10 -A -u 144.32.128.240:53 -s rr -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10 -A -u 144.32.128.242:53 -s rr -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10 ** Affects: linux (Ubuntu) Importance: Undecided Status: Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1681847 Title: IPVS incorrectly reverse-NATs traffic to LVS host Status in linux package in Ubuntu: Incomplete Bug description: We have observed the following behaviour on our LVS systems, which is causing issues with our monitor scripts. The systems are running Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and -116) and the 4.4.0-72 xenial kernel. Our systems are set up in direct routing mode for the services they handle. Our monitor scripts for DNS send test queries to our DNS servers using their real IPs. Sporadically, we have seen these checks fail as the DNS answers are coming back from the wrong IP address. We debugged this using tcpdump, and found that the response packets coming into the LVS systems were using the correct IPs (i.e. the real IPs on the DNS servers). However, applications see the responses as coming from a VIP instead. All of this has been established using UDP traffic. I have tracked this behaviour down to a specific case, which I can only assume is associated with how the kernel handles LVS NAT connections (i.e. masquerade mode): - If a DNS query is made on the LVS server to a DNS VIP, that creates an entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and associated with (vip:vport) - for example, (lvsip:5 -> dnsip:53) associated with (dnsvip:53) - If a subsequent DNS query is made from the same UDP port, the response is correct as seen by tcpdump - When the response is seen by the application, the source IP address for the response is wrong I have inferred that as the response comes back from d
