Removing duplicate to perform additional checks
** This bug is no longer a duplicate of bug 1478925 Instances on the same compute node unable to connect to each other's ports -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1514017 Title: Sec Group IPtables dropping all TCP ACK as Invalid for all connections between instances on different networks on same libvirt host. Status in neutron: New Bug description: Sec Group rules are set to allow all TCP connections in both directions. When 2 instance on the same host but in different networks try to start a TCP session, the TCP handshake never completes as the ACK packet gets dropped as invalid by the security group iptable rules on the Linux bridge. The SYN and SYN_ACK packets get through. This happens on all libvirt hosts, when the 2 instances are the same host. When the instance are on separate hosts, the ACK packet gets through and the session is established. VM1 --> SYN --> VM2 VM1 <-- SYN_ACK <-- VM2 VM1 --> ACK --> ??? In the case of the above running tcpdump on the host on the VM1 tap interface shows the ACK packet entering the bridge, a tcpdump on qvb interface on the far side of the bridge show the packet missing. The following is a sample tcpdump from the client tap interface showing SYN, SYN_ACK and ACK packets. 12:08:34.685284 fa:16:3e:6d:ff:9f (oui Unknown) > 00:00:0c:07:ac:47 (oui Cisco), ethertype IPv4 (0x0800), length 74: 10.0.116.15.58374 > 10.0.115.11.ssh: Flags [S], seq 942942122, win 14600, options [mss 1460,sackOK,TS val 1211694066 ecr 0,nop,wscale 6], length 0 12:08:34.686254 a2:30:cf:00:00:1e (oui Unknown) > fa:16:3e:6d:ff:9f (oui Unknown), ethertype IPv4 (0x0800), length 74: 10.0.115.11.ssh > 10.0.116.15.58374: Flags [S.], seq 2764187879, ack 942942123, win 5792, options [mss 1380,sackOK,TS val 1212626150 ecr 1211694066,nop,wscale 7], length 0 12:08:34.686425 fa:16:3e:6d:ff:9f (oui Unknown) > 00:00:0c:07:ac:47 (oui Cisco), ethertype IPv4 (0x0800), length 66: 10.0.116.15.58374 > 10.0.115.11.ssh: Flags [.], ack 1, win 229, options [nop,nop,TS val 1211694067 ecr 1212626150], length 0 12:08:38.096191 a2:30:cf:00:00:1e (oui Unknown) > fa:16:3e:6d:ff:9f (oui Unknown), ethertype IPv4 (0x0800), length 74: 10.0.115.11.ssh > 10.0.116.15.58374: Flags [S.], seq 2764187879, ack 942942123, win 5792, options [mss 1380,sackOK,TS val 1212629559 ecr 1211694066,nop,wscale 7], length 0 12:08:38.096379 fa:16:3e:6d:ff:9f (oui Unknown) > 00:00:0c:07:ac:47 (oui Cisco), ethertype IPv4 (0x0800), length 66: 10.0.116.15.58374 > 10.0.115.11.ssh: Flags [.], ack 1, win 229, options [nop,nop,TS val 1211697477 ecr 1212626150], length 0 After examining iptable rule counters while attempting to open a TCP session, the following rule in the neutron-openvswi-od080d3d6-c chain is dropping the packets. DROP all -- anywhere anywhere state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ conntrack shows the TCP session as being in the correct SYN_RECV state. conntrack -L -s 10.0.116.15 -d 10.0.115.11 -p tcp tcp 6 57 SYN_RECV src=10.0.116.15 dst=10.0.115.11 sport=58619 dport=22 src=10.0.115.11 dst=10.0.116.15 sport=22 dport=58619 mark=0 use=1 Bypassing Sec Groups firewalls entirely by disabling IPtable filtering on Linux bridges resolves the issue. net.bridge.bridge-nf-call-iptables=0 Software Versions: Neutron: Kilo 2015.1.1 Nova: Kilo 2015.1.1 OS: CentOS 7.1 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1514017/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp