On 06/07/2021 03:51, li...@rc.inesa.com wrote: >> On 05/07/2021 09:08, li...@rc.inesa.com wrote: >>> Dear OVS Team, >>> >>> We are seeing an issue occurred in OVS version 2.11.0. Right now we are >>> trying to encrypt our VXLAN packets in communication. The problem is, when >>> the tunnel is already built between host 1 and host 2, if we run the >>> command to let host 1 ping host 2 and at the same time, run tcpdump command >>> to capture the processing packets, we observed that packets from host 1 to >>> host 2 are well encrypted while packets from host 2 to host 1 are in >>> plaintext (it is supposed to be encrypted too). >>> >>> Here is the whole processes of how we found this problem: >>> >>> Firstly, we install OVS 2.11.0 and ovs-ipsec on both two centos VMs in >>> virtual box. >>> External IP of host 1 is: 10.200.46.186, host 2 is: 10.200.46.187 >>> >>> Next, we run the following command in order to build up a VXLAN tunnel: >>> (used https://docs.openvswitch.org/en/latest/tutorials/ipsec/ as a >>> reference) >>> In host 1: >>> # ovs-vsctl add-br br-ipsec >>> # ip addr add 192.0.0.1/24 dev br-ipsec >>> # ip link set br-ipsec up >>> >>> In host 2: >>> # ovs-vsctl add-br br-ipsec >>> # ip addr add 192.0.0.2/24 dev br-ipsec >>> # ip link set br-ipsec up >>> >>> And then, we run this command in host 1: >>> ovs-vsctl add-port br-ipsec tun -- set interface tun type=vxlan >>> options:remote_ip=10.200.46.187 options:psk=swordfish >>> >>> And this command in host 2: >>> ovs-vsctl add-port br-ipsec tun -- set interface tun type=vxlan >>> options:remote_ip=10.200.46.186 options:psk=swordfish >>> >>> The network topology looks like this: >>> Host1: >>> [root@ovs-ipsec-001 ~]# ovs-vsctl show >>> 81d4eef4-ebd7-4cb4-b1ab-d2c4a4469496 >>> Bridge br-ipsec >>> Port tun >>> Interface tun >>> type: vxlan >>> options: {psk=swordfish, remote_ip="10.200.46.187"} >>> Port br-ipsec >>> Interface br-ipsec >>> type: internal >>> ovs_version: "2.11.0" >>> >>> >>> Host2: >>> [root@ovs-ipsec-002 ~]# ovs-vsctl show >>> f2bff28e-4adc-4c8c-941a-602d249e6d6e >>> Bridge br-ipsec >>> Port br-ipsec >>> Interface br-ipsec >>> type: internal >>> Port tun >>> Interface tun >>> type: vxlan >>> options: {psk=swordfish, remote_ip="10.200.46.186"} >>> ovs_version: "2.11.0" >>> >>> >>> The ping connection between host 1 and host 2 is good, however, the tcpdump >>> processes shows the "come back" packets of ping is in plaintext: >> >> It looks like the "request" is clear text here (not the reply) but .... >> >>> [root@ovs-ipsec-001 ~]# tcpdump -e -ni any net 10.200.46.187 >> >> .... it may be because you are specifying "-i any" above. You should run >> tcpdump on the physical interface ( -i <physical interface name ). I >> think you are seeing the same packet twice. > > We've done this experiment again and tried the command you adviced us as > this: "tcpdump -ni enp0s3 net 10.200.46.187". > > However, the result still shows that the "coming back" packets are in > plaintext:
I added ovs-dev to the thread again. My bad, I was reading the trace incorrectly. I guess the remote side is not sending out encrypted traffic. Is there anything suspect in the logs? > > Here is the result of tcpdump: > [root@ovs-ipsec-001 ~]# tcpdump -ni enp0s3 net 10.200.46.187 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on enp0s3, link-type EN10MB (Ethernet), capture size 262144 bytes > 10:35:51.070218 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0x6), length 148 > 10:35:51.071096 IP 10.200.46.187.52430 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > IP 192.0.0.2 > 192.0.0.1: ICMP echo reply, id 2760, seq 5, length 64 > 10:35:51.645724 IP 10.200.46.186.isakmp > 10.200.46.187.isakmp: isakmp: > parent_sa child_sa > 10:35:51.647024 IP 10.200.46.187.isakmp > 10.200.46.186.isakmp: isakmp: > parent_sa child_sa[IR] > 10:35:52.072198 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0x7), length 148 > 10:35:52.072970 IP 10.200.46.187.52430 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > IP 192.0.0.2 > 192.0.0.1: ICMP echo reply, id 2760, seq 6, length 64 > 10:35:52.079039 ARP, Request who-has 10.200.46.187 tell 10.200.46.186, length > 28 > 10:35:52.079520 ARP, Reply 10.200.46.187 is-at 08:00:27:b1:a5:c0, length 46 > 10:35:52.079771 IP 10.200.46.187.35886 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > ARP, Request who-has 192.0.0.1 tell 192.0.0.2, length 28 > 10:35:52.080149 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0x8), length 92 > 10:35:52.146977 IP 10.200.46.186.isakmp > 10.200.46.187.isakmp: isakmp: > parent_sa child_sa > 10:35:52.147701 IP 10.200.46.187.isakmp > 10.200.46.186.isakmp: isakmp: > child_sa child_sa[I] > 10:35:52.648676 IP 10.200.46.186.isakmp > 10.200.46.187.isakmp: isakmp: > parent_sa child_sa > 10:35:52.649707 IP 10.200.46.187.isakmp > 10.200.46.186.isakmp: isakmp: > child_sa child_sa[I] > 10:35:53.073706 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0x9), length 148 > 10:35:53.074533 IP 10.200.46.187.52430 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > IP 192.0.0.2 > 192.0.0.1: ICMP echo reply, id 2760, seq 7, length 64 > 10:35:53.506174 IP 10.200.46.187.isakmp > 10.200.46.186.isakmp: isakmp: > child_sa child_sa[I] > 10:35:53.650223 IP 10.200.46.186.isakmp > 10.200.46.187.isakmp: isakmp: > parent_sa child_sa > 10:35:53.651311 IP 10.200.46.187.isakmp > 10.200.46.186.isakmp: isakmp: > child_sa child_sa[I] > 10:35:54.075673 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0xa), length 148 > 10:35:54.076278 IP 10.200.46.187.52430 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > IP 192.0.0.2 > 192.0.0.1: ICMP echo reply, id 2760, seq 8, length 64 > 10:35:55.075808 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0xb), length 148 > 10:35:55.076479 IP 10.200.46.187.52430 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > IP 192.0.0.2 > 192.0.0.1: ICMP echo reply, id 2760, seq 9, length 64 > > [root@ovs-ipsec-001 ~]# tcpdump -ni enp0s3 net 10.200.46.187 > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on enp0s3, link-type EN10MB (Ethernet), capture size 262144 bytes > 10:35:51.070218 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0x6), length 148 > 10:35:51.071096 IP 10.200.46.187.52430 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > IP 192.0.0.2 > 192.0.0.1: ICMP echo reply, id 2760, seq 5, length 64 > 10:35:51.645724 IP 10.200.46.186.isakmp > 10.200.46.187.isakmp: isakmp: > parent_sa child_sa > 10:35:51.647024 IP 10.200.46.187.isakmp > 10.200.46.186.isakmp: isakmp: > parent_sa child_sa[IR] > 10:35:52.072198 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0x7), length 148 > 10:35:52.072970 IP 10.200.46.187.52430 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > IP 192.0.0.2 > 192.0.0.1: ICMP echo reply, id 2760, seq 6, length 64 > 10:35:52.079039 ARP, Request who-has 10.200.46.187 tell 10.200.46.186, length > 28 > 10:35:52.079520 ARP, Reply 10.200.46.187 is-at 08:00:27:b1:a5:c0, length 46 > 10:35:52.079771 IP 10.200.46.187.35886 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > ARP, Request who-has 192.0.0.1 tell 192.0.0.2, length 28 > 10:35:52.080149 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0x8), length 92 > 10:35:52.146977 IP 10.200.46.186.isakmp > 10.200.46.187.isakmp: isakmp: > parent_sa child_sa > 10:35:52.147701 IP 10.200.46.187.isakmp > 10.200.46.186.isakmp: isakmp: > child_sa child_sa[I] > 10:35:52.648676 IP 10.200.46.186.isakmp > 10.200.46.187.isakmp: isakmp: > parent_sa child_sa > 10:35:52.649707 IP 10.200.46.187.isakmp > 10.200.46.186.isakmp: isakmp: > child_sa child_sa[I] > 10:35:53.073706 IP 10.200.46.186 > 10.200.46.187: > ESP(spi=0x2ff055b4,seq=0x9), length 148 > 10:35:53.074533 IP 10.200.46.187.52430 > 10.200.46.186.4789: VXLAN, flags [I] > (0x08), vni 0 > IP 192.0.0.2 > 192.0.0.1: ICMP echo reply, id 2760, seq 7, length 64 >> >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode >>> listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 >>> bytes >>> 16:02:53.267379 In 08:00:27:b1:a5:c0 ethertype IPv4 (0x0800), length 150: >>> 10.200.46.187.44471 > 10.200.46.186.4789: VXLAN, flags [I] (0x08), vni 0 >>> 02:39:d5:12:45:4a > 9a:2f:5d:68:cc:4b, ethertype IPv4 (0x0800), length 98: >>> 192.0.0.2 > 192.0.0.1: ICMP echo request, id 2717, seq 1388, length 64 >>> 16:02:53.267549 Out 08:00:27:ca:9b:1a ethertype IPv4 (0x0800), length 184: >>> 10.200.46.186 > 10.200.46.187: ESP(spi=0x6914b456,seq=0x6d7), length 148 >>> 16:02:54.268637 In 08:00:27:b1:a5:c0 ethertype IPv4 (0x0800), length 150: >>> 10.200.46.187.44471 > 10.200.46.186.4789: VXLAN, flags [I] (0x08), vni 0 >>> 02:39:d5:12:45:4a > 9a:2f:5d:68:cc:4b, ethertype IPv4 (0x0800), length 98: >>> 192.0.0.2 > 192.0.0.1: ICMP echo request, id 2717, seq 1389, length 64 >>> 16:02:54.268844 Out 08:00:27:ca:9b:1a ethertype IPv4 (0x0800), length 184: >>> 10.200.46.186 > 10.200.46.187: ESP(spi=0x6914b456,seq=0x6d8), length 148 >>> 16:02:55.270073 In 08:00:27:b1:a5:c0 ethertype IPv4 (0x0800), length 150: >>> 10.200.46.187.44471 > 10.200.46.186.4789: VXLAN, flags [I] (0x08), vni 0 >>> 02:39:d5:12:45:4a > 9a:2f:5d:68:cc:4b, ethertype IPv4 (0x0800), length 98: >>> 192.0.0.2 > 192.0.0.1: ICMP echo request, id 2717, seq 1390, length 64 >>> 16:02:55.270265 Out 08:00:27:ca:9b:1a ethertype IPv4 (0x0800), length 184: >>> 10.200.46.186 > 10.200.46.187: ESP(spi=0x6914b456,seq=0x6d9), length 148 >>> 16:02:56.272043 In 08:00:27:b1:a5:c0 ethertype IPv4 (0x0800), length 150: >>> 10.200.46.187.44471 > 10.200.46.186.4789: VXLAN, flags [I] (0x08), vni 0 >>> 02:39:d5:12:45:4a > 9a:2f:5d:68:cc:4b, ethertype IPv4 (0x0800), length 98: >>> 192.0.0.2 > 192.0.0.1: ICMP echo request, id 2717, seq 1391, length 64 >>> 16:02:56.272231 Out 08:00:27:ca:9b:1a ethertype IPv4 (0x0800), length 184: >>> 10.200.46.186 > 10.200.46.187: ESP(spi=0x6914b456,seq=0x6da), length 148 >>> 16:02:57.274848 In 08:00:27:b1:a5:c0 ethertype IPv4 (0x0800), length 150: >>> 10.200.46.187.44471 > 10.200.46.186.4789: VXLAN, flags [I] (0x08), vni 0 >>> 02:39:d5:12:45:4a > 9a:2f:5d:68:cc:4b, ethertype IPv4 (0x0800), length 98: >>> 192.0.0.2 > 192.0.0.1: ICMP echo request, id 2717, seq 1392, length 64 >>> 16:02:57.275044 Out 08:00:27:ca:9b:1a ethertype IPv4 (0x0800), length 184: >>> 10.200.46.186 > 10.200.46.187: ESP(spi=0x6914b456,seq=0x6db), length 148 >>> >>> As is shown, when the connection come back from host 2 to host 1, it is not >>> encrypted, and it is obvious that the original VXLAN and ICMP packets are >>> revealed. >>> >>> Can you please help us figure out why this happen? Or any other alternative >>> solution of encrypting VXLAN packets? >>> >>> Thank you very much! >>> >>> Bests, >>> Yiming Shi >>> >>> >>> _______________________________________________ >>> discuss mailing list >>> disc...@openvswitch.org >>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >>> >> > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev
