** Changed in: nova Status: New => In Progress ** Changed in: nova Assignee: (unassigned) => Josephine Seifert (josei)
** Changed in: nova Importance: Undecided => High ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/victoria Importance: Undecided Status: New ** Also affects: nova/train Importance: Undecided Status: New ** Also affects: nova/ussuri Importance: Undecided Status: New ** Changed in: nova/stein Importance: Undecided => High ** Changed in: nova/train Importance: Undecided => High ** Changed in: nova/ussuri Importance: Undecided => High ** Changed in: nova/victoria Importance: Undecided => High ** Tags added: tls ** Tags added: live-migration -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1919357 Title: "Secure live migration with QEMU-native TLS in nova"-guide misses essential config option Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) stein series: New Status in OpenStack Compute (nova) train series: New Status in OpenStack Compute (nova) ussuri series: New Status in OpenStack Compute (nova) victoria series: New Status in OpenStack Security Advisory: Won't Fix Status in OpenStack Security Notes: New Bug description: - [x] This doc is inaccurate in this way: ______ I followed the guide to setup qemu native tls for live migration. After checking, that libvirt is able to use tls using tcpdump to listen on the port for tls, I also wanted to check that it works when I live migrate an instance. Apparently it didn't. But it used the port for unencrypted TCP [1]. After digging through documentation and code afterwards I found that in this code part: https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L1120 @staticmethod def _live_migration_uri(dest): uris = { 'kvm': 'qemu+%(scheme)s://%(dest)s/system', 'qemu': 'qemu+%(scheme)s://%(dest)s/system', 'xen': 'xenmigr://%(dest)s/system', 'parallels': 'parallels+tcp://%(dest)s/system', } dest = oslo_netutils.escape_ipv6(dest) virt_type = CONF.libvirt.virt_type # TODO(pkoniszewski): Remove fetching live_migration_uri in Pike uri = CONF.libvirt.live_migration_uri if uri: return uri % dest uri = uris.get(virt_type) if uri is None: raise exception.LiveMigrationURINotAvailable(virt_type=virt_type) str_format = { 'dest': dest, 'scheme': CONF.libvirt.live_migration_scheme or 'tcp', } return uri % str_format the uri is calculated using the config parameter 'live_migration_scheme' or using the hard coded tcp parameter. Coming from the guide for qemu native tls, there was no hint that this config option needs to be set. In fact without setting this 'live_migration_scheme' config option to tls, there is no way to see, that the live migration still uses the unencrypted tcp connection - one has to use tcpdump and listen for tcp or tls to recognize it. Neither in the logs nor in any debug output there is any hint that it is still unencrypted! Thus I conclude there might be OpenStack deployments which are configured as the guide say but these config changes have no effect! - [x] This is a doc addition request. To fix this the config parameter 'live_migration_scheme' should be set to tls and maybe there should be a warning in the documentation, that without doing this, the traffic is still unencrypted. - [ ] I have a fix to the document that I can paste below including example: input and output. [1] without setting 'live_migration_scheme' in the nova.conf $ tcpdump -i INTERFACE -n -X port 16509 and '(tcp[((tcp[12] & 0xf0) >> 2)] < 0x14 || tcp[((tcp[12] & 0xf0) >> 2)] > 0x17)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 17:10:56.387407 IP 192.168.70.101.50900 > 192.168.70.100.16509: Flags [P.], seq 304:6488, ack 285, win 502, options [nop,nop,TS val 424149655 ecr 1875309961], length 6184 0x0000: 4500 185c ad05 4000 4006 677c c0a8 4665 E..\..@.@.g|..Fe 0x0010: c0a8 4664 c6d4 407d a407 70a6 15ad 0a5a ..Fd..@}..p....Z 0x0020: 8018 01f6 2669 0000 0101 080a 1948 0297 ....&i.......H.. 0x0030: 6fc6 f589 0000 1828 2000 8086 0000 0001 o......(........ 0x0040: 0000 012f 0000 0000 0000 0009 0000 0000 .../............ 0x0050: 0000 0001 0000 000f 6465 7374 696e 6174 ........destinat 0x0060: 696f 6e5f 786d 6c00 0000 0007 0000 129b ion_xml......... 0x0070: 3c64 6f6d 6169 6e20 7479 7065 3d27 6b76 <domain.type='kv 0x0080: 6d27 3e0a 2020 3c6e 616d 653e 696e 7374 m'>...<name>inst 0x0090: 616e 6365 2d30 3030 3032 6539 393c 2f6e ance-00002e99</n 0x00a0: 616d 653e 0a20 203c 7575 6964 3e37 6532 ame>...<uuid>7e2 0x00b0: 6364 3839 352d 6263 3765 2d34 6634 352d cd895-bc7e-4f45- 0x00c0: 6166 6264 2d33 3732 3166 3735 6134 3064 afbd-3721f75a40d 0x00d0: 383c 2f75 7569 643e 0a20 203c 6d65 7461 8</uuid>...<meta [2] after setting 'live_migration_scheme = tls' $ tcpdump -i INTERFACE -n -X port 16514 and '(tcp[((tcp[12] & 0xf0) >> 2)] > 0x13 && tcp[((tcp[12] & 0xf0) >> 2)] < 0x18)' tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on INTERFACE, link-type EN10MB (Ethernet), capture size 262144 bytes 16:55:47.746851 IP 192.168.70.100.35620 > 192.168.70.101.16514: Flags [P.], seq 1849334708:1849334914, ack 3121294199, win 502, options [nop,nop,TS val 1874401351 ecr 423241020], length 206 0x0000: 4500 0102 a605 4000 4006 85d6 c0a8 4664 E.....@.@.....Fd 0x0010: c0a8 4665 8b24 4082 6e3a 9bb4 ba0b 2b77 ..Fe.$@.n:....+w 0x0020: 8018 01f6 2f43 0000 0101 080a 6fb9 1847 ..../C......o..G 0x0030: 193a 253c 1603 0100 c901 0000 c503 0360 .:%<...........` 0x0040: 50e3 23f7 2ab9 57b2 47f5 e0e8 b550 014b P.#.*.W.G....P.K 0x0050: 35c6 fe95 3475 cc66 9f81 fc0d 1646 ab00 5...4u.f.....F.. 0x0060: 0052 c02c c087 cca9 c0ad c00a c02b c086 .R.,.........+.. 0x0070: c0ac c009 c008 c030 c08b cca8 c014 c02f .......0......./ All in all: I can switch back and forth using TLS or unencrypted TCP via the different ports with just changing the config parameter 'live_migration_scheme'. All tests were done on an OpenStack Train deployment. ----------------------------------- Release: 22.1.0.dev604 on 2019-03-11 17:38:39 SHA: 6e6d78d204dbf23d1ffaf232456fb2917b4f8372 Source: https://opendev.org/openstack/nova/src/doc/source/admin/secure-live-migration-with-qemu-native-tls.rst URL: https://docs.openstack.org/nova/latest/admin/secure-live-migration-with-qemu-native-tls.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1919357/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp