On Fri, Nov 19, 2021 at 1:42 AM Vladislav Yaroshchuk <yaroshchuk2...@gmail.com> wrote: > > > > ср, 17 нояб. 2021 г. в 09:10, Jason Wang <jasow...@redhat.com>: >> >> On Tue, Nov 16, 2021 at 11:30 PM Vladislav Yaroshchuk >> <yaroshchuk2...@gmail.com> wrote: >> > >> > >> > >> > вт, 16 нояб. 2021 г. в 10:23, Jason Wang <jasow...@redhat.com>: >> >> >> >> On Mon, Nov 15, 2021 at 6:45 PM Vladislav Yaroshchuk >> >> <yaroshchuk2...@gmail.com> wrote: >> >> > >> >> > >> >> > >> >> > пн, 15 нояб. 2021 г. в 07:53, Jason Wang <jasow...@redhat.com>: >> >> >> >> >> >> On Fri, Nov 12, 2021 at 5:14 PM Vladislav Yaroshchuk >> >> >> <yaroshchuk2...@gmail.com> wrote: >> >> >> > >> >> >> > macOS provides networking API for VMs called 'vmnet.framework': >> >> >> > https://developer.apple.com/documentation/vmnet >> >> >> > >> >> >> > We can provide its support as the new QEMU network backends which >> >> >> > represent three different vmnet.framework interface usage modes: >> >> >> > >> >> >> > * `vmnet-shared`: >> >> >> > allows the guest to communicate with other guests in shared mode >> >> >> > and >> >> >> > also with external network (Internet) via NAT. Has >> >> >> > (macOS-provided) >> >> >> > DHCP server; subnet mask and IP range can be configured; >> >> >> > >> >> >> > * `vmnet-host`: >> >> >> > allows the guest to communicate with other guests in host mode. >> >> >> > By default has enabled DHCP as `vmnet-shared`, but providing >> >> >> > network unique id (uuid) can make `vmnet-host` interfaces >> >> >> > isolated >> >> >> > from each other and also disables DHCP. >> >> >> > >> >> >> > * `vmnet-bridged`: >> >> >> > bridges the guest with a physical network interface. >> >> >> > >> >> >> > This backends cannot work on macOS Catalina 10.15 cause we use >> >> >> > vmnet.framework API provided only with macOS 11 and newer. Seems >> >> >> > that it is not a problem, because QEMU guarantees to work on two most >> >> >> > recent versions of macOS which now are Big Sur (11) and Monterey >> >> >> > (12). >> >> >> > >> >> >> > Also, we have one inconvenient restriction: vmnet.framework >> >> >> > interfaces >> >> >> > can create only privileged user: >> >> >> > `$ sudo qemu-system-x86_64 -nic vmnet-shared` >> >> >> > >> >> >> > Attempt of `vmnet-*` netdev creation being unprivileged user fails >> >> >> > with >> >> >> > vmnet's 'general failure'. >> >> >> > >> >> >> > This happens because vmnet.framework requires >> >> >> > `com.apple.vm.networking` >> >> >> > entitlement which is: "restricted to developers of virtualization >> >> >> > software. >> >> >> > To request this entitlement, contact your Apple representative." as >> >> >> > Apple >> >> >> > documentation says: >> >> >> > https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_vm_networking >> >> >> >> >> >> Do you know how multipass work? Looks like it uses vmnet without >> >> >> privileges. >> >> >> >> >> > >> >> > I've just checked this, and they still need root privileges. They have a >> >> > `multipassd` daemon which is launched as root by launchd by default. >> >> > >> >> > ``` >> >> > bash-5.1$ ps axo ruser,ruid,comm | grep multipass >> >> > root 0 /Library/Application >> >> > Support/com.canonical.multipass/bin/multipassd >> >> > root 0 /Library/Application >> >> > Support/com.canonical.multipass/bin/hyperkit >> >> > ``` >> >> > >> >> > That's the reason why it's required to 'enter a password' while >> >> > multipass installation: >> >> > it creates launchd plist (kinda launch rule) and places it to >> >> > /Library/LaunchDaemons/, >> >> > which can be done only with root privileges. >> >> > >> >> > ``` >> >> > bash-5.1$ ll /Library/LaunchDaemons | grep multipass >> >> > -rw-r--r-- 1 root wheel 1.1K 15 Nov 12:47 >> >> > com.canonical.multipassd.plist >> >> > ``` >> >> > >> >> > And after this launchd launches this multipassd daemon as root every >> >> > boot. >> >> > >> >> > So an unprivileged user can launch a multipass VM instance, but >> >> > actually the >> >> > `hyperkit` process which interacts with vmnet is gonna be launched by >> >> > multipassd >> >> > running as root. >> >> >> >> I wonder how it passes the vmnet object to qemu? Nothing obvious from >> >> the qemu command line that multipass launched: >> >> >> >> -nic vmnet-macos,mode=shared,model=virtio-net-pci,mac=52:54:00:52:e8:e4 >> >> >> >> (But I haven't had time to check their qemu codes). >> >> >> > >> > It seems they just use QEMU with patch by Phillip Tennen: >> > https://patchew.org/QEMU/20210218134947.1860-1-phillip.en...@gmail.com/ >> > >> > In that patch he does quite the same as we in this series, the >> > difference remains in foreground: he creates one new 'vmnet-macos' >> > netdev, and uses 'mode=shared' property to choose vmnet >> > operating mode. I decided to create three different netdevs instead >> > (vmnet-shared, vmnet-host, vmnet-bridged). Also I've added some >> > features related to isolation and ipv6. >> >> Ok. >> >> > >> > > I wonder how it passes the vmnet object to qemu >> > I hope I clearly described this. >> >> A silly question, how did the 'hyperkit' process pass the vmnet object to >> qemu? >> > I think I didn't describe it very well in the previous mail. > The 'hyperkit' and QEMU are two multipass 'drivers'. > > hyperkit is an independent hypervisor based on xhyve (bhyve): > https://github.com/moby/hyperkit > > So there are many ways to launch multipass VMs: > * VM <-> hyperkit <-> host > * VM <-> QEMU <-> host > * VM <-> virtualbox <-> host > * etc. > > https://discourse.ubuntu.com/t/announcing-the-first-release-candidate-for-apple-m1-support/24445 > > So hyperkit doesn't pass anything to QEMU, it's a separate > hypervisor and multipass driver. But it uses vmnet backend > too, the same way as we do: > https://github.com/moby/hyperkit/blob/2f061e447e1435cdf1b9eda364cea6414f2c606b/src/lib/pci_virtio_net_vmnet.c#L250
Ok, I think I get it. > >> > >> >> > >> >> > tl;dr sadly, we can't interact with vmnet.framework without having our >> >> > binary correctly >> >> > signed and being an unprivileged user. Root privileges or special >> >> > signature with >> >> > entitlement is required. >> >> >> >> This is something similar to what happens in other OS. E.g for the tap >> >> backend, it can't be created without privileges. So qemu allows to: >> >> >> >> 1) the TAP to be created by privileged program like libvirt, and its >> >> fd could be passed to qemu via SCM_RIGHTS >> >> 2) run a set-uid helper to create and config TAP >> >> >> >> This is something we need to consider now or in the future probably. >> >> >> > >> > Seems we can do nothing with this if we have qemu-bundled & >> > direct vmnet.framework interaction, it always requires privileges >> > or entitlement. >> > The workaround can be moving vmnet-related things to >> > another helper binary running with privileges, and usage of >> > this helper somewhere between qemu and vmnet. >> >> Yes, that's my point. >> >> > >> > I think for now it's applicable to leave it as is, having qemu >> > that requires privileges for vmnet.framework usage. >> >> Right, but we need to consider it for the future. >> >> Btw, if you wish, you can list yourself as the maintainer for this backend. >> > > Ok, thank you! Let me do this in the next patch version, after > we finish our discussion and I fix all the issues you pointed > in your review. Sure. Thanks > >> >> Thanks >> >> > >> >> >> >> Thanks >> >> >> >> > >> >> > >> >> >> Thanks >> >> >> >> >> > >> >> > Thank you for your review, I will check it this week and reply as soon >> >> > as possible. >> >> > >> >> >> >> >> >> > >> >> >> > One more note: we still have quite useful but not supported >> >> >> > 'vmnet.framework' features as creating port forwarding rules, IPv6 >> >> >> > NAT prefix specifying and so on. >> >> >> > >> >> >> > Nevertheless, new backends work fine and tested within >> >> >> > `qemu-system-x86-64` >> >> >> > on macOS Bir Sur 11.5.2 host with such nic models: >> >> >> > * e1000-82545em >> >> >> > * virtio-net-pci >> >> >> > * vmxnet3 >> >> >> > >> >> >> > The guests were: >> >> >> > * macOS 10.15.7 >> >> >> > * Ubuntu Bionic (server cloudimg) >> >> >> > >> >> >> > >> >> >> > This series partially reuses patches by Phillip Tennen: >> >> >> > https://patchew.org/QEMU/20210218134947.1860-1-phillip.en...@gmail.com/ >> >> >> > So I included his signed-off line into one of the commit messages and >> >> >> > also here. >> >> >> > >> >> >> > v1 -> v2: >> >> >> > Since v1 minor typos were fixed, patches rebased onto latest master, >> >> >> > redundant changes removed (small commits squashed) >> >> >> > >> >> >> > v2 -> v3: >> >> >> > - QAPI style fixes >> >> >> > - Typos fixes in comments >> >> >> > - `#include`'s updated to be in sync with recent master >> >> >> > v3 -> v4: >> >> >> > - Support vmnet interfaces isolation feature >> >> >> > - Support vmnet-host network uuid setting feature >> >> >> > - Refactored sources a bit >> >> >> > v4 -> v5: >> >> >> > - Missed 6.2 boat, now 7.0 candidate >> >> >> > - Fix qapi netdev descriptions and styles >> >> >> > (@subnetmask -> @subnet-mask) >> >> >> > - Support vmnet-shared IPv6 prefix setting feature >> >> >> > >> >> >> > Signed-off-by: Phillip Tennen <phil...@axleos.com> >> >> >> > Signed-off-by: Vladislav Yaroshchuk <yaroshchuk2...@gmail.com> >> >> >> > >> >> >> > Vladislav Yaroshchuk (6): >> >> >> > net/vmnet: add vmnet dependency and customizable option >> >> >> > net/vmnet: add vmnet backends to qapi/net >> >> >> > net/vmnet: implement shared mode (vmnet-shared) >> >> >> > net/vmnet: implement host mode (vmnet-host) >> >> >> > net/vmnet: implement bridged mode (vmnet-bridged) >> >> >> > net/vmnet: update qemu-options.hx >> >> >> > >> >> >> > meson.build | 4 + >> >> >> > meson_options.txt | 2 + >> >> >> > net/clients.h | 11 ++ >> >> >> > net/meson.build | 7 + >> >> >> > net/net.c | 10 ++ >> >> >> > net/vmnet-bridged.m | 111 ++++++++++++ >> >> >> > net/vmnet-common.m | 325 >> >> >> > ++++++++++++++++++++++++++++++++++ >> >> >> > net/vmnet-host.c | 111 ++++++++++++ >> >> >> > net/vmnet-shared.c | 92 ++++++++++ >> >> >> > net/vmnet_int.h | 48 +++++ >> >> >> > qapi/net.json | 127 ++++++++++++- >> >> >> > qemu-options.hx | 25 +++ >> >> >> > scripts/meson-buildoptions.sh | 3 + >> >> >> > 13 files changed, 874 insertions(+), 2 deletions(-) >> >> >> > create mode 100644 net/vmnet-bridged.m >> >> >> > create mode 100644 net/vmnet-common.m >> >> >> > create mode 100644 net/vmnet-host.c >> >> >> > create mode 100644 net/vmnet-shared.c >> >> >> > create mode 100644 net/vmnet_int.h >> >> >> > >> >> >> > -- >> >> >> > 2.23.0 >> >> >> > >> >> >> >> >> > >> >> > >> >> > -- >> >> > Best Regards, >> >> > >> >> > Vladislav Yaroshchuk >> >> >> > >> > >> > -- >> > Best Regards, >> > >> > Vladislav Yaroshchuk >>