Rust Xen Guest Agent
A new pre-release of our guest agent prototype written in Rust is available, numbered 0.2.0 [1]. Identified issues and work to be done are tracked in Gitlab issue tracker [3]. More details can be found in this blog post [2]. As always, feedback will be greatly appreciated! [1] https://gitlab.com/xen-project/xen-guest-agent/-/releases/0.2.0 [2] https://xcp-ng.org/blog/2023/10/12/updates-on-the-rust-guest-tools/ [3] https://gitlab.com/xen-project/xen-guest-agent/-/issues Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
[RFC] Rust Xen guest-agent
XCP-ng and our friends at XenServer have a need to collect a number of informations inside guests, to give to toolstacks and admins enough info to manage the guests properly. Today Xen downstreams have to implement a guest agent for this when needed (see below), and we think it would be beneficial to have one in the Xen Project, that could be configured for the needs of various downstreams. Playing with this idea we started on several aspects: - document what information needs to be collected and for what need, and propose a XenStore structure (very preliminary ideas in [0]) - a prototype agent [1] that: - could be used both as a replacement for what we use now, and for any improved data set we would want, as well as any specific needs other downstreams may have - would react on event where possible, to make changes visible ASAP to the toolstack - would be multi-platform enough to run in (ideally) all guests - we decided to write in Rust for various reasons [2] We'd like to hear from the community what you think about those ideas. What currently exists that we are aware of are: - description for a few xenstore paths to store VIF information (iface name, MAC, IP addresses) [3] - Xenserver's xe-guest-utilities, also used by XCP-ng; details of collected info in [4] - until versions 6.x it was implemented as a Linux-specific shell-script daemon [5] - this version has been forked for FreeBSD support [6] - versions 7.x [7] are a rewrite in Go, which is slightly more efficient by using a single connection to XenStore throughout its lifetime, but still uses a "spawn processes every minute" approach to data collection; despite its design that should allow support for other OS than Linux, it has not been extended for any (one user started a FreeBSD port, starting with conversion of the legacy shell script still doing OS detection [8], but the effort stopped short of any modification of Go code). - QubesOS' meminfo-writer [9]; only collects a "used memory" metric - apparently nothing for OpenXT ([10] only mentions installing PV drivers for Windows) [0] https://gitlab.com/ydirson/xen-guest-agent/-/blob/ydi/rust-proto/doc/structure.md [https://gitlab.com/ydirson/xen-guest-agent/-/blob/ydi/rust-proto/doc/structure.md] [1] https://gitlab.com/ydirson/xen-guest-agent/-/tree/ydi/rust-proto [https://gitlab.com/ydirson/xen-guest-agent/-/tree/ydi/rust-proto] https://gitlab.com/xen-project/xen-guest-agent/-/merge_requests/2 [2] https://xcp-ng.org/blog/2023/03/17/bringing-rust-to-the-xen-project/ [https://xcp-ng.org/blog/2023/03/17/bringing-rust-to-the-xen-project/] [3] https://xenbits.xen.org/docs/unstable/misc/xenstore-paths.html#domain-controlled-paths [https://xenbits.xen.org/docs/unstable/misc/xenstore-paths.html#domain-controlled-paths] [4] https://github.com/xenserver/xe-guest-utilities#collected-information-by-lifetime [https://github.com/xenserver/xe-guest-utilities#collected-information-by-lifetime] [5] https://aur.archlinux.org/packages/xe-guest-utilities [https://aur.archlinux.org/packages/xe-guest-utilities] [6] http://distcache.freebsd.org/local-distfiles/feld/xe-guest-utilities-6.0.2.tar.gz [http://distcache.freebsd.org/local-distfiles/feld/xe-guest-utilities-6.0.2.tar.gz] [7] https://github.com/xenserver/xe-guest-utilities [https://github.com/xenserver/xe-guest-utilities] [8] https://github.com/xcp-ng/xe-guest-utilities/pull/1 [https://github.com/xcp-ng/xe-guest-utilities/pull/1] [9] https://gitlab.com/QubesOS/qubes-linux-utils/-/blob/master/qmemman/meminfo-writer.c [https://gitlab.com/QubesOS/qubes-linux-utils/-/blob/master/qmemman/meminfo-writer.c] [10] https://openxt.atlassian.net/wiki/spaces/OD/pages/44924936/How+to+Install+Guest+VMs [https://openxt.atlassian.net/wiki/spaces/OD/pages/44924936/How+to+Install+Guest+VMs] Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
Re: xenstored: EACCESS error accessing control/feature-balloon 1
Hi all, On 4/12/23 18:05, Andrew Cooper wrote: > On 12/04/2023 4:48 pm, zithro wrote: >> Hi all, >> >> this is what I have in "xenstored-access.log" in dom0 : >> >> [20230411T23:48:27.917Z] D5 write control/feature-balloon 1 >> [20230411T23:48:27.917Z] D5 error EACCES >> [20230411T23:48:27.923Z] D5 write data/updated Wed Apr 12 >> 01:48:27 CEST 2023 >> >> It happens once each minute, on two different hosts, both amd64. >> (both hosts using the same config, only the hardware differs). >> >> I tried looking up for a similar bug, but didn't find one. >> I apologize in advance if this error is known, and if this is not the >> place to report this ! >> >> --- >> Technical infos >> --- >> dom0s: >> >> Debian stable, kernel 5.10.0-21-amd64 >> Xen 4.14.5 >> xl.conf has : autoballoon="0" >> GRUB_CMDLINE_XEN="dom0_mem=2048M,max:2048M dom0_max_vcpus=4 >> dom0_vcpus_pin loglvl=all guest_loglvl=all ucode=scan iommu=verbose" >> Running "xenstore-ls -f -p | grep balloon" returns no result >> --- >> domUs (D5 in above logs): >> >> HVM TrueNAS Core, based on FreeBSD 13.1-RELEASE-p7 >> (it happened also on previous FreeBSD realeases, but don't remember when >> it started, logs have been filled and rotated). >> In cfg files, using either the same value for "memory" and "maxmem" or >> only setting "memory" give the same results. >> >> What's strange is that I have xen* commands in FreeNAS : >> >> xen-detect xenstore-control xenstore-ls xenstore-watch >> xenstore xenstore-exists xenstore-read xenstore-write >> xenstore-chmod xenstore-list xenstore-rm >> >> root@truenas[~]# xenstore-ls >> xenstore-ls: xs_directory (/): Permission denied >> >> root@truenas[~]# ps aux >> root [...] 0:36.98 [xenwatch] >> root [...] 0:01.01 [xenstore_rcv] >> root [...] 0:00.00 [balloon] >> root [...] 0:01.74 /bin/sh /usr/local/sbin/xe-daemon -p >> /var/run/xe-daemon.pid >> [...] >> >> The xe-daemon looks strange also, I don't use XenServer/XCP-ng, only >> "raw" Xen. >> And this script which hand >> >> I also use PFsense domUs (based on FreeBSD), but they don't exhibit >> this behaviour (ie. no xenstore access error in dom0, no xen* >> commands in domU). >> >> So is this a problem with TrueNAS rather than with Xen ? >> If so I apologize for wasting your time. >> >> Thanks, have a nice day ! >> (and as it's my first post here: thx for Xen, it rocks) > Hello, > > (Leaving the full report intact so CC'd people can see it whole) > > Yes, it is TrueNAS trying to re-write that file every minute. It > appears that TrueNAS has inherited (from debian?) a rather old version > of https://github.com/xenserver/xe-guest-utilities/ TrueNAS being FreeBSD-based probably inherits this from the sysutils/xe-guest-utilities port, which installs a fork of the shell version which predates this golang repository. > https://xenbits.xen.org/docs/unstable/misc/xenstore-paths.html doesn't > list feature-balloon as a permitted feature node. > > But, I suspect that it used to be the case that guests could write > arbitrary feature nodes, and I suspect we tightened the permissions in a > security fix to reduce worst-case memory usage of xenstored. > > I suspect the best (/least bad) thing to do here is formally introduce > feature-ballon as a permitted node, and have the toolstack initialise it > to "" like we do with all other nodes, after which TrueNAS ought to be > able to set it successfully and not touch it a second time. Is there anything besides XAPI using this node, or the other data published by xe-daemon? Maybe the original issue is just that there is no reason to have xe-guest-utilities installed in this setup? Best regards, -- Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.tech | xcp-ng.org | xen-orchestra.com Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
Re: xenstored: EACCESS error accessing control/feature-balloon 1
On 5/4/23 15:58, zithro wrote: > Hi, > > [ snipped for brevity, report summary: > XAPI daemon in domU tries to write to a non-existent xenstore node in > a non-XAPI dom0 ] > > On 12 Apr 2023 18:41, Yann Dirson wrote: >> Is there anything besides XAPI using this node, or the other data >> published by xe-daemon? > > On my vanilla Xen (ie. non-XAPI), I have no node about "balloon"-ing > in xenstore (either dom0 or domU nodes, but I'm not using ballooning > in both). > >> Maybe the original issue is just that there is no reason to have >> xe-guest-utilities installed in this setup? > > That's what I thought as I'm not using XAPI, so maybe the problem > should only be addressed to the truenas team ? I posted on their forum > but got no answer. > I killed the 'xe-daemon' in both setups without loss of functionality. > > My wild guess is that 'xe-daemon', 'xe-update-guest-attrs' and all > 'xenstore* commands' are leftovers from when Xen was working as a dom0 > under FreeBSD (why would a *domU* have them ?). That would not be correct: xenstore* are useful in guests, should you want to read/write to the XenStore manually or from scripts; xe-deamon and xe-update-guest-attrs both come from xe-guest-utilities 6.x, which is really a domU tool as well, but is there to support XAPI in dom0. Best regards, -- Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.tech | xcp-ng.org | xen-orchestra.com Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
Re: xenstored: EACCESS error accessing control/feature-balloon 1
On 5/4/23 20:04, zithro wrote: > On 04 May 2023 17:59, Yann Dirson wrote: >> >> On 5/4/23 15:58, zithro wrote: >>> Hi, >>> >>> [ snipped for brevity, report summary: >>> XAPI daemon in domU tries to write to a non-existent xenstore node in >>> a non-XAPI dom0 ] >>> >>> On 12 Apr 2023 18:41, Yann Dirson wrote: >>>> Is there anything besides XAPI using this node, or the other data >>>> published by xe-daemon? >>> >>> On my vanilla Xen (ie. non-XAPI), I have no node about "balloon"-ing >>> in xenstore (either dom0 or domU nodes, but I'm not using ballooning >>> in both). >>> >>>> Maybe the original issue is just that there is no reason to have >>>> xe-guest-utilities installed in this setup? >>> >>> That's what I thought as I'm not using XAPI, so maybe the problem >>> should only be addressed to the truenas team ? I posted on their forum >>> but got no answer. >>> I killed the 'xe-daemon' in both setups without loss of functionality. >>> >>> My wild guess is that 'xe-daemon', 'xe-update-guest-attrs' and all >>> 'xenstore* commands' are leftovers from when Xen was working as a dom0 >>> under FreeBSD (why would a *domU* have them ?). >> >> That would not be correct: xenstore* are useful in guests, should you >> want to read/write to the XenStore manually or from scripts; > > Didn't know that, can you give some use cases (or URLs) for which it > is useful, with or without XAPI ? > I've read xenstore* man pages and could not infer a use case. > Although I may already see some : updating ballooned memory values, or > as Julien Grall pointed out, updating "feature-s3/4" values ? You can get other examples in https://xenbits.xen.org/docs/unstable/misc/xenstore-paths.html#domain-controlled-paths > > PS: small mistake in "man/xenstore-write.1.html" (from at least 4.14, > and onward), the synopsis reads "xenstore-read" ieof "xenstore-read". Patch sent, thanks. > Also, the -s option disappeared from unstable, although it may be > expected. I don't know it's purpose either. See https://github.com/xen-project/xen/commit/c65687ed16d2289ec91036ec2862a4b4bd34ea4f Best regards, -- Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.tech | xcp-ng.org | xen-orchestra.com Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
[PATCH] docs/man: fix xenstore-write synopsis
Reported-by: zithro Signed-off-by: Yann Dirson --- docs/man/xenstore-write.1.pod | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/man/xenstore-write.1.pod b/docs/man/xenstore-write.1.pod index a0b1bca333..74f80f7b1b 100644 --- a/docs/man/xenstore-write.1.pod +++ b/docs/man/xenstore-write.1.pod @@ -4,7 +4,7 @@ xenstore-write - write Xenstore values =head1 SYNOPSIS -B [I]... I I... +B [I]... I I... =head1 DESCRIPTION -- 2.30.2 Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
[PATCH] docs: fix xenstore-paths doc structure
We currently have "Per Domain Paths" as an empty section, whereas it looks like "General Paths" was not indended to include all the following sections. Signed-off-by: Yann Dirson --- docs/misc/xenstore-paths.pandoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/misc/xenstore-paths.pandoc b/docs/misc/xenstore-paths.pandoc index bffb8ea544..f07ef90f63 100644 --- a/docs/misc/xenstore-paths.pandoc +++ b/docs/misc/xenstore-paths.pandoc @@ -129,7 +129,7 @@ create writable subdirectories as necessary. ## Per Domain Paths -## General Paths +### General Paths ~/vm = PATH [] -- 2.30.2 Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
[PATCH 1/3] docs: fix complex-and-wrong xenstore-path wording
"0 or 1 ... to indicate whether it is capable or incapable, respectively"
is luckily just swapped words. Making this shorter will
make the reading easier.
Signed-off-by: Yann Dirson
---
docs/misc/xenstore-paths.pandoc | 10 --
1 file changed, 4 insertions(+), 6 deletions(-)
diff --git a/docs/misc/xenstore-paths.pandoc b/docs/misc/xenstore-paths.pandoc
index f07ef90f63..a604f6b1c6 100644
--- a/docs/misc/xenstore-paths.pandoc
+++ b/docs/misc/xenstore-paths.pandoc
@@ -454,9 +454,8 @@ The precise protocol is not yet documented.
~/control/feature-suspend = (""|"0"|"1") [w]
These may be initialized to "" by the toolstack and may then be set
-to 0 or 1 by a guest to indicate whether it is capable or incapable,
-respectively, of responding to the corresponding command when written
-to ~/control/shutdown.
+to 0 or 1 by a guest to indicate whether it is capable of responding
+to the corresponding command when written to ~/control/shutdown.
A toolstack may then sample the feature- value at the point of issuing
a PV control command and respond accordingly:
@@ -507,9 +506,8 @@ string back to the control node.
~/control/feature-laptop-slate-mode = (""|"0"|"1") [w]
This may be initialized to "" by the toolstack and may then be set
-to 0 or 1 by a guest to indicate whether it is capable or incapable,
-respectively, of responding to a mode value written to
-~/control/laptop-slate-mode.
+to 0 or 1 by a guest to indicate whether it is capable of responding
+to a mode value written to ~/control/laptop-slate-mode.
### Domain Controlled Paths
--
2.30.2
Yann Dirson | Vates Platform Developer
XCP-ng & Xen Orchestra - Vates solutions
w: vates.fr | xcp-ng.org | xen-orchestra.com
[PATCH 0/3] officializing xenstore control/feature-balloon entry
The main topic of this patch series is the ~/control/feature-balloon entry used by XAPI, prompted by the report of xe-guest-utilities on FreeBSD not being able to report the feature when using just libxl on the host. First patch is a bit off-topic, but included here because it fixes the text from which this feature description was adapted from. Yann Dirson (3): docs: fix complex-and-wrong xenstore-path wording docs: document ~/control/feature-balloon libxl: create ~/control/feature-balloon docs/misc/xenstore-paths.pandoc | 16 ++-- tools/libs/light/libxl_create.c | 3 +++ 2 files changed, 13 insertions(+), 6 deletions(-) -- 2.30.2 Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
[PATCH 2/3] docs: document ~/control/feature-balloon
This flag has been in use by XAPI for a long time (was present at creation of the github xen-api/squeezed repo in 2013), and could be used by other toolstacks. Signed-off-by: Yann Dirson --- docs/misc/xenstore-paths.pandoc | 6 ++ 1 file changed, 6 insertions(+) diff --git a/docs/misc/xenstore-paths.pandoc b/docs/misc/xenstore-paths.pandoc index a604f6b1c6..6c4e2c3da2 100644 --- a/docs/misc/xenstore-paths.pandoc +++ b/docs/misc/xenstore-paths.pandoc @@ -509,6 +509,12 @@ This may be initialized to "" by the toolstack and may then be set to 0 or 1 by a guest to indicate whether it is capable of responding to a mode value written to ~/control/laptop-slate-mode. + ~/control/feature-balloon + +This may be initialized to "" by the toolstack and may then be set to +0 or 1 by a guest to indicate whether it is capable of memory +ballooning, and responds to values written to ~/memory/target. + ### Domain Controlled Paths ~/data/* [w] -- 2.30.2 Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
[PATCH 3/3] libxl: create ~/control/feature-balloon
Like other feature controls it has to be created by the toolstack before
the guest can advertise the feature.
Reported-by: zithro
Signed-off-by: Yann Dirson
---
tools/libs/light/libxl_create.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tools/libs/light/libxl_create.c b/tools/libs/light/libxl_create.c
index ec8eab02c2..85eccc90cd 100644
--- a/tools/libs/light/libxl_create.c
+++ b/tools/libs/light/libxl_create.c
@@ -863,6 +863,9 @@ retry_transaction:
libxl__xs_mknod(gc, t,
GCSPRINTF("%s/control/sysrq", dom_path),
rwperm, ARRAY_SIZE(rwperm));
+libxl__xs_mknod(gc, t,
+GCSPRINTF("%s/control/feature-balloon", dom_path),
+rwperm, ARRAY_SIZE(rwperm));
libxl__xs_mknod(gc, t,
GCSPRINTF("%s/data", dom_path),
--
2.30.2
Yann Dirson | Vates Platform Developer
XCP-ng & Xen Orchestra - Vates solutions
w: vates.fr | xcp-ng.org | xen-orchestra.com
Re: [PATCH 2/3] docs: document ~/control/feature-balloon
On 5/11/23 11:21, Jan Beulich wrote: > On 10.05.2023 16:20, Yann Dirson wrote: >> --- a/docs/misc/xenstore-paths.pandoc >> +++ b/docs/misc/xenstore-paths.pandoc >> @@ -509,6 +509,12 @@ This may be initialized to "" by the toolstack and may >> then be set >> to 0 or 1 by a guest to indicate whether it is capable of responding >> to a mode value written to ~/control/laptop-slate-mode. >> >> + ~/control/feature-balloon >> + >> +This may be initialized to "" by the toolstack and may then be set to >> +0 or 1 by a guest to indicate whether it is capable of memory >> +ballooning, and responds to values written to ~/memory/target. > > Besides correctly saying "may", I guess this wants to go further and also > clarify what the (intended) behavior is when the node is absent. Aiui PV > guests are always expected to have a balloon driver, so the assumed > value likely needs to be "1" there. Furthermore I'm afraid it doesn't > really become clear what value this node is if it's only optionally > present, while its absence doesn't really allow uniform assumptions > towards a default value. Things are indeed more complicated than I originally identified, the way this xenstore entry is used currently seems to make it difficult to introduce it in a backward-compatible manner I guess this and a number of details ought to be discussed at the XAPI level first. Details: the squeezed assumption [1] is that a domain which has not set this to 1 is not ready yet to get ballooned, which implies the default has to be 0 whatever the guest type, as it requires to know the total number of pages used by the domain to be stable. So I guess we can see it as not being "not just a feature flag". [1] https://github.com/xapi-project/xen-api/tree/master/ocaml/squeezed/doc/design#environmental-assumptions -- Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.tech | xcp-ng.org | xen-orchestra.com Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
[PATCH 0/1] RFC: clarify intended usage of ~/control/ xentore path
This proposal, spurred by a discrepancy between how toolstacks handles the control nodes, tries to summarize what I understand to be the spirit of ~/control/, from its children already described in the xenstore-paths document, and from the libxl behaviour. Yann Dirson (1): doc: clarify intended usage of ~/control/ xentore path docs/misc/xenstore-paths.pandoc | 29 + 1 file changed, 29 insertions(+) -- 2.30.2 Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.fr | xcp-ng.org | xen-orchestra.com
[PATCH 1/1] doc: clarify intended usage of ~/control/ xentore path
Signed-off-by: Yann Dirson
---
docs/misc/xenstore-paths.pandoc | 29 +
1 file changed, 29 insertions(+)
diff --git a/docs/misc/xenstore-paths.pandoc b/docs/misc/xenstore-paths.pandoc
index f07ef90f63..5501033893 100644
--- a/docs/misc/xenstore-paths.pandoc
+++ b/docs/misc/xenstore-paths.pandoc
@@ -432,6 +432,35 @@ by udev ("0") or will be run by the toolstack directly
("1").
### Platform Feature and Control Paths
+ ~/control = "" []
+
+Directory to hold feature and control paths. This directory is not
+guest-writable, only the toolstack is allowed to create new child
+nodes under this.
+
+Children of this nodes can have one of several types:
+
+* platform features: using name pattern `platform-feature-*`, they may
+ be set by the toolstack to inform the guest, and are not writable by
+ the guest.
+
+* guest features: using name pattern `feature-*`, they may be created
+ by the toolstack with an empty value (`""`), should be set writable
+ by the guest which can then advertize to the toolstack its
+ (non-)usage of the feature with values `"0"` and `"1"` respectively.
+ The lack of update by the guest can be interpreted by the toolstack
+ as the lack of supporting software (PV driver, guest agent, ...) in
+ the guest.
+
+* control nodes: using any name not matching the above pattern, they
+ are used by the toolstack or by the guest to signal a specific
+ condition to the other end, which is expected to watch it to react
+ to changes.
+
+Note: the presence of a control node in itself advertises the
+underlying toolstack feature, it is not necessary to add an extra
+platform-feature for such cases.
+
~/control/sysrq = (""|COMMAND) [w]
This is the PV SysRq control node. A toolstack can write a single character
--
2.30.2
Yann Dirson | Vates Platform Developer
XCP-ng & Xen Orchestra - Vates solutions
w: vates.fr | xcp-ng.org | xen-orchestra.com
Re: [PATCH 1/1] doc: clarify intended usage of ~/control/ xentore path
As discussed in Xen Summit, we likely don't want to merge it as is after all, but rather acknowledge that XAPI has taken the opposite route and allow the toolstack to give ownership of ~/control (or at least write permission?) to the guest -- maybe also recommending not to do that. Would that sound OK to everyone? On 6/27/23 09:04, Juergen Gross wrote: > On 24.06.23 16:07, Julien Grall wrote: >> Hi Yann, >> >> Adding Juergen. >> >> On 31/05/2023 11:35, Yann Dirson wrote: >>> Signed-off-by: Yann Dirson >> >> Reviewed-by: Julien Grall > > Reviewed-by: Juergen Gross > > > Juergen > -- Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.tech | xcp-ng.org | xen-orchestra.com Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Xen Summit 2023 Design Session: Communicating guest informations to the management stack
Here are my notes from memory, organized by topic (and somehow enriched with more details in places) # context Guest-management stacks typically need the guest to communicate some information back. This essentially happens through Xenstore and some of it has been normalized in the Xen docs. Some of those pieces of information are filled by PV drivers (eg. ~/control/feature-poweroff), others like information on network configuration are more commonly filled by a userspace daemon (eg. xe-guest-utilities for XAPI-based platforms). People in need to publish such info even turn to forked old versions of those tools for lack of a central solution. We have started to work on a prototype guest agent (written in Rust) with the goal of providing such an official guest service within the Xen Project, that would be easily portable and use low resources. # guest agent Rust prototype Question was asked why we selected Rust: aside from the language's well-known selling-points this was largely a political decision, intended to send a signal that the Xen project values such new technologies. There is a separate Xenserver guest agent for Windows guests. The Rust agent could likely get implementations for writing to the Windows idiom for Xenstore (through WMI), and to get notified of network config changes, which would still allow to share the rest (policy, xenstore layout, likely other things in the future). It has been pointed out the Xenserver Windows guest agent has support for other features not available in the Linux version, like Copy-paste (which currently is problematic, using Xenstore to stream potentially-large amounts of data). Such features should be made better, and available everywhere. Another lesser-known feature of the Xenserver windows guest agent allows (depending on a guest registry key for enabling the feature) to let dom0 execute commands in the guest. It was noted the concept is not that far for QubesOS qrexec mechanism. # NIC information The most visible info provided by guest tools is external IP addresses. This is useful for scripts in dom0 to discover it to communicate with guests through a given VNIC, and for users to get the information without having to manually log through the VM console. Today addresses are reported for VIF and SRIOV-passthrough interfaces, but not for those on PCI-passthrough, USB-passthrough[0] or bridges that include such interfaces. Reporting more info in a backward-compatible way would make it easier for the rest of the world. For additional passed-through devices we need a way for the guest to identify the devices to report about (MAC address? hw topology?), and we'll have to dedicate a new xenstore subtree to those. For bridged VNICs the bridge IP can accurately be reported for the VNIC itself, as if it were not bridged. But confusion can occur if 2 VNICs get bridged together, as they would both report the same IP. This could be made better by adding more information to the VNIC in addition to addresses, e.g. adding a guest-unique identifier for the bridge interface, so the toolstack can deduce the existence of the bridge. [0] although we likely don't care much about that one in the server space, desktop xen distros could have a need for this - edge computing and homelab possibly too, eg. for the RaspberryPi Ethernet # control of memory ballooning Current Xenserver guest agent sets a `feature-balloon` flag. It is not meant as a feature flag though, but signals to the XAPI balloon controller (squeezed) that the guest has booted far enough that all its memory has been allocated, and that it can now "calibrate its behaviour" to be able to decide when a balloon driver has reached the goal it was set. My evaluation was that the balloon driver would better report to xenstore when it has reached its target, rather than having the balloon controller doing guesswork. It was added that it would be useful for the balloon driver to report impossibility to reach a target, and possibly other conditions. Guest agent would still have to write this node for backward compatibility. It was mentioned that a "guest booted" report could be more generally useful. (we did not discuss any particular use, or whether that would be a job for guest tools or PV drivers) # standardization of xenstore interface Usage of xenstore by the guest agent and PV drivers was documented after the fact in the xenstore-paths document in Xen source tree. That could be cleaned up and made more useful. My proposed patch to standardize the permissions on `~/control/` as set by libxl was noted as problematic for XAPI: standardization should leave the current situation legal and allow both alternatives. Thanks to everyone who attended and contributed. -- Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.tech | xcp-ng.org | xen-orchestra.com
Re: Detecting whether dom0 is in a VM
On 7/6/23 17:35, zithro wrote: > On 06 Jul 2023 09:02, Jan Beulich wrote: >> On 05.07.2023 18:20, zithro wrote: >>> So I'm wondering, isn't that path enough for correct detection ? >>> I mean, if "/sys/class/dmi/id/sys_vendor" reports Xen (or KVM, or any >>> other known hypervisor), it's nested, otherwise it's on hardware ? >>> >>> Is that really mandatory to use CPUID leaves ? >> >> Let me ask the other way around: In user mode code under a non-nested >> vs nested Xen, what would you be able to derive from CPUID? The >> "hypervisor" bit is going to be set in both cases. (All assuming you >> run on new enough hardware+Xen such that CPUID would be intercepted >> even for PV.) > > I'm a bit clueless about CPUID stuff, but if I understand correctly, > you're essentially saying that using CPUID may not be the perfect way ? > Also, I don't get why the cpuid command returns two different values, > depending on the -k switch : > # cpuid -l 0x4000 > hypervisor_id (0x4000) = "\0\0\0\0\0\0\0\0\0\0\0\0" > # cpuid -k -l 0x4000 > hypervisor_id (0x4000) = "XenVMMXenVMM" > >> Yet relying on DMI is fragile, too: Along the lines of >> https://lists.xen.org/archives/html/xen-devel/2022-01/msg00604.html >> basically any value in there could be "inherited" from the host (i.e. >> from the layer below, to be precise). > > So using "/sys/class/dmi/id/sys_vendor", or simply doing "dmesg | grep > DMI:" is also not perfect, as values can be inherited/spoofed by > underneath hypervisor ? > >> The only way to be reasonably >> certain is to ask Xen about its view. The raw or host featuresets >> should give you this information, in the "mirror" of said respective >> CPUID leave's "hypervisor" bit. > > As said above, I'm clueless, can you expand please ? > >> But of course that still won't tell >> you which _kind_ of hypervisor is the immediate next one underneath >> Xen. >> >> This then further raises the question of what use it is to know the >> kind of the next level hypervisor, when multiple may be stacked on >> top of one another ... > > We need an answer from systemd guys, but the commiter expanded on the > reasons why the change was made [1] : "the detect_vm_cpuid check was > returning a VIRTUALIZATION_NONE result on non-nested dom0 (checking the > log from back then I was getting No virtualization found in CPUID), but > would report other CPUID-detectable hypervisors when dom0 was nested, so > we still wanted to check it for this case". > > Systemd is using this information to not start some services when > nested, like SMART/smartmontools (which have > "ConditionVirtualization=no" in their systemd unit files). > As the check now fails on baremetal dom0s, the service is not starting. Is this really wise? With any passed-through device the user likely wants such a service to start. Are they not misled with this effort? > > [1] https://github.com/systemd/systemd/issues/28113#issuecomment-1621559642 > -- Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions w: vates.tech | xcp-ng.org | xen-orchestra.com Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Rust Xen Guest Agent 0.4.0
A new pre-release of our guest agent prototype written in Rust is available, numbered 0.3.0 [1]. Identified issues and work to be done are tracked in Gitlab issue tracker [2]. As always, feedback will be greatly appreciated! Highlights: ### new features * can be linked statically with libxenstore to distribute a more standalone binary (`-F static`). Used for official Linux binary. Notably useful for guests running a RHEL-derived Linux distro. ### bugfixes * stale network information in xenstore is now removed on startup ### other noteworthy changes * CI pipelines stopped producing binaries for EOL'd FreeBSD 12.4, switched to 13.2 * CI now produces an (unofficial) binary for FreeBSD with Netlink support [1] https://gitlab.com/xen-project/xen-guest-agent/-/releases/0.4.0 [2] https://gitlab.com/xen-project/xen-guest-agent/-/issues Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Mirroring git repositories to gitlab
Hello all, The official project git repositories on https://xenbits.xen.org/ do not let people subscribe to get eg. notifications on push. A few repos are mirrored in https://gitlab.com/xen-project/ but it does not look like there are that many of them, aside from xen.git. I would love to have all repositories publicly mirrored there, maybe in a "xenbits-mirror" subgroub. Could we do that, or would there be any problem with it? Best regards, -- Yann Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Next steps for Rust guest agent
Current status: - primary goal: to have one guest agent all downstreams can use, in all guests (with Linux and FreeBSD already supported), as efficient as possible (with Netlink already supported on Linux) - developed at https://gitlab.com/xen-project/xen-guest-agent (till now using gitlab PRs) - works fine as a replacement for the Xenserver xe-guest-utilities Some points raised during the community call: - we likely want first to agree on a core set of collected information - could be made more configurable (eg. define a xenstore schema at runtime, we don't want specific schemas needs to cause forks) -> it could be the agent requesting a specific xenstore schema - what should be the criteria to advertise it as official Xenproject guest agent ? Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Rust Xen Guest Agent 0.3.0
A new pre-release of our guest agent prototype written in Rust is available, numbered 0.3.0 [1]. Identified issues and work to be done are tracked in Gitlab issue tracker [2]. As always, feedback will be greatly appreciated! Highlights: ### new features * available and total guest memory are now collected in FreeBSD guests * command-line flags `--stderr` and `--loglevel` were added to help troubleshooting ### behavior changes * logs are now sent to syslog by default on Unix-like OS ### bugfixes * the agent does not require the `libxenstore.so` symlink typically coming from Xen development package, only the runtime library package is now required * VIF hot(un)plug is now properly handled ### other noteworthy changes * executables and packages for supported guest platforms (currently Linux/Glibc and FreeBSD, both for x86_64 guests) are now available from Gitlab CI pipelines * APT repositories (though not signed) are now available from Gitlab CI pipelines * CI pipelines now testbuilds every commit in a merge request [1] https://gitlab.com/xen-project/xen-guest-agent/-/releases/0.3.0 [2] https://gitlab.com/xen-project/xen-guest-agent/-/issues Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
Re: Next steps for Rust guest agent
On 12/8/23 10:38, Yann Dirson wrote:
> Current status:
> - primary goal: to have one guest agent all downstreams can use, in all
> guests (with Linux and FreeBSD already supported), as efficient as
> possible (with Netlink already supported on Linux)
> - developed at https://gitlab.com/xen-project/xen-guest-agent (till now
> using gitlab PRs)
> - works fine as a replacement for the Xenserver xe-guest-utilities
Let's try to reboot the discussion.
> Some points raised during the community call:
> - we likely want first to agree on a core set of collected information
Currently I see the set of information collected as divided in the
following categories:
- those that are genuinely useful
- OS identifier (data/os_distro), and more detailed descriptive
string (data/os_name)
- kernel version (data/os_uname)
- IP addresses assigned to VIFs attached to the guest
- those that could be more useful but XAPI wants them
- free memory (data/meminfo_free) and total memory
(data/meminfo_total) according to guest OS (not necessarily well defined)
- control/feature-balloon=1 (necessary for XAPI's ballooning control
to do anything today)
- the version of the running agent, split in components
(attr/PVAddons/{Major,Minor,Micro,Build}Version) (including constraints
like Major being at least 1)
- those we provide for XAPI to be but without which it seems to be not
too sad, and I'd happily drop
- OS major and minor version (data/os_majorver, data/os_minorver)
What set of information (not necessarily from this list) do you think
would qualify as "core set of information to collect" ?
> - could be made more configurable (eg. define a xenstore schema at
> runtime, we don't want specific schemas needs to cause forks)
> -> it could be the agent requesting a specific xenstore schema
I do find some appeal to the idea that a toolstack should decide what
info the guest should give it and where. That could take the form of a
TBD string written to xenstore before the domain starts, e.g. matching
well-known IDs for pieces of information to xenstore paths.
> - what should be the criteria to advertise it as official Xenproject
> guest agent ?
What do people think here?
There is at least one known issue I'd like to address rapidly, which is
that the FreeBSD ports ship a buggy bash script [1] derived from
obsolete version of a XenServer tool. Maybe at least it's not necessary
to wait before approaching them to replace that old script with the Rust
agent in its current state?
[1]
https://github.com/freebsd/freebsd-ports/tree/main/sysutils/xe-guest-utilities
Best regards,
--
Yann
Yann Dirson | Vates Platform Developer
XCP-ng & Xen Orchestra - Vates solutions
web: https://vates.tech
Re: Community Process Group - Proposal
s. > > *Expectations* > CPG members are expected to use their best judgement of what is best for > the community in terms of conflict resolution and process improvements. > They can propose an outcome that progresses the project forward. > The CPG is also expected to address wider concerns, feedback, and ideas > during a monthly meeting with all community members. > > For example: > > * If someone is displaying repeated concerning behaviour that disrupts > the community, members can ask the CPG for help on a solution. (This > is different from a code of conduct violation which would be for > serious offences only.) > * Help drive discussions on how much we deviate from technical > specifications > > *Next steps* > Given this suggestion is a big change in what I hope is a positive > direction, we will require your feedback and a final formal vote on the > process, before it is implemented into the governance policies. The > specific wording can be decided after this proposal. > > This will hopefully help us overcome some of the frustrations and issues > we have seen in the community from a difference in opinion as a > collective discussion will now be made. Should we need to, the process > can be reviewed to improve at later stages. > > I welcome your feedback as a community on this proposal. > > Many thanks, > Kelly Choi > > Community Manager > Xen Project Yann Dirson | Vates Platform Developer XCP-ng & Xen Orchestra - Vates solutions web: https://vates.tech
