Re: Kexec and libxenctlr.so
On Thu, Jul 02, 2020 at 06:34:48PM +0100, Julien Grall wrote: > Hi Wei, > > On 26/06/2020 12:08, Wei Liu wrote: > > On Thu, Jun 11, 2020 at 03:57:37PM +0100, Julien Grall wrote: > > > Hi all, > > > > > > kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) > > > (see [1]). > > > > > > Given that the library has never been considered stable, it is probably a > > > disaster that is waiting to happen. > > > > > > Looking at the tree kexec is using the follow libxc functions: > > > - xc_kexec_get_range() > > > - xc_kexec_load() > > > - xc_kexec_unload() > > > - xc_kexec_status() > > > - xc_kexec_exec() > > > - xc_version() > > > - xc_interface_open() > > > - xc_interface_close() > > > - xc_get_max_cpus() > > > - xc_get_machine_memory_map() > > > > > > I think it is uncontroversial that we want a new stable library for all > > > the > > > xc_kexec_* functions (maybe libxenexec)? > > > > That sounds fine to me. > > > > Looking at the list of functions, all the xc_kexec_* ones are probably > > already rather stable. > > That's my understanding as well. > > Although, we may want to rethink some of the hypercalls (such as > KEXEC_cmd_kexec_get_range) in the future as they have different layout > between 32-bit and 64-bit. Thanksfully this wasn't exposed outside of libxc, > so it shouldn't be an issue to have a stable library. > Oh, that's good to hear. > > > > For xc_interface_open / close, they are perhaps used only to obtain an > > xc handle such that it can be used to make hypercalls. You new kexec > > library is going to expose its own handle with a xencall handle wrapped > > inside, so you can do away with an xc handle. > > I have already a PoC for the new library. I had to tweak a bit the list of > helpers as some use hypercalls arguments directly. Below, the proposed > interface: > > /* Callers who don't care don't need to #include */ > struct xentoollog_logger; > > typedef struct xenkexec_handle xenkexec_handle; > > typedef struct xenkexec_segments xenkexec_segments; > > xenkexec_handle *xenkexec_open(struct xentoollog_logger *logger, >unsigned int open_flags); > int xenkexec_close(xenkexec_handle *khdl); > > int xenkexec_exec(xenkexec_handle *khdl, int type); > int xenkexec_get_range(xenkexec_handle *khdl, int range, int nr, >uint64_t *size, uint64_t *start); > int xenkexec_load(xenkexec_handle *khdl, uint8_t type, uint16_t arch, > uint64_t entry_maddr, uint32_t nr_segments, > xenkexec_segments *segments); > int xenkexec_unload(xenkexec_handle *khdl, int type); > int xenkexec_status(xenkexec_handle *khdl, int type); > > xenkexec_segments *xenkexec_allocate_segments(xenkexec_handle *khdl, > unsigned int nr); > void xenkexec_free_segments(xenkexec_handle *khdl, xenkexec_segments *segs); > > int xenkexec_update_segment(xenkexec_handle *khdl, xenkexec_segments *segs, > unsigned int idx, void *buffer, size_t > buffer_size, > uint64_t dest_maddr, size_t dest_size); > You definitely have more experience in kexec than I do. This list looks sensible. > > > > > > > > > However I am not entirely sure where to put the others. > > > > > > I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a > > > XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite > > > it is a XENMEM_). > > > > > > > Introducing an libxensysctl before we stabilise sysctl interface seems > > wrong to me. We can bury the call inside libxenkexec itself for the time > > being. > > That would work for me. > > > > > > For xc_version(), I am thinking to extend libxentoolcore to also include > > > "stable xen API". > > > > > > > If you can do without an xc handle, do you still need to call > > xc_version? > > Looking at kexec, xc_version() is used by crashdump to determine which > architecture is used by Xen (in this case 32-bit x86 vs 64-bit x86). > > The was introcuded by commit: > > commit cdbc9b011fe43407908632d842e3a39e495e48d9 > Author: Ian Campbell > Date: Fri Mar 16 10:10:24 2007 + > > Set crash dump ELF header e_machine field based on underlying hypervisor > architecture. > > This is necessary when running Xen with a 64 bit hypervisor and 32 bit > domain 0 since the CPU crash notes will be 64 bit. > > Detecting the hypervisor archiecture requires libxenctrl and therefore > this > support is optional and disabled by default. > > Signed-off-by: Ian Campbell > Acked-by: Magnus Damm > Signed-off-by: Simon Horman > > As we drop support for 32-bit Xen quite a long time ago, we may be able to > remove the call to xc_version(). > Does Arm care about the bitness of the hypervisor? Wei. > Cheers, > > -- > Julien Grall
Re: Kexec and libxenctlr.so
Hi Wei, On 26/06/2020 12:08, Wei Liu wrote: On Thu, Jun 11, 2020 at 03:57:37PM +0100, Julien Grall wrote: Hi all, kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) (see [1]). Given that the library has never been considered stable, it is probably a disaster that is waiting to happen. Looking at the tree kexec is using the follow libxc functions: - xc_kexec_get_range() - xc_kexec_load() - xc_kexec_unload() - xc_kexec_status() - xc_kexec_exec() - xc_version() - xc_interface_open() - xc_interface_close() - xc_get_max_cpus() - xc_get_machine_memory_map() I think it is uncontroversial that we want a new stable library for all the xc_kexec_* functions (maybe libxenexec)? That sounds fine to me. Looking at the list of functions, all the xc_kexec_* ones are probably already rather stable. That's my understanding as well. Although, we may want to rethink some of the hypercalls (such as KEXEC_cmd_kexec_get_range) in the future as they have different layout between 32-bit and 64-bit. Thanksfully this wasn't exposed outside of libxc, so it shouldn't be an issue to have a stable library. For xc_interface_open / close, they are perhaps used only to obtain an xc handle such that it can be used to make hypercalls. You new kexec library is going to expose its own handle with a xencall handle wrapped inside, so you can do away with an xc handle. I have already a PoC for the new library. I had to tweak a bit the list of helpers as some use hypercalls arguments directly. Below, the proposed interface: /* Callers who don't care don't need to #include */ struct xentoollog_logger; typedef struct xenkexec_handle xenkexec_handle; typedef struct xenkexec_segments xenkexec_segments; xenkexec_handle *xenkexec_open(struct xentoollog_logger *logger, unsigned int open_flags); int xenkexec_close(xenkexec_handle *khdl); int xenkexec_exec(xenkexec_handle *khdl, int type); int xenkexec_get_range(xenkexec_handle *khdl, int range, int nr, uint64_t *size, uint64_t *start); int xenkexec_load(xenkexec_handle *khdl, uint8_t type, uint16_t arch, uint64_t entry_maddr, uint32_t nr_segments, xenkexec_segments *segments); int xenkexec_unload(xenkexec_handle *khdl, int type); int xenkexec_status(xenkexec_handle *khdl, int type); xenkexec_segments *xenkexec_allocate_segments(xenkexec_handle *khdl, unsigned int nr); void xenkexec_free_segments(xenkexec_handle *khdl, xenkexec_segments *segs); int xenkexec_update_segment(xenkexec_handle *khdl, xenkexec_segments *segs, unsigned int idx, void *buffer, size_t buffer_size, uint64_t dest_maddr, size_t dest_size); However I am not entirely sure where to put the others. I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite it is a XENMEM_). Introducing an libxensysctl before we stabilise sysctl interface seems wrong to me. We can bury the call inside libxenkexec itself for the time being. That would work for me. For xc_version(), I am thinking to extend libxentoolcore to also include "stable xen API". If you can do without an xc handle, do you still need to call xc_version? Looking at kexec, xc_version() is used by crashdump to determine which architecture is used by Xen (in this case 32-bit x86 vs 64-bit x86). The was introcuded by commit: commit cdbc9b011fe43407908632d842e3a39e495e48d9 Author: Ian Campbell Date: Fri Mar 16 10:10:24 2007 + Set crash dump ELF header e_machine field based on underlying hypervisor architecture. This is necessary when running Xen with a 64 bit hypervisor and 32 bit domain 0 since the CPU crash notes will be 64 bit. Detecting the hypervisor archiecture requires libxenctrl and therefore this support is optional and disabled by default. Signed-off-by: Ian Campbell Acked-by: Magnus Damm Signed-off-by: Simon Horman As we drop support for 32-bit Xen quite a long time ago, we may be able to remove the call to xc_version(). Cheers, -- Julien Grall
Re: Kexec and libxenctlr.so
On Thu, Jun 11, 2020 at 03:57:37PM +0100, Julien Grall wrote: > Hi all, > > kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) > (see [1]). > > Given that the library has never been considered stable, it is probably a > disaster that is waiting to happen. > > Looking at the tree kexec is using the follow libxc functions: >- xc_kexec_get_range() >- xc_kexec_load() >- xc_kexec_unload() >- xc_kexec_status() >- xc_kexec_exec() >- xc_version() >- xc_interface_open() >- xc_interface_close() >- xc_get_max_cpus() >- xc_get_machine_memory_map() > > I think it is uncontroversial that we want a new stable library for all the > xc_kexec_* functions (maybe libxenexec)? That sounds fine to me. Looking at the list of functions, all the xc_kexec_* ones are probably already rather stable. For xc_interface_open / close, they are perhaps used only to obtain an xc handle such that it can be used to make hypercalls. You new kexec library is going to expose its own handle with a xencall handle wrapped inside, so you can do away with an xc handle. > > However I am not entirely sure where to put the others. > > I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a > XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite > it is a XENMEM_). > Introducing an libxensysctl before we stabilise sysctl interface seems wrong to me. We can bury the call inside libxenkexec itself for the time being. > For xc_version(), I am thinking to extend libxentoolcore to also include > "stable xen API". > If you can do without an xc handle, do you still need to call xc_version? Wei. > Any opinion on the approach? > > Cheers, > > > [1] > https://git.kernel.org/pub/scm/utils/kernel/kexec/kexec-tools.git/commit/?id=894bea9335f57b62cbded7b02ab7d58308b647d8
Re: Kexec and libxenctlr.so
On 12/06/2020 10:18, Jürgen Groß wrote: I just wanted to point out that in case you need a new stable interface between kexec and the hypervisor you could consider using hypfs for that purpose instead of introducing a new hypercall. I am not looking to introduce a new interface with the hypervisor. Instead I am looking to have a stable library that can be used by Kexec. The underlying communication with the hypervisor doesn't need to be stable. Cheers, -- Julien Grall
Re: Kexec and libxenctlr.so
On 12.06.20 10:25, Julien Grall wrote: Hi Juergen, On 12/06/2020 05:25, Jürgen Groß wrote: On 11.06.20 18:00, Julien Grall wrote: On 11/06/2020 16:35, Jürgen Groß wrote: On 11.06.20 17:27, Julien Grall wrote: Hi, On 11/06/2020 16:21, Jürgen Groß wrote: On 11.06.20 16:57, Julien Grall wrote: Hi all, kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) (see [1]). Given that the library has never been considered stable, it is probably a disaster that is waiting to happen. Looking at the tree kexec is using the follow libxc functions: - xc_kexec_get_range() - xc_kexec_load() - xc_kexec_unload() - xc_kexec_status() - xc_kexec_exec() - xc_version() - xc_interface_open() - xc_interface_close() - xc_get_max_cpus() - xc_get_machine_memory_map() I think it is uncontroversial that we want a new stable library for all the xc_kexec_* functions (maybe libxenexec)? However I am not entirely sure where to put the others. I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite it is a XENMEM_). For xc_version(), I am thinking to extend libxentoolcore to also include "stable xen API". Any opinion on the approach? You could consider hypfs (at least for some of the functionality). That would work! xc_version() and xc_get_max_cpus() would be rather easy. I am guessing we will need a fallback to the normal hypercalls if hypfs is not present. Or we don't support kexec-tools running on a system without hypfs (or the related functions would return an error on those systems). AFAICT, hypfs allows you to modify runtime parameters which is not required for kexec. Well, and? kexec won't use this functionality. libxenctrl allows to create domains, which isn't required for kexec. And kexec doesn't do it. And so does libc... so that was clear not my point... We could still have the entry points for that functionality in libxenexec, which could use libxenhypfs (and maybe others). Such feature could be undesirable in some setup and therefore I don't think this is acceptable to impose that for kexec. If we really have setups where this would be an issue we'd need to modify the flask integration of hypfs to be able to disallow write access to hypfs. Or we could even add per-node access rights. ... Not everyone wants to use flask and I don't think this should be a condition to forbid runtime configuration change for all the system. You may want to set a minimal Xen with no runtime configuration support and no flask. Yet you may want to kexec for updating your Xen. Even with the runtime bits removed, I still don't think we should impose to build hypfs in the hypervisor given there are already always built hypercalls. There might be a misunderstanding here. I just wanted to point out that in case you need a new stable interface between kexec and the hypervisor you could consider using hypfs for that purpose instead of introducing a new hypercall. In case all hypercalls are already present, then fine. If not I don't see why introducing a new hypercall should be preferred over using an existing mechanism. Juergen
Re: Kexec and libxenctlr.so
Hi Juergen, On 12/06/2020 05:25, Jürgen Groß wrote: On 11.06.20 18:00, Julien Grall wrote: On 11/06/2020 16:35, Jürgen Groß wrote: On 11.06.20 17:27, Julien Grall wrote: Hi, On 11/06/2020 16:21, Jürgen Groß wrote: On 11.06.20 16:57, Julien Grall wrote: Hi all, kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) (see [1]). Given that the library has never been considered stable, it is probably a disaster that is waiting to happen. Looking at the tree kexec is using the follow libxc functions: - xc_kexec_get_range() - xc_kexec_load() - xc_kexec_unload() - xc_kexec_status() - xc_kexec_exec() - xc_version() - xc_interface_open() - xc_interface_close() - xc_get_max_cpus() - xc_get_machine_memory_map() I think it is uncontroversial that we want a new stable library for all the xc_kexec_* functions (maybe libxenexec)? However I am not entirely sure where to put the others. I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite it is a XENMEM_). For xc_version(), I am thinking to extend libxentoolcore to also include "stable xen API". Any opinion on the approach? You could consider hypfs (at least for some of the functionality). That would work! xc_version() and xc_get_max_cpus() would be rather easy. I am guessing we will need a fallback to the normal hypercalls if hypfs is not present. Or we don't support kexec-tools running on a system without hypfs (or the related functions would return an error on those systems). AFAICT, hypfs allows you to modify runtime parameters which is not required for kexec. Well, and? kexec won't use this functionality. libxenctrl allows to create domains, which isn't required for kexec. And kexec doesn't do it. And so does libc... so that was clear not my point... We could still have the entry points for that functionality in libxenexec, which could use libxenhypfs (and maybe others). Such feature could be undesirable in some setup and therefore I don't think this is acceptable to impose that for kexec. If we really have setups where this would be an issue we'd need to modify the flask integration of hypfs to be able to disallow write access to hypfs. Or we could even add per-node access rights. ... Not everyone wants to use flask and I don't think this should be a condition to forbid runtime configuration change for all the system. You may want to set a minimal Xen with no runtime configuration support and no flask. Yet you may want to kexec for updating your Xen. Even with the runtime bits removed, I still don't think we should impose to build hypfs in the hypervisor given there are already always built hypercalls. Cheers, -- Julien Grall
Re: Kexec and libxenctlr.so
On 11.06.20 18:00, Julien Grall wrote: On 11/06/2020 16:35, Jürgen Groß wrote: On 11.06.20 17:27, Julien Grall wrote: Hi, On 11/06/2020 16:21, Jürgen Groß wrote: On 11.06.20 16:57, Julien Grall wrote: Hi all, kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) (see [1]). Given that the library has never been considered stable, it is probably a disaster that is waiting to happen. Looking at the tree kexec is using the follow libxc functions: - xc_kexec_get_range() - xc_kexec_load() - xc_kexec_unload() - xc_kexec_status() - xc_kexec_exec() - xc_version() - xc_interface_open() - xc_interface_close() - xc_get_max_cpus() - xc_get_machine_memory_map() I think it is uncontroversial that we want a new stable library for all the xc_kexec_* functions (maybe libxenexec)? However I am not entirely sure where to put the others. I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite it is a XENMEM_). For xc_version(), I am thinking to extend libxentoolcore to also include "stable xen API". Any opinion on the approach? You could consider hypfs (at least for some of the functionality). That would work! xc_version() and xc_get_max_cpus() would be rather easy. I am guessing we will need a fallback to the normal hypercalls if hypfs is not present. Or we don't support kexec-tools running on a system without hypfs (or the related functions would return an error on those systems). AFAICT, hypfs allows you to modify runtime parameters which is not required for kexec. Well, and? kexec won't use this functionality. libxenctrl allows to create domains, which isn't required for kexec. And kexec doesn't do it. We could still have the entry points for that functionality in libxenexec, which could use libxenhypfs (and maybe others). Such feature could be undesirable in some setup and therefore I don't think this is acceptable to impose that for kexec. If we really have setups where this would be an issue we'd need to modify the flask integration of hypfs to be able to disallow write access to hypfs. Or we could even add per-node access rights. Juergen
Re: Kexec and libxenctlr.so
On 11/06/2020 16:35, Jürgen Groß wrote: On 11.06.20 17:27, Julien Grall wrote: Hi, On 11/06/2020 16:21, Jürgen Groß wrote: On 11.06.20 16:57, Julien Grall wrote: Hi all, kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) (see [1]). Given that the library has never been considered stable, it is probably a disaster that is waiting to happen. Looking at the tree kexec is using the follow libxc functions: - xc_kexec_get_range() - xc_kexec_load() - xc_kexec_unload() - xc_kexec_status() - xc_kexec_exec() - xc_version() - xc_interface_open() - xc_interface_close() - xc_get_max_cpus() - xc_get_machine_memory_map() I think it is uncontroversial that we want a new stable library for all the xc_kexec_* functions (maybe libxenexec)? However I am not entirely sure where to put the others. I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite it is a XENMEM_). For xc_version(), I am thinking to extend libxentoolcore to also include "stable xen API". Any opinion on the approach? You could consider hypfs (at least for some of the functionality). That would work! xc_version() and xc_get_max_cpus() would be rather easy. I am guessing we will need a fallback to the normal hypercalls if hypfs is not present. Or we don't support kexec-tools running on a system without hypfs (or the related functions would return an error on those systems). AFAICT, hypfs allows you to modify runtime parameters which is not required for kexec. Such feature could be undesirable in some setup and therefore I don't think this is acceptable to impose that for kexec. xc_get_machine_memory_map() is using a stable hypercall used by the kernel, too. IIUC, you are suggesting to put this one in hypfs library as well. Is that correct? Not really. I wanted to point out that this call would be a good candidate for another stable library, maybe part of libxenexec? In theory the memory map could be dumped via a hypfs node, either as a string (not nice for working with it) or as binary blob, of course. Juergen -- Julien Grall
Re: Kexec and libxenctlr.so
On 11.06.20 17:27, Julien Grall wrote: Hi, On 11/06/2020 16:21, Jürgen Groß wrote: On 11.06.20 16:57, Julien Grall wrote: Hi all, kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) (see [1]). Given that the library has never been considered stable, it is probably a disaster that is waiting to happen. Looking at the tree kexec is using the follow libxc functions: - xc_kexec_get_range() - xc_kexec_load() - xc_kexec_unload() - xc_kexec_status() - xc_kexec_exec() - xc_version() - xc_interface_open() - xc_interface_close() - xc_get_max_cpus() - xc_get_machine_memory_map() I think it is uncontroversial that we want a new stable library for all the xc_kexec_* functions (maybe libxenexec)? However I am not entirely sure where to put the others. I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite it is a XENMEM_). For xc_version(), I am thinking to extend libxentoolcore to also include "stable xen API". Any opinion on the approach? You could consider hypfs (at least for some of the functionality). That would work! xc_version() and xc_get_max_cpus() would be rather easy. I am guessing we will need a fallback to the normal hypercalls if hypfs is not present. Or we don't support kexec-tools running on a system without hypfs (or the related functions would return an error on those systems). xc_get_machine_memory_map() is using a stable hypercall used by the kernel, too. IIUC, you are suggesting to put this one in hypfs library as well. Is that correct? Not really. I wanted to point out that this call would be a good candidate for another stable library, maybe part of libxenexec? In theory the memory map could be dumped via a hypfs node, either as a string (not nice for working with it) or as binary blob, of course. Juergen
Re: Kexec and libxenctlr.so
Hi, On 11/06/2020 16:21, Jürgen Groß wrote: On 11.06.20 16:57, Julien Grall wrote: Hi all, kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) (see [1]). Given that the library has never been considered stable, it is probably a disaster that is waiting to happen. Looking at the tree kexec is using the follow libxc functions: - xc_kexec_get_range() - xc_kexec_load() - xc_kexec_unload() - xc_kexec_status() - xc_kexec_exec() - xc_version() - xc_interface_open() - xc_interface_close() - xc_get_max_cpus() - xc_get_machine_memory_map() I think it is uncontroversial that we want a new stable library for all the xc_kexec_* functions (maybe libxenexec)? However I am not entirely sure where to put the others. I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite it is a XENMEM_). For xc_version(), I am thinking to extend libxentoolcore to also include "stable xen API". Any opinion on the approach? You could consider hypfs (at least for some of the functionality). That would work! xc_version() and xc_get_max_cpus() would be rather easy. I am guessing we will need a fallback to the normal hypercalls if hypfs is not present. xc_get_machine_memory_map() is using a stable hypercall used by the kernel, too. IIUC, you are suggesting to put this one in hypfs library as well. Is that correct? Thank you for the input! Cheers, -- Julien Grall
Re: Kexec and libxenctlr.so
On 11.06.20 16:57, Julien Grall wrote: Hi all, kexec-tools has an option to load dynamically libxenctlr.so (not .so.4.x) (see [1]). Given that the library has never been considered stable, it is probably a disaster that is waiting to happen. Looking at the tree kexec is using the follow libxc functions: - xc_kexec_get_range() - xc_kexec_load() - xc_kexec_unload() - xc_kexec_status() - xc_kexec_exec() - xc_version() - xc_interface_open() - xc_interface_close() - xc_get_max_cpus() - xc_get_machine_memory_map() I think it is uncontroversial that we want a new stable library for all the xc_kexec_* functions (maybe libxenexec)? However I am not entirely sure where to put the others. I am thinking to introduce libxensysctl for xc_get_max_cpus() as it is a XEN_SYSCTL. We could possibly include xc_get_machine_memory_map() (despite it is a XENMEM_). For xc_version(), I am thinking to extend libxentoolcore to also include "stable xen API". Any opinion on the approach? You could consider hypfs (at least for some of the functionality). xc_version() and xc_get_max_cpus() would be rather easy. xc_get_machine_memory_map() is using a stable hypercall used by the kernel, too. Juergen
