My guess is that the issue is one of these two: 1) version mismatch between Xen and Xen tools E.g. Xen from staging and Xen Tools from 4.18
2) missing Xen modules in Linux Dom0 Was /etc/init.d/xencommons run at boot? Is /proc/xen mounted? Are the privcmd, gntdev, xenfs, modules loaded? On Thu, 16 Nov 2023, Mario Marietto wrote: > I've found a similar error on this post,made on 2012 : > https://groups.google.com/g/qubes-devel/c/W1lM4ELuVVI > and according to what has been asked there,I want to post some further > relevant informations to help you to help me to debug > the problem : > root@devuan-bunsen:/Dati/xen# xl dmesg ---> > https://pastebin.ubuntu.com/p/YvtdCPwMWW/ > > root@devuan-bunsen:/Dati/xen# dmesg ---> > https://pastebin.ubuntu.com/p/9cNxCTXVrd/ > root@devuan-bunsen:/var/log/xen# mousepad xenstored-access.log ---> > https://pastebin.ubuntu.com/p/RTPBG9nS8R/ > root@devuan-bunsen:/var/log/xen# mousepad xenstored.log ---> > https://pastebin.ubuntu.com/p/T354ts33nP/ > > very thanks. > > On Thu, Nov 16, 2023 at 10:51 AM Mario Marietto <marietto2...@gmail.com> > wrote: > Hello to everyone. > > I'm trying to boot Linux 6.1.y as Xen dom0 on the Chromebook xe303c12, aka > Snow and configure and start a very basic domU > guest,following the Chuck's tutorial,located here : > > https://github.com/mobile-virt/u-boot-chromebook-xe303c12/tree/chromebook/xen#starting-a-domu-guest > > What I did has been to carefully follow his instructions,but I haven't found > a solution to fix this problem,yet : > > # sudo xl create devuan.cfg -c > > Parsing config from devuan.cfg > libxl: error: libxl_create.c:720:libxl__domain_make: domain creation fail: > Permission denied > libxl: error: libxl_create.c:1309:initiate_domain_create: cannot make domain: > -3 > > This is my devuan.cfg file : > > kernel = '/Dati/xen/kernels/zImage-6.1.59-stb-xen-cbe+' > memory = '512' > name = 'Devuan' > vcpus = '1' > disk = [ '/Dati/xen/devuan.img,,xvda,w' ] > extra = 'console=hvc0 root=/dev/xvda rw init=/sbin/init > xen-fbfront.video=24,1024,768' > > (I have tried also with root=/dev/xvda1 and root=/dev/xvda2,but leaving disk > = [ '/Dati/xen/devuan.img,,xvda,w' ] and not xvda1 or 2) > > I have no idea about the reason(s) I always get that error,but I don't think > it is caused by a wrong creation of the devuan.img > file. Can someone point me in the right direction to understand what could be > wrong ? I haven't found any useful information on the > internet. > > This is bootxen.scr file where I have configured dom0_mem=768 : > > mmc dev 1 > ext2load mmc 1:3 0x42000000 zImage-6.6.0-xen-iommu-dma-on-xen > ext2load mmc 1:3 0x51000000 xen-4.17-armhf-armmp-0x51004000.ub > ext2load mmc 1:3 0x5ffec000 exynos5250-snow.dtb > fdt addr 0x5ffec000 > fdt resize 1024 > fdt set /chosen \#address-cells <0x2> > fdt set /chosen \#size-cells <0x2> > fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 dom0_mem=768M > dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null" > fdt mknod /chosen dom0 > fdt set /chosen/dom0 compatible "xen,linux-zimage" "xen,multiboot-module" > "multiboot,module" > fdt set /chosen/dom0 reg <0x0 0x42000000 0x0 0x87C200 > > fdt set /chosen xen,dom0-bootargs "console=tty1 root=/dev/mmcblk1p4 rw > rootwait clk_ignore_unused --no-log" > bootm 0x51000000 - 0x5ffec000 > > and I've rebooted the Chromebook using this command : > > SMDK5250 # mmc dev 1 > SMDK5250 # ext2load mmc 1:3 0x50000000 bootxen.scr; source 0x50000000 > > This is the memory available on the machine after having booted the machine > ready for xen : > # free -m > total used free shared buff/cache > available > Mem: 741 329 108 7 332 > 412 > Swap: 0 0 0 > > Thanks in advance for any support. > > On Wed, Nov 15, 2023 at 8:41 PM Mario Marietto <marietto2...@gmail.com> wrote: > ---> So I plan to do some tests and see what DMA ops the other devices > use if swiotlb-xen is disabled and also what DMA > ops the other devices use when Linux runs on the Chromebook on bare > metal without Xen. If these tests show the problem > can be fixed by disabling swiotlb-xen with a Kconfig or command line > option, I will propose v2 to implement that as a > solution. > > and this could bring you to the next level of our project. Try to install xen > on different devices. At least it is my next > project. I've already bought two arm64 phones where xen can be installed > because there is a hack to overcome the bootloader / > hypervisor protection mechanism. For sure I hope that you also want to buy > them to work on this together. And don't worry about > how much money they will cost you. I've bought them used and refurbished. Or > you could buy only one,that I suggest could be the > SM-A600G (Samsung Galaxy A6) with Exynos7870. Please start looking for it at > a good price. > > On Wed, Nov 15, 2023 at 6:57 PM Chuck Zmudzinski <brchu...@netscape.net> > wrote: > On 11/14/2023 5:20 PM, Stefano Stabellini wrote: > > On Tue, 14 Nov 2023, Robin Murphy wrote: > >> On 11/11/2023 6:45 pm, Chuck Zmudzinski wrote: > >> > Enabling the new option, ARM_DMA_USE_IOMMU_XEN, fixes this error > when > >> > attaching the Exynos mixer in Linux dom0 on Xen on the Chromebook > Snow > >> > (and probably on other devices that use the Exynos mixer): > >> > > >> > [drm] Exynos DRM: using 14400000.fimd device for DMA mapping > operations > >> > exynos-drm exynos-drm: bound 14400000.fimd (ops 0xc0d96354) > >> > exynos-mixer 14450000.mixer: [drm:exynos_drm_register_dma] *ERROR* > Device > >> > 14450000.mixer lacks support for > IOMMU > >> > exynos-drm exynos-drm: failed to bind 14450000.mixer (ops > 0xc0d97554): -22 > >> > exynos-drm exynos-drm: adev bind failed: -22 > >> > exynos-dp: probe of 145b0000.dp-controller failed with error -22 > >> > > >> > Linux normally uses xen_swiotlb_dma_ops for DMA for all devices > when > >> > xen_swiotlb is detected even when Xen exposes an IOMMU to Linux. > Enabling > >> > the new config option allows devices such as the Exynos mixer to > use the > >> > IOMMU instead of xen_swiotlb_dma_ops for DMA and this fixes the > error. > >> > > >> > The new config option is not set by default because it is likely > some > >> > devices that use IOMMU for DMA on Xen will cause DMA errors and > memory > >> > corruption when Xen PV block and network drivers are in use on the > system. > >> > > >> > Link: > >> > > https://lore.kernel.org/xen-devel/acfab1c5-eed1-4930-8c70-8681e256c...@netscape.net/ > >> > > >> > Signed-off-by: Chuck Zmudzinski <brchu...@aol.com> > >> > --- > >> > The reported error with the Exynos mixer is not fixed by default > by adding > >> > a second patch to select the new option in the Kconfig definition > for the > >> > Exynos mixer if EXYNOS_IOMMU and SWIOTLB_XEN are enabled because > it is > >> > not certain setting the config option is suitable for all cases. > So it is > >> > necessary to explicitly select the new config option during the > config > >> > stage of the Linux kernel build to fix the reported error or > similar > >> > errors that have the same cause of lack of support for IOMMU on > Xen. This > >> > is necessary to avoid any regressions that might be caused by > enabling the > >> > new option by default for the Exynos mixer. > >> > arch/arm/mm/dma-mapping.c | 6 ++++++ > >> > drivers/xen/Kconfig | 16 ++++++++++++++++ > >> > 2 files changed, 22 insertions(+) > >> > > >> > diff --git a/arch/arm/mm/dma-mapping.c b/arch/arm/mm/dma-mapping.c > >> > index 5409225b4abc..ca04fdf01be3 100644 > >> > --- a/arch/arm/mm/dma-mapping.c > >> > +++ b/arch/arm/mm/dma-mapping.c > >> > @@ -1779,6 +1779,12 @@ void arch_setup_dma_ops(struct device *dev, > u64 > >> > dma_base, u64 size, > >> > if (iommu) > >> > arm_setup_iommu_dma_ops(dev, dma_base, size, iommu, > coherent); > >> > +#ifdef CONFIG_ARM_DMA_USE_IOMMU_XEN > >> > >> FWIW I don't think this really needs a config option - if Xen *has* > made an > >> IOMMU available, then there isn't really much reason not to use it, > and if for > >> some reason someone really didn't want to then they could simply > disable the > >> IOMMU driver anyway. > > > > The fact that the Exynos IOMMU is exposed to Linux is a mistake. Xen > > doesn't recognize the Exynos IOMMU (it is not one of the IOMMUs Xen > has > > a driver for) so it assigns the IOMMU to Dom0. It doesn't happen on > > purpose, it happens by accident. Certain things are going to break, > > specifically I am fairly certain PV drivers are going to break. > > > > If Xen recognized the Exynos IOMMU as an IOMMU it would probably hide > it > > from Dom0. (Today Xen doesn't have a list of IOMMUs Xen recognizes but > > doesn't have a driver for.) > > > > I think it is OK for Chuck and others to play around with this > > configuration but I wouldn't add a new kconfig option to Linux to > > support it. > > > > If we do want a kconfig option, I would add a kconfig option or Linux > > command line option to enable/disable swiotlb-xen. Basically a way to > > force-enable or force-disable xen_swiotlb_detect(). That could be > > generally useful for debugging and would also solve the problem here > as > > it could be used to force-disable swiotlb-xen. I would imagine that > the > > end result is the same: the default ops (iommu_ops) are used. > > I will try this. It isn't exactly what I have tested until now because > in all my tests so far all the DMA capable devices on the Chromebook use > swioltlb-xen except for the two devices that need to use the Exynos > IOMMU > to fix the error with the Exynos mixer. > > > > > > > > >> > + if (dev->dma_ops == &iommu_ops) { > >> > + dev->archdata.dma_ops_setup = true; > >> > >> The existing assignment is effectively unconditional by this point > anyway, so > >> could probably just be moved earlier to save duplicating it (or > perhaps just > >> make the xen_setup_dma_ops() call conditional instead to save the > early return > >> as well). > >> > >> However, are the IOMMU DMA ops really compatible with Xen? The > comments about > >> hypercalls and foreign memory in xen_arch_need_swiotlb() leave me > concerned > >> that assuming non-coherent DMA to any old Dom0 page is OK might not > actually > >> work in general :/ > > > > Xen has (not yet upstreaming) support for nested IOMMU (Xen uses the > > IOMMU while also it exposes a virtual IOMMU to guests.) In those cases > > the iommu_ops should be compatible with Xen. > > > > swiotlb-xen is useful in cases where there is no IOMMU on the platform > > (or the IOMMU doesn't cover all DMA-capable devices) and Dom0 is 1:1 > > mapped. See include/xen/arm/swiotlb-xen.h:xen_swiotlb_detect. If Dom0 > is > > not 1:1 mapped swiotlb-xen doesn't work. If an IOMMU is present and > > covers all DMA-capable devices, then swiotlb-xen is superfluous. > > It seems that swiotlb-xen works on this Chromebook since all but two > of the DMA capable devices use it when configured with the Kconfig > option > added here and it seems to work fine so I presume Dom0 is 1:1 mapped as > expected. It is possible that on this device, the IOMMU is only covering > the two devices that need to use the Exynos IOMMU in the tests I have > done. > There are many other DMA capable devices that use swiotlb-xen DMA ops > on Xen, but I have not checked what DMA ops the other devices use when > Linux runs on the Chromebook on bare metal without Xen. > > So I plan to do some tests and see what DMA ops the other devices use if > swiotlb-xen is disabled and also what DMA ops the other devices use when > Linux runs on the Chromebook on bare metal without Xen. If these tests > show the problem can be fixed by disabling swiotlb-xen with a Kconfig > or > command line option, I will propose v2 to implement that as a solution. > > > This last case is the interesting case for virtual IOMMU and Linux > usage of > > iommu_ops. > > > > -- > Mario. > > > > -- > Mario. > > > > -- > Mario. > >
