On 11/27/2023 9:28 PM, Stefano Stabellini wrote: > On Mon, 27 Nov 2023, Chuck Zmudzinski wrote: >> On 11/27/2023 10:22 AM, Chuck Zmudzinski wrote: >> > On 11/27/2023 7:45 AM, Mario Marietto wrote: >> >> @Chuck Zmudzinski <mailto:brchu...@netscape.net> : Stay tuned. They want >> >> to help us. The xen developers are great. Very good support for us. I'm >> >> sure that you can give a good contribution to understand what's our >> >> problem and how to implement a fix with the help of all those good guys. >> >> >> >> On Mon, Nov 27, 2023 at 11:56 AM Roger Pau Monné <roger....@citrix.com >> >> <mailto:roger....@citrix.com>> wrote: >> >> >> >> On Mon, Nov 27, 2023 at 10:28:13AM +0000, Henry Wang wrote: >> >> > +(xen-devel and Arm maintainers, including Julien) >> >> > >> >> > > On Nov 27, 2023, at 18:03, Mario Marietto <marietto2...@gmail.com >> >> <mailto:marietto2...@gmail.com>> >> >> > > wrote: >> >> > > >> >> > > Hello. We have just virtualized Debian 12 on our arm (32 bit) >> >> > > Chromebook model xe303c12 . As host / dom0 we have chosen Devuan >> >> > > 5,and for guest / domU,Debian 12. It works great. But our goal is >> >> > > different. We want to virtualize FreeBSD as domU. Can we have a >> >> > > working Xen PV network driver for a FreeBSD arm guest ?. I found >> >> > > that Julien Grall has ported the Xen drivers to FreeBSD on arm. I >> >> > > would like to know if Julien's work was accepted upstream by >> >> > > FreeBSD, in which case FreeBSD as a Xen guest on arm should work >> >> > > if we enable the Xen PV drivers in the FreeBSD on arm kernel. If >> >> > > Julien's work was not accepted upstream by FreeBSD, we will have >> >> > > to find his patches and apply them ourselves to the FreeBSD on arm >> >> > > kernel. >> >> >> >> I've added Elliot on Cc as he is working on upstreaming the patches to >> >> FreeBSD. He will be able to provide a better update than myself. >> >> >> >> Regards, Roger. >> > >> > I have been collaborating with Mario, and I can explain what we have done >> > so far : >> > >> > We are using Julien's patch set against an old development version of >> > FreeBSD 11 >> > from 2014-12-03 : >> > >> > https://xenbits.xen.org/gitweb/?p=people/julieng/freebsd.git;a=shortlog;h=refs/heads/xen-arm-v2.2 >> > >> > We successfully built the XENVIRT kernel and FreeBSD world, and created the >> > FreeBSD rootfs according to Julien's instructions here : >> > >> > https://lists.freebsd.org/pipermail/freebsd-xen/2014-November/002202.html >> > >> > There were some adjustments to the instructions : >> > >> > To build the kernel, we used : >> > >> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENVIRT buildkernel >> > >> > instead of >> > >> > $ sudo make TARGET_ARCH=armv6 KERNCONF=XENHVM buildkernel >> > >> > The FreeBSD 'kernel' file is in ELF format and did not work, and we spent >> > some time trying to convert it to the zImage format without realizing the >> > build of the FreeBSD kernel creates the 'kernel.bin' file in the zImage >> > format. >> > So when booting with the 'kernel.bin' file instead, it actually boots : >> > >> > user@devuan-bunsen ~ % sudo xl create freebsd.cfg >> > Parsing config from freebsd.cfg >> > user@devuan-bunsen ~ % sudo xl li >> > Name ID Mem VCPUs State Time(s) >> > Domain-0 0 768 2 r----- >> > 1439.4 >> > freebsd 1 1152 1 r----- >> > 3.0 >> > user@devuan-bunsen ~ % >> > >> > However, the guest is still not working correctly : >> > >> > 1. Attaching the console with the -c option at creation or with >> > 'xl console freebsd' results in no output to the console. >> > >> > 2. The timestamp on the virtual disk image file shows that the filesystem >> > was at best mounted read-only, if it was mounted at all by the guest >> > FreeBSD kernel. >> > >> > 3. The 'xl shutdown freebsd' command does not work, it just hangs. To stop >> > the guest, you need to do 'xl destroy freebsd'. >> > >> > However, I think we can get the console to work and the rootfs to mount >> > because I >> > just realized I forgot to do the steps from Julien's instructions of >> > editing the >> > /etc/fstab and /etc/ttys files in the FreeBSD rootfs : >> > >> > $ echo "/dev/xbd0 / ufs rw 1 1" > /mnt/etc/fstab >> > $ vi /mnt/etc/ttys (add the line 'xc0 "/usr/libexec/getty Pc" xterm on >> > secure") >> > >> > I will add those and see if the console and disk are working. >> >> Unfortunately, adding xc0 to /etc/ttys and /dev/xbd0 as the root device in >> /etc/fstab did not make the console or disk work. Still no output on the >> xen console from the guest kernel, and the timestamp on the rootfs image >> file did not change so it did not mount read-write. >> >> We could use some advice for troubleshooting this. Now, we are blind because >> we are not getting any xen console output But I am pleased we were able to >> demonstrate that Julien's old patch set for FreeBSD 11 allows us to boot >> FreeBSD 11 on a modern version of Xen on arm - we are using the Debian >> stable Xen 4.17 packages. > > You can use the DEBUG hypercalls to check how far we got into the > booting process: > https://wiki.xenproject.org/wiki/Xen_ARM_DEBUG_hypercalls > > For instance add the following to FreeBSD code: > > asm volatile("hvc 0xfffd"); >
It took me a while, but I finally got this approach to work to debug the FreeBSD kernel. Thanks! The problem was the compiler was reporting hvc is an invalid instruction. To get the compiler to accept the hvc instruction as valid, I first spent quite a bit of time porting the patches from the old development version of FreeBSD 11 on which Julien's patches were based to FreeBSD 12.4, because that old development version of FreeBSD did not support armv7 but only armv6, and I thought maybe the compiler is rejecting the hvc instruction because the kernel build was targeting armv6 and I was not sure hypervisor extensions were available for armv6. But FreeBSD 12 and higher has support to target armv7 for the kernel. There were quite a few changes to account for between FreeBSD 11 and FreeBSD 12.4, I had to add about 12 more patches, but I also removed some of Julien's patches that were either applied in FreeBSD 12.4 or no longer applicable to FreeBSD 12.4. So when I was able to build a FreeBSD 12.4 kernel + Julien's arm/xenvirt patches targeting armv7 instead of armv6, I got the same behavior: the guest started but no output on the console, and the compiler at first still did not accept the hvc instruction (FreeBSD uses the clang compiler by default to build the kernel). After some searches on the Internet I discovered that adding the -mthumb CFLAG when compiling the objects with an hvc instruction enabled the compiler to accept the hvc instruction. So I was able to get output like this in the dom0 Xen console log from the hvc instruction in the guest : (XEN) arch/arm/traps.c:1983:d2v0 HSR=0x80000005 pc=0xffff000c gva=0xffff000c gpa=0x000000ffff000c (XEN) arch/arm/traps.c:1983:d2v0 HSR=0x80000005 pc=0xffff000c gva=0xffff000c gpa=0x000000ffff000c ... For now, I only put one hvc instruction in the FreeBSD code - it is where the kernel prints the copyright and version information to the console. So I don't understand why the message from the hvc instruction is appearing multiple times in the Xen logs... In any case, this provides a way to debug the boot of FreeBSD / arm on Xen, so thanks, Stefano, for this suggestion! Cheers, Chuck
