Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 11/11/2021 16:40, Peter Maydell wrote: > On Mon, 8 Nov 2021 at 18:05, David Fernandez wrote: >> I couldn't get the stock qemu-system-x86_04 to boot correctly, as it was >> an old version 2.11.1, I decided to recompile from sources to see if >> that would fix the problem, but the problem still persists, using both >> top of master and stable-2.12 (currently on that).[ TIME ] Timed out >> waiting for device dev-ttyS0.device. > FWIW, I tried to repro this on an aarch64 server I have access to, > and I don't see that error -- the system boots (eventually) to > the installer UI. > > QEMU version built: upstream git, commit 70f872ca916ac45. > > Configure options: > --target-list=x86_64-softmmu --disable-tools --disable-docs > > vmlinuz and initrd extracted from the iso with > isoinfo -i ubuntu-18.04.6-live-server-amd64.iso -x '/CASPER/INITRD.;1' > > initrd > isoinfo -i ubuntu-18.04.6-live-server-amd64.iso -x > '/CASPER/VMLINUZ.;1' > vmlinuz > > QEMU command: > ./build/tgt-x86/qemu-system-x86_64 -boot order=dc,menu=on -cdrom > ubuntu-18.04.6-live-server-amd64.iso -nographic -serial mon:stdio -m > 16384 -kernel /tmp/vmlinuz -initrd /tmp/initrd -append 'boot=casper > console=ttyS0 ---' > > (I didn't bother creating any of the devices for this test, so as > you note there's a bunch of harmless complaints from the kernel > floppy driver.) > > Total time to reach first screen of the text installer: 5m26s > > It's maybe also worth mentioning that "emulate x86-64" is, although > a supported use case, not one that as far as I'm aware anybody is > paid to work on, so enhancements and bugfixes to it are largely > done on a volunteer basis. (This is as distinct from, for example, > "using KVM on either x86-64 or aarch64" and "emulation of aarch64 > on x86-64", which have more people working on them.) > > -- PMM For whatever is worth, I compiled v6.1.0 on my Fedora Intel laptop and the issue is not there either, so this seems closely related to aarch64 and most likely to Ubuntu 18.04.5 as a host... and I am more and more afraid that it might have to do with the kernel configuration for the Jetson... I can recompile it if needed, so if there is some test that needs to be done to find out, I am happy to do.
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 11/11/2021 16:10, Peter Maydell wrote: > On Mon, 8 Nov 2021 at 18:05, David Fernandez wrote: >> ../configure \ >> --enable-lto \ > Does disabling LTO make a difference? That's about the only > thing in the configure options that stands out as maybe > making a difference. > > -- PMM Hi Peter, That option is not available anymore in the tagged versions... it was only available in the stable-2.12. I believe I mentioned that in my previous emails... the last email sumarizing does not have it anymore, as I updated things according to my testing with v5.2.0 and v6.1.0 It soes not seem to make any difference. Regards
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 12/11/2021 10:27, Thomas Huth wrote: > [No suele recibir correo electrónico de th...@redhat.com. Obtenga > información acerca de por qué esto es importante en > http://aka.ms/LearnAboutSenderIdentification.] > > On 11/11/2021 17.10, Peter Maydell wrote: >> On Mon, 8 Nov 2021 at 18:05, David Fernandez >> wrote: >>> >>> ../configure \ >> >>> --enable-lto \ >> >> Does disabling LTO make a difference? That's about the only >> thing in the configure options that stands out as maybe >> making a difference. > > Please don't use LTO on non-x86 hosts, there are some known problems with > these optimizations (though the symptoms look rather differently to > what has > been described here). > > Thomas > Sure, as mentioned in my previous response, it was only available for the branched versions, but not available anymore for v5.2.0 and v6.1.0 or top of master, which are the latest ones I have been testing. I did see also problems with lto when compiling for arm, so I tend to be away from them, but I did try it as I assumed people use it in general. Regards
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 11/11/2021 17.10, Peter Maydell wrote: On Mon, 8 Nov 2021 at 18:05, David Fernandez wrote: ../configure \ --enable-lto \ Does disabling LTO make a difference? That's about the only thing in the configure options that stands out as maybe making a difference. Please don't use LTO on non-x86 hosts, there are some known problems with these optimizations (though the symptoms look rather differently to what has been described here). Thomas
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On Mon, 8 Nov 2021 at 18:05, David Fernandez wrote: > I couldn't get the stock qemu-system-x86_04 to boot correctly, as it was > an old version 2.11.1, I decided to recompile from sources to see if > that would fix the problem, but the problem still persists, using both > top of master and stable-2.12 (currently on that).[ TIME ] Timed out > waiting for device dev-ttyS0.device. FWIW, I tried to repro this on an aarch64 server I have access to, and I don't see that error -- the system boots (eventually) to the installer UI. QEMU version built: upstream git, commit 70f872ca916ac45. Configure options: --target-list=x86_64-softmmu --disable-tools --disable-docs vmlinuz and initrd extracted from the iso with isoinfo -i ubuntu-18.04.6-live-server-amd64.iso -x '/CASPER/INITRD.;1' > initrd isoinfo -i ubuntu-18.04.6-live-server-amd64.iso -x '/CASPER/VMLINUZ.;1' > vmlinuz QEMU command: ./build/tgt-x86/qemu-system-x86_64 -boot order=dc,menu=on -cdrom ubuntu-18.04.6-live-server-amd64.iso -nographic -serial mon:stdio -m 16384 -kernel /tmp/vmlinuz -initrd /tmp/initrd -append 'boot=casper console=ttyS0 ---' (I didn't bother creating any of the devices for this test, so as you note there's a bunch of harmless complaints from the kernel floppy driver.) Total time to reach first screen of the text installer: 5m26s It's maybe also worth mentioning that "emulate x86-64" is, although a supported use case, not one that as far as I'm aware anybody is paid to work on, so enhancements and bugfixes to it are largely done on a volunteer basis. (This is as distinct from, for example, "using KVM on either x86-64 or aarch64" and "emulation of aarch64 on x86-64", which have more people working on them.) -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On Mon, 8 Nov 2021 at 18:05, David Fernandez wrote: > > ../configure \ >--enable-lto \ Does disabling LTO make a difference? That's about the only thing in the configure options that stands out as maybe making a difference. -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 11/11/2021 13:42, Peter Maydell wrote: > [No suele recibir correo electrónico de peter.mayd...@linaro.org. Obtenga > información acerca de por qué esto es importante en > http://aka.ms/LearnAboutSenderIdentification.] > > On Thu, 11 Nov 2021 at 12:45, David Fernandez wrote: >> On 11/11/2021 11:23, Peter Maydell wrote: >>>The problem is that the >>> udev machinery that creates nodes in /dev is being too slow >>> (or possibly is failing for some other reason, but given all the >>> other timeouts I'm guessing "everything is too slow") and so >>> the systemd unit that is waiting for /dev/ttyS0 to be created >>> times out. >> What is a bit puzzling is that this is supposed to all run in an >> emulated machine having its own simulated time, so yes things are slow, >> but everything should happen as expected, just slowly. > That's not the way QEMU's emulation of time works. In non-icount > mode, which is the default, wall clock time in the VM follows > wall clock time in the outside world. The guest just sees > itself as running on a rather slow CPU (and one with some odd > performance characteristics about what is slow compared to > what other things). I see... I wonder if a redhat system is likely to run better in such a situation, or if any other debian/ubuntu distro is known to run better when host performance might have an impact. >> I guess I will compile from sources on Fedora and see if I get the same >> problem, as it is a bit hard to believe that the only way to run qemu is >> to have a high end machine dedicated just to run an install cd. > There's probably something odd going on, it's just not clear > what and trying to diagnose it is going to be really hard. > It is the case that if the host system is underpowered then > it's not going to be able to run complicated guests in > an acceptably performant way, but that ought to apply more > to situations like "I want to emulate a Windows guest on my > first-generation raspberry pi". > > What does 'file' say about the QEMU binary you're running > on the aarch64 system? (This is a check to eliminate an > almost-certainly-not-the-problem theory.) sen@vpm-devkit:~$ which qemu-system-x86_64 /usr/local/bin/qemu-system-x86_64 sen@vpm-devkit:~$ file /usr/local/bin/qemu-system-x86_64 /usr/local/bin/qemu-system-x86_64: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=e05b26921bd35d96b6c749d23d5bfa5e6e43ab4c, stripped > -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 11/11/2021 13:42, Peter Maydell wrote: > [No suele recibir correo electrónico de peter.mayd...@linaro.org. Obtenga > información acerca de por qué esto es importante en > http://aka.ms/LearnAboutSenderIdentification.] > > On Thu, 11 Nov 2021 at 12:45, David Fernandez wrote: >> On 11/11/2021 11:23, Peter Maydell wrote: >>>The problem is that the >>> udev machinery that creates nodes in /dev is being too slow >>> (or possibly is failing for some other reason, but given all the >>> other timeouts I'm guessing "everything is too slow") and so >>> the systemd unit that is waiting for /dev/ttyS0 to be created >>> times out. >> What is a bit puzzling is that this is supposed to all run in an >> emulated machine having its own simulated time, so yes things are slow, >> but everything should happen as expected, just slowly. > That's not the way QEMU's emulation of time works. In non-icount > mode, which is the default, wall clock time in the VM follows > wall clock time in the outside world. The guest just sees > itself as running on a rather slow CPU (and one with some odd > performance characteristics about what is slow compared to > what other things). I see... I wonder if a redhat system is likely to run better in such a situation, or if any other debian/ubuntu distro is known to run better when host performance might have an impact. >> I guess I will compile from sources on Fedora and see if I get the same >> problem, as it is a bit hard to believe that the only way to run qemu is >> to have a high end machine dedicated just to run an install cd. > There's probably something odd going on, it's just not clear > what and trying to diagnose it is going to be really hard. > It is the case that if the host system is underpowered then > it's not going to be able to run complicated guests in > an acceptably performant way, but that ought to apply more > to situations like "I want to emulate a Windows guest on my > first-generation raspberry pi". > > What does 'file' say about the QEMU binary you're running > on the aarch64 system? (This is a check to eliminate an > almost-certainly-not-the-problem theory.) sen@vpm-devkit:~$ which qemu-system-x86_64 /usr/local/bin/qemu-system-x86_64 sen@vpm-devkit:~$ file /usr/local/bin/qemu-system-x86_64 /usr/local/bin/qemu-system-x86_64: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=e05b26921bd35d96b6c749d23d5bfa5e6e43ab4c, stripped > -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On Thu, 11 Nov 2021 at 12:45, David Fernandez wrote: > On 11/11/2021 11:23, Peter Maydell wrote: > > The problem is that the > > udev machinery that creates nodes in /dev is being too slow > > (or possibly is failing for some other reason, but given all the > > other timeouts I'm guessing "everything is too slow") and so > > the systemd unit that is waiting for /dev/ttyS0 to be created > > times out. > > What is a bit puzzling is that this is supposed to all run in an > emulated machine having its own simulated time, so yes things are slow, > but everything should happen as expected, just slowly. That's not the way QEMU's emulation of time works. In non-icount mode, which is the default, wall clock time in the VM follows wall clock time in the outside world. The guest just sees itself as running on a rather slow CPU (and one with some odd performance characteristics about what is slow compared to what other things). > I guess I will compile from sources on Fedora and see if I get the same > problem, as it is a bit hard to believe that the only way to run qemu is > to have a high end machine dedicated just to run an install cd. There's probably something odd going on, it's just not clear what and trying to diagnose it is going to be really hard. It is the case that if the host system is underpowered then it's not going to be able to run complicated guests in an acceptably performant way, but that ought to apply more to situations like "I want to emulate a Windows guest on my first-generation raspberry pi". What does 'file' say about the QEMU binary you're running on the aarch64 system? (This is a check to eliminate an almost-certainly-not-the-problem theory.) -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 11/11/2021 11:23, Peter Maydell wrote: > [No suele recibir correo electrónico de peter.mayd...@linaro.org. Obtenga > información acerca de por qué esto es importante en > http://aka.ms/LearnAboutSenderIdentification.] > > On Wed, 10 Nov 2021 at 18:56, David Fernandez wrote: >> As I have not hear anything yet, I thought I would summarize the current >> status >> for this problem. Please, let me know if any other tests or information are >> needed. >> >> I am running qemu-system-x86_64 v5.2.0 (also tried v6.1.0 and top of >> master) on: >> - aarch64 (Jetson AGX Xavier) with Ubuntu 18.04.5 as a host >> (compiled from >>git sources as distro version for it was 2.11, which is too old), >> and on >> - x86_64 (my laptop) with Fedora 34 as a host (here the >> qemu-system-x86_64 >>distro version is 5.2.0). >> >> Running Ubuntu 18.04.6 server install cdrom (also tried Ubuntu 20.04.3) >> as the >> guest. >> >> The following services fail on the Jetson, but not on the laptop. The >> first one >> is the ttyS0 console, which seems the most important thing as it is provided >> directly by the virtual emulation (-serial mon:stdio): >> >> [ TIME ] Timed out waiting for device dev-ttyS0.device. > I'm pretty sure this isn't actually a problem with the emulation > of the serial device (after all you are seeing all these messages > so far on the serial console, right?). Right, may be it is not, I do not know what the problem is. > The problem is that the > udev machinery that creates nodes in /dev is being too slow > (or possibly is failing for some other reason, but given all the > other timeouts I'm guessing "everything is too slow") and so > the systemd unit that is waiting for /dev/ttyS0 to be created > times out. What is a bit puzzling is that this is supposed to all run in an emulated machine having its own simulated time, so yes things are slow, but everything should happen as expected, just slowly. I guess I will compile from sources on Fedora and see if I get the same problem, as it is a bit hard to believe that the only way to run qemu is to have a high end machine dedicated just to run an install cd. If the qemu compiled from sources on Fedora fails, then we should conclude that there is something about the distro supplied qemu that is not properly done when compiling from sources as I do... may be my fault or a bug, but As mentioned, let me know if you know of something that could fix this. Regards > > -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On Wed, 10 Nov 2021 at 18:56, David Fernandez wrote: > > As I have not hear anything yet, I thought I would summarize the current > status > for this problem. Please, let me know if any other tests or information are > needed. > > I am running qemu-system-x86_64 v5.2.0 (also tried v6.1.0 and top of > master) on: > - aarch64 (Jetson AGX Xavier) with Ubuntu 18.04.5 as a host > (compiled from > git sources as distro version for it was 2.11, which is too old), > and on > - x86_64 (my laptop) with Fedora 34 as a host (here the > qemu-system-x86_64 > distro version is 5.2.0). > > Running Ubuntu 18.04.6 server install cdrom (also tried Ubuntu 20.04.3) > as the > guest. > > The following services fail on the Jetson, but not on the laptop. The > first one > is the ttyS0 console, which seems the most important thing as it is provided > directly by the virtual emulation (-serial mon:stdio): > > [ TIME ] Timed out waiting for device dev-ttyS0.device. I'm pretty sure this isn't actually a problem with the emulation of the serial device (after all you are seeing all these messages so far on the serial console, right?). The problem is that the udev machinery that creates nodes in /dev is being too slow (or possibly is failing for some other reason, but given all the other timeouts I'm guessing "everything is too slow") and so the systemd unit that is waiting for /dev/ttyS0 to be created times out. -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
As I have not hear anything yet, I thought I would summarize the current status for this problem. Please, let me know if any other tests or information are needed. I am running qemu-system-x86_64 v5.2.0 (also tried v6.1.0 and top of master) on: - aarch64 (Jetson AGX Xavier) with Ubuntu 18.04.5 as a host (compiled from git sources as distro version for it was 2.11, which is too old), and on - x86_64 (my laptop) with Fedora 34 as a host (here the qemu-system-x86_64 distro version is 5.2.0). Running Ubuntu 18.04.6 server install cdrom (also tried Ubuntu 20.04.3) as the guest. The following services fail on the Jetson, but not on the laptop. The first one is the ttyS0 console, which seems the most important thing as it is provided directly by the virtual emulation (-serial mon:stdio): [ TIME ] Timed out waiting for device dev-ttyS0.device. [DEPEND] Dependency failed for Serial Getty on ttyS0. ... [FAILED] Failed to start Dispatcher daemon for systemd-networkd. <== network does start fine though. See 'systemctl status networkd-dispatcher.service' for details. ... [FAILED] Failed to start Wait until snapd is fully seeded. <== snapd runs fine though. See 'systemctl status snapd.seeded.service' for details. ... [FAILED] Failed to start Holds Snappy daemon refresh. See 'systemctl status snapd.hold.service' for details. [ OK ] Started Update UTMP about System Runlevel Changes. ... waits forever ... Note that the Jetson has 8 cores running at 2.25GHz, and this tests is run just after boot with no other user applications launched. I wonder if I need something in my build options or if I need to rebuild my kernel with some added kernel configuration options... Hopefully, some experts around here can help me with that if it is a known thing (I google around but other than mentioning that 2.11 is too old, could not find any clear reason about this problem). My build options for the Jetson (I did not do cross-compiling, as I was a bit unsure about pkg-config/glib-2.0 for build and host/target, so I compiled natively on the Jetson host machine, using a separate build folder): ../configure \ --target-list=x86_64-softmmu \ --enable-plugins \ --enable-attr \ --enable-auth-pam \ --enable-cap-ng \ --enable-curl \ --enable-gnutls \ --enable-kvm \ <== not available as an accelerator for Ubuntu host on Jetson --enable-libnfs \ --enable-libudev \ --enable-libusb \ --enable-libxml2 \ --enable-linux-aio \ --enable-nettle \ --enable-seccomp \ --enable-snappy \ --enable-spice \ --enable-usb-redir \ --enable-vde \ --enable-virtfs \ --enable-virtiofsd \ --enable-xkbcommon \ --enable-pie \ --enable-modules \ --enable-membarrier \ --enable-tools \ --enable-vvfat Installed as per the default prefix in /usr/local (the distro already have that in the path before the standard distro folders, so all runs as expected): $ which qemu-system-x86_64 /usr/local/bin/qemu-system-x86_64 Some things like the following could not be used due to current kernel or ubuntu packages available (perhaps I need to compile fuse from sources?): - --enable-libpmem (absent package, couldn't find the right one) - --enable-libssh (0.8.0 but >= 0.8.7 for libssh-4-dev) - --enable-fuse --enable-fuse-lseek (fuse2 available but fuse3 needed) - --enable-netmap (not in current kernel kernel, the required header only exists for newer kernels) I run it with the following command line on both Jetson and laptop: qemu-system-x86_64 \ -boot order=dc,menu=on \ -cdrom ubuntu-18.04.6-live-server-amd64.iso \ -nographic \ -serial mon:stdio \ -kernel ufm/vmlinuz \ -initrd ufm/initrd \ -append 'boot=casper console=ttyS0 ---' \ -m 16384 \ -drive file=ufm/ufm.fd0,format=raw,if=floppy \ <= empty image to avoid ubuntu complaining about fd0. -drive file=ufm/ufm.img,format=raw,if=ide \ -netdev bridge,br=virbr0,id=net0 \ -device virtio-net-pci,netdev=net0,id=nic1 \ -device usb-ehci,id=ehci The vmlinuz & initrd come from the ubuntu iso in the casper folder, the append uses what the grub configuration had for the normal default kernel in the iso. The virtual bridge works as expected with the right allow line in /usr/local/etc/qemu/bridge.conf and setting the qemu-bridge-helper u+s (plus a few extra packages). Anything else is as per the Ubunt u 18.04.5 (LTS) repos used by the host (I did not upgraded the packages, other than the packages needed to get the bridge working and the dev packages to compile the qemu on the aarch64). In the Jetson machine running Ubuntu 18.04.5 I get: $ uname -a Linux vpm-devkit 4.9.201-tegra #1 SMP PREEMPT Fri Jul 2 15:24:18 BST 2021 aarch64 aarch64 aarch64 GNU/Linux $ cat /proc/cpuinfo processor : 0 model name : ARMv8 Processor rev 0 (v8l) BogoMIPS : 62.50 Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp CPU
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 08/11/2021 22:00, David Fernandez wrote: > On 08/11/2021 21:19, David Fernandez wrote: >> On 08/11/2021 20:57, David Fernandez wrote: >>> On 08/11/2021 20:50, Peter Maydell wrote: On Mon, 8 Nov 2021 at 20:22, David Fernandez wrote: > Is there a specific branch I should use? Could not see more than > 2.12 in > git.qemu.org regarding stable branches, but happy to compile and > try any > other. We switched some time ago to using tags rather than branches; you could use the v6.1.0 tag for the most recent release, or master for bleeding-edge. >>> >>> As noted above, I did not realize straight away that the fedora >>> version is 5.2.0... >>> >>> I'll try 5.2.0 first, then 6.1.0 (I tried master but the problem was >>> still there). >>> >>> Then I might do the newer guest version to see what happens. >> >> Right, checked out v5.2.0 tag, recompiled (no >> --enable-spice-protocol, only --enable-spice, and no --enable-lto), >> and the problem is still there >> >> Hopefully doing a git checkout -f v5.2.0 is enough (not sure if I >> need some other option to force some submodule change... I beleieve >> the configure or make do that). The qemu-system-x84_64 -version gives >> 5.2.0, as in the host, but it still does not find the ttyS0 device. >> >> Will try now 6.1.0 then download the Ubunto 20 iso (and extract the >> vmlinuz and initrd) and see if that changes anything. >> > Tried v6.1.0 from a clean qemu repo, just to be sure that make checks > out the right submodules, and still the same, so will try Ububntu > 20... this probably take until tomorrow to get results... let you know. Right, seems that the torrent download for the iso was really quick at this time, so I tested qemu 6.1.0 with Ubuntu 20.04.3 in both my intel desktop and the aarch64 Jetson, and it works fine in my desktop and fails in the same way in the Jetson. Any ideas? -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 08/11/2021 21:19, David Fernandez wrote: > On 08/11/2021 20:57, David Fernandez wrote: >> On 08/11/2021 20:50, Peter Maydell wrote: >>> On Mon, 8 Nov 2021 at 20:22, David Fernandez >>> wrote: Is there a specific branch I should use? Could not see more than 2.12 in git.qemu.org regarding stable branches, but happy to compile and try any other. >>> We switched some time ago to using tags rather than branches; >>> you could use the v6.1.0 tag for the most recent release, or >>> master for bleeding-edge. >> >> As noted above, I did not realize straight away that the fedora >> version is 5.2.0... >> >> I'll try 5.2.0 first, then 6.1.0 (I tried master but the problem was >> still there). >> >> Then I might do the newer guest version to see what happens. > > Right, checked out v5.2.0 tag, recompiled (no --enable-spice-protocol, > only --enable-spice, and no --enable-lto), and the problem is still there > > Hopefully doing a git checkout -f v5.2.0 is enough (not sure if I need > some other option to force some submodule change... I beleieve the > configure or make do that). The qemu-system-x84_64 -version gives > 5.2.0, as in the host, but it still does not find the ttyS0 device. > > Will try now 6.1.0 then download the Ubunto 20 iso (and extract the > vmlinuz and initrd) and see if that changes anything. > Tried v6.1.0 from a clean qemu repo, just to be sure that make checks out the right submodules, and still the same, so will try Ububntu 20... this probably take until tomorrow to get results... let you know. >> >> >>> -- PMM > >
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 08/11/2021 20:57, David Fernandez wrote: > On 08/11/2021 20:50, Peter Maydell wrote: >> On Mon, 8 Nov 2021 at 20:22, David Fernandez >> wrote: Does the x86-64 host still work OK >> if you run it with KVM turned off >> (ie matching the aarch64 host setup) ? >>> Have not tried that... is there an easy way/option to run that test? Or >>> do I need >>> to compile from sources in Fedora? >> As long as your QEMU commandline doesn't have -enable-kvm or any >> other kvm-related option in it it should default to emulation. > > I believe I used to run without it... I'll recheck and confirm. So yes, I run without -enable-kvm in the command line: From my first email: I run it like this: qemu-system-x86_64 \ -boot order=dc,menu=on \ -cdrom ubuntu-18.04.6-live-server-amd64.iso \ -nographic \ -serial mon:stdio \ -kernel ufm/vmlinuz \ -initrd ufm/initrd \ -append 'boot=casper console=ttyS0 ---' \ -m 16384 \ -drive file=ufm/ufm.fd0,format=raw,if=floppy \ <= empty image to avoid ubuntu complaining about fd0. -drive file=ufm/ufm.img,format=raw,if=ide \ -netdev bridge,br=virbr0,id=net0 \ -device virtio-net-pci,netdev=net0,id=nic1 \ -device usb-ehci,id=ehci > Hopefully, some experts around here can help me with that if it is a > known thing (I google around but other than mentioning that 2.11 > is too > old, could not find any clear reason about this problem). For aarch64 host, I would be a bit dubious about running 2.11 or 2.12 -- they are both absolutely ancient in QEMU terms. >>> Is there a specific branch I should use? Could not see more than >>> 2.12 in >>> git.qemu.org regarding stable branches, but happy to compile and try >>> any >>> other. >> We switched some time ago to using tags rather than branches; >> you could use the v6.1.0 tag for the most recent release, or >> master for bleeding-edge. > > As noted above, I did not realize straight away that the fedora > version is 5.2.0... > > I'll try 5.2.0 first, then 6.1.0 (I tried master but the problem was > still there). > > Then I might do the newer guest version to see what happens. Right, checked out v5.2.0 tag, recompiled (no --enable-spice-protocol, only --enable-spice, and no --enable-lto), and the problem is still there Hopefully doing a git checkout -f v5.2.0 is enough (not sure if I need some other option to force some submodule change... I beleieve the configure or make do that). The qemu-system-x84_64 -version gives 5.2.0, as in the host, but it still does not find the ttyS0 device. Will try now 6.1.0 then download the Ubunto 20 iso (and extract the vmlinuz and initrd) and see if that changes anything. > > >> -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 08/11/2021 20:50, Peter Maydell wrote: > On Mon, 8 Nov 2021 at 20:22, David Fernandez wrote: >> Hi Peter, >> >> Answers in line. >> >> On 08/11/2021 19:59, Peter Maydell wrote: >>> On Mon, 8 Nov 2021 at 18:05, David Fernandez >>> wrote: I am running qemu-system-x86_64 on aarch64 running Ubuntu 18.04 as both guest and host. I couldn't get the stock qemu-system-x86_04 to boot correctly, as it was an old version 2.11.1, I decided to recompile from sources to see if that would fix the problem, but the problem still persists, using both top of master and stable-2.12 (currently on that). [ TIME ] Timed out waiting for device dev-ttyS0.device. >>> Is there any more error message ? How long does the guest wait on >>> this step before it times out ? >> The guest waits at the end forever... probably because it tries to use the >> normal console instead and that does not get displayed with my options. >> >> These are all the services that fail: >> >> [ TIME ] Timed out waiting for device dev-ttyS0.device. >> [DEPEND] Dependency failed for Serial Getty on ttyS0. >> ... >> [FAILED] Failed to start Dispatcher daemon for systemd-networkd. <== >> network does start fine though. >> See 'systemctl status networkd-dispatcher.service' for details. >> ... >> [FAILED] Failed to start Wait until snapd is fully seeded. <== snapd >> runs fine though. >> See 'systemctl status snapd.seeded.service' for details. >> ... >> [FAILED] Failed to start Holds Snappy daemon refresh. >> See 'systemctl status snapd.hold.service' for details. >> [ OK ] Started Update UTMP about System Runlevel Changes. >> ... waits forever ... > This does sound like a lot of things might be timing out, not just > the "wait for the serial port" part. OTOH the host CPU is supposed to be > 2.26GHz so it shouldn't really be having that much trouble (assuming > you aren't heavily loading the host with other stuff!). > > There used to be a bug years back where a bug in a guest udev > rule meant that the guest would spawn a lot of processes in a way > that was invisible for running on real hardware but was just enough > extra load to make the slower emulated setup timeout in various > ways including that "Timed out waiting for device" error: > https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1615021 > But I think that should have been fixed by 18.04. You might try > a 20.04 Ubuntu guest just in case, I guess... I'll try that and let you know... > The problem does not happen when using qemu-system-x86_64 on my Fedora desktop as host, so I wonder if I need something in my build options or if I need to rebuild my kernel with some added kernel configuration options... >>> Are you testing with the exact same: >>>* command line >>>* files (guest kernel, initrd, iso, etc) >>>* QEMU version >>> on both the aarch64 and x86-64 host ? >> Yes. Sorry, I missed that the version on fedora is 5.2.0 (re-sent the email but the list is slow). >> >> >>> Does the x86-64 host still work OK if you run it with KVM turned off >>> (ie matching the aarch64 host setup) ? >> Have not tried that... is there an easy way/option to run that test? Or >> do I need >> to compile from sources in Fedora? > As long as your QEMU commandline doesn't have -enable-kvm or any > other kvm-related option in it it should default to emulation. I believe I used to run without it... I'll recheck and confirm. > Hopefully, some experts around here can help me with that if it is a known thing (I google around but other than mentioning that 2.11 is too old, could not find any clear reason about this problem). >>> For aarch64 host, I would be a bit dubious about running 2.11 or 2.12 -- >>> they are both absolutely ancient in QEMU terms. >> Is there a specific branch I should use? Could not see more than 2.12 in >> git.qemu.org regarding stable branches, but happy to compile and try any >> other. > We switched some time ago to using tags rather than branches; > you could use the v6.1.0 tag for the most recent release, or > master for bleeding-edge. As noted above, I did not realize straight away that the fedora version is 5.2.0... I'll try 5.2.0 first, then 6.1.0 (I tried master but the problem was still there). Then I might do the newer guest version to see what happens. > -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On Mon, 8 Nov 2021 at 20:22, David Fernandez wrote: > > Hi Peter, > > Answers in line. > > On 08/11/2021 19:59, Peter Maydell wrote: > > On Mon, 8 Nov 2021 at 18:05, David Fernandez > > wrote: > >> I am running qemu-system-x86_64 on aarch64 running Ubuntu 18.04 as both > >> guest and host. > >> > >> I couldn't get the stock qemu-system-x86_04 to boot correctly, as it was > >> an old version 2.11.1, I decided to recompile from sources to see if > >> that would fix the problem, but the problem still persists, using both > >> top of master and stable-2.12 (currently on that). > >> > >> [ TIME ] Timed out waiting for device dev-ttyS0.device. > > Is there any more error message ? How long does the guest wait on > > this step before it times out ? > The guest waits at the end forever... probably because it tries to use the > normal console instead and that does not get displayed with my options. > > These are all the services that fail: > > [ TIME ] Timed out waiting for device dev-ttyS0.device. > [DEPEND] Dependency failed for Serial Getty on ttyS0. > ... > [FAILED] Failed to start Dispatcher daemon for systemd-networkd. <== > network does start fine though. > See 'systemctl status networkd-dispatcher.service' for details. > ... > [FAILED] Failed to start Wait until snapd is fully seeded. <== snapd > runs fine though. > See 'systemctl status snapd.seeded.service' for details. > ... > [FAILED] Failed to start Holds Snappy daemon refresh. > See 'systemctl status snapd.hold.service' for details. > [ OK ] Started Update UTMP about System Runlevel Changes. > ... waits forever ... This does sound like a lot of things might be timing out, not just the "wait for the serial port" part. OTOH the host CPU is supposed to be 2.26GHz so it shouldn't really be having that much trouble (assuming you aren't heavily loading the host with other stuff!). There used to be a bug years back where a bug in a guest udev rule meant that the guest would spawn a lot of processes in a way that was invisible for running on real hardware but was just enough extra load to make the slower emulated setup timeout in various ways including that "Timed out waiting for device" error: https://bugs.launchpad.net/ubuntu/+source/debian-installer/+bug/1615021 But I think that should have been fixed by 18.04. You might try a 20.04 Ubuntu guest just in case, I guess... > >> The problem does not happen when using qemu-system-x86_64 on my Fedora > >> desktop as host, so I wonder if I need something in my build options or > >> if I need to rebuild my kernel with some added kernel configuration > >> options... > > Are you testing with the exact same: > > * command line > > * files (guest kernel, initrd, iso, etc) > > * QEMU version > > on both the aarch64 and x86-64 host ? > > Yes. > > > > Does the x86-64 host still work OK if you run it with KVM turned off > > (ie matching the aarch64 host setup) ? > > Have not tried that... is there an easy way/option to run that test? Or > do I need > to compile from sources in Fedora? As long as your QEMU commandline doesn't have -enable-kvm or any other kvm-related option in it it should default to emulation. > >> Hopefully, some experts around here can help me with that if it is a > >> known thing (I google around but other than mentioning that 2.11 is too > >> old, could not find any clear reason about this problem). > > For aarch64 host, I would be a bit dubious about running 2.11 or 2.12 -- > > they are both absolutely ancient in QEMU terms. > Is there a specific branch I should use? Could not see more than 2.12 in > git.qemu.org regarding stable branches, but happy to compile and try any > other. We switched some time ago to using tags rather than branches; you could use the v6.1.0 tag for the most recent release, or master for bleeding-edge. -- PMM
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On 08/11/2021 20:21, David Fernandez wrote: > Hi Peter, > > Answers in line. > > On 08/11/2021 19:59, Peter Maydell wrote: >> On Mon, 8 Nov 2021 at 18:05, David Fernandez >> wrote: >>> I am running qemu-system-x86_64 on aarch64 running Ubuntu 18.04 as both >>> guest and host. >>> >>> I couldn't get the stock qemu-system-x86_04 to boot correctly, as it >>> was >>> an old version 2.11.1, I decided to recompile from sources to see if >>> that would fix the problem, but the problem still persists, using both >>> top of master and stable-2.12 (currently on that). >>> >>> [ TIME ] Timed out waiting for device dev-ttyS0.device. >> Is there any more error message ? How long does the guest wait on >> this step before it times out ? > The guest waits at the end forever... probably because it tries to use > the > normal console instead and that does not get displayed with my options. > > These are all the services that fail: > > [ TIME ] Timed out waiting for device dev-ttyS0.device. > [DEPEND] Dependency failed for Serial Getty on ttyS0. > ... > [FAILED] Failed to start Dispatcher daemon for systemd-networkd. <== > network does start fine though. > See 'systemctl status networkd-dispatcher.service' for details. > ... > [FAILED] Failed to start Wait until snapd is fully seeded. <== snapd > runs fine though. > See 'systemctl status snapd.seeded.service' for details. > ... > [FAILED] Failed to start Holds Snappy daemon refresh. > See 'systemctl status snapd.hold.service' for details. > [ OK ] Started Update UTMP about System Runlevel Changes. > ... waits forever ... > > >>> The problem does not happen when using qemu-system-x86_64 on my Fedora >>> desktop as host, so I wonder if I need something in my build options or >>> if I need to rebuild my kernel with some added kernel configuration >>> options... >> Are you testing with the exact same: >> * command line >> * files (guest kernel, initrd, iso, etc) >> * QEMU version >> on both the aarch64 and x86-64 host ? > > Yes. -- Correction -- The Fedora version is: > $ qemu-system-x86_64 -version > QEMU emulator version 5.2.0 (qemu-5.2.0-8.fc34) > Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers > > > >> Does the x86-64 host still work OK if you run it with KVM turned off >> (ie matching the aarch64 host setup) ? > > Have not tried that... is there an easy way/option to run that test? > Or do I need > to compile from sources in Fedora? > > >> >>> Hopefully, some experts around here can help me with that if it is a >>> known thing (I google around but other than mentioning that 2.11 is too >>> old, could not find any clear reason about this problem). >> For aarch64 host, I would be a bit dubious about running 2.11 or 2.12 -- >> they are both absolutely ancient in QEMU terms. > Is there a specific branch I should use? Could not see more than 2.12 in > git.qemu.org regarding stable branches, but happy to compile and try > any other. > >> >> What are the specs of the host CPU (in particular, how fast is it)? >> If it's too underpowered it's possible it just can't run the guest >> fast enough for it to boot up before the guest's systemd tasks >> time out (though it would have to be pretty bad for this to be >> the problem). > The machine is a Jetson AGX Xavier, uses a "Volta" CPU with 8 cores. > In theory should be powerful enough, but you tell me, nVidia does not > offer a > lot of information on their systems anyway. > >>> --enable-kvm \ <== does not seem to ba available as an accelerator >> That is expected -- KVM can only accelerate guests where the >> host and guest are the same CPU architecture, so it can do >> aarch64-on-aarch64 and x86-on-x86, but not x86-on-aarch64. > Good to learn that... here you are the output of virt-host-validate > that I > happened to find about: > > $ sudo virt-host-validate > QEMU: Checking if device /dev/kvm > exists : FAIL (Check that CPU and > firmware supports virtualization and kvm module is loaded) > QEMU: Checking if device /dev/vhost-net > exists : WARN (Load the 'vhost_net' module > to improve performance of virtio networking) > QEMU: Checking if device /dev/net/tun > exists : PASS > QEMU: Checking for cgroup 'memory' controller > support : PASS > QEMU: Checking for cgroup 'memory' controller > mount-point : PASS > QEMU: Checking for cgroup 'cpu' controller > support : PASS > QEMU: Checking for cgroup 'cpu' controller > mount-point : PASS > QEMU: Checking for cgroup 'cpuacct' controller > support : PASS > QEMU: Checking for cgroup 'cpuacct' controller > mount-point : PASS > QEMU: Checking for cgroup 'cpuset' controller > support : PASS > QEMU: Checking for cgroup 'cpuset' controller > mount-point : PASS > QEMU:
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
Hi Peter, Answers in line. On 08/11/2021 19:59, Peter Maydell wrote: > On Mon, 8 Nov 2021 at 18:05, David Fernandez wrote: >> I am running qemu-system-x86_64 on aarch64 running Ubuntu 18.04 as both >> guest and host. >> >> I couldn't get the stock qemu-system-x86_04 to boot correctly, as it was >> an old version 2.11.1, I decided to recompile from sources to see if >> that would fix the problem, but the problem still persists, using both >> top of master and stable-2.12 (currently on that). >> >> [ TIME ] Timed out waiting for device dev-ttyS0.device. > Is there any more error message ? How long does the guest wait on > this step before it times out ? The guest waits at the end forever... probably because it tries to use the normal console instead and that does not get displayed with my options. These are all the services that fail: [ TIME ] Timed out waiting for device dev-ttyS0.device. [DEPEND] Dependency failed for Serial Getty on ttyS0. ... [FAILED] Failed to start Dispatcher daemon for systemd-networkd. <== network does start fine though. See 'systemctl status networkd-dispatcher.service' for details. ... [FAILED] Failed to start Wait until snapd is fully seeded. <== snapd runs fine though. See 'systemctl status snapd.seeded.service' for details. ... [FAILED] Failed to start Holds Snappy daemon refresh. See 'systemctl status snapd.hold.service' for details. [ OK ] Started Update UTMP about System Runlevel Changes. ... waits forever ... >> The problem does not happen when using qemu-system-x86_64 on my Fedora >> desktop as host, so I wonder if I need something in my build options or >> if I need to rebuild my kernel with some added kernel configuration >> options... > Are you testing with the exact same: > * command line > * files (guest kernel, initrd, iso, etc) > * QEMU version > on both the aarch64 and x86-64 host ? Yes. > Does the x86-64 host still work OK if you run it with KVM turned off > (ie matching the aarch64 host setup) ? Have not tried that... is there an easy way/option to run that test? Or do I need to compile from sources in Fedora? > >> Hopefully, some experts around here can help me with that if it is a >> known thing (I google around but other than mentioning that 2.11 is too >> old, could not find any clear reason about this problem). > For aarch64 host, I would be a bit dubious about running 2.11 or 2.12 -- > they are both absolutely ancient in QEMU terms. Is there a specific branch I should use? Could not see more than 2.12 in git.qemu.org regarding stable branches, but happy to compile and try any other. > > What are the specs of the host CPU (in particular, how fast is it)? > If it's too underpowered it's possible it just can't run the guest > fast enough for it to boot up before the guest's systemd tasks > time out (though it would have to be pretty bad for this to be > the problem). The machine is a Jetson AGX Xavier, uses a "Volta" CPU with 8 cores. In theory should be powerful enough, but you tell me, nVidia does not offer a lot of information on their systems anyway. >> --enable-kvm \ <== does not seem to ba available as an accelerator > That is expected -- KVM can only accelerate guests where the > host and guest are the same CPU architecture, so it can do > aarch64-on-aarch64 and x86-on-x86, but not x86-on-aarch64. Good to learn that... here you are the output of virt-host-validate that I happened to find about: $ sudo virt-host-validate QEMU: Checking if device /dev/kvm exists : FAIL (Check that CPU and firmware supports virtualization and kvm module is loaded) QEMU: Checking if device /dev/vhost-net exists : WARN (Load the 'vhost_net' module to improve performance of virtio networking) QEMU: Checking if device /dev/net/tun exists : PASS QEMU: Checking for cgroup 'memory' controller support : PASS QEMU: Checking for cgroup 'memory' controller mount-point : PASS QEMU: Checking for cgroup 'cpu' controller support : PASS QEMU: Checking for cgroup 'cpu' controller mount-point : PASS QEMU: Checking for cgroup 'cpuacct' controller support : PASS QEMU: Checking for cgroup 'cpuacct' controller mount-point : PASS QEMU: Checking for cgroup 'cpuset' controller support : PASS QEMU: Checking for cgroup 'cpuset' controller mount-point : PASS QEMU: Checking for cgroup 'devices' controller support : PASS QEMU: Checking for cgroup 'devices' controller mount-point : PASS QEMU: Checking for cgroup 'blkio' controller support : PASS QEMU: Checking for cgroup 'blkio' controller mount-point : PASS WARN (Unknown if this platform has IOMMU support) <= it does have IOMMU, I know for
Re: Guest Ubuntu 18.04 fails to boot with -serial mon:stdio, cannot find ttyS0.
On Mon, 8 Nov 2021 at 18:05, David Fernandez wrote: > I am running qemu-system-x86_64 on aarch64 running Ubuntu 18.04 as both > guest and host. > > I couldn't get the stock qemu-system-x86_04 to boot correctly, as it was > an old version 2.11.1, I decided to recompile from sources to see if > that would fix the problem, but the problem still persists, using both > top of master and stable-2.12 (currently on that).[ TIME ] Timed out > waiting for device dev-ttyS0.device. Is there any more error message ? How long does the guest wait on this step before it times out ? > The problem does not happen when using qemu-system-x86_64 on my Fedora > desktop as host, so I wonder if I need something in my build options or > if I need to rebuild my kernel with some added kernel configuration > options... Are you testing with the exact same: * command line * files (guest kernel, initrd, iso, etc) * QEMU version on both the aarch64 and x86-64 host ? Does the x86-64 host still work OK if you run it with KVM turned off (ie matching the aarch64 host setup) ? > Hopefully, some experts around here can help me with that if it is a > known thing (I google around but other than mentioning that 2.11 is too > old, could not find any clear reason about this problem). For aarch64 host, I would be a bit dubious about running 2.11 or 2.12 -- they are both absolutely ancient in QEMU terms. What are the specs of the host CPU (in particular, how fast is it)? If it's too underpowered it's possible it just can't run the guest fast enough for it to boot up before the guest's systemd tasks time out (though it would have to be pretty bad for this to be the problem). >--enable-kvm \ <== does not seem to ba available as an accelerator That is expected -- KVM can only accelerate guests where the host and guest are the same CPU architecture, so it can do aarch64-on-aarch64 and x86-on-x86, but not x86-on-aarch64. -- PMM
