Re: [fedora-arm] Debugging our kernels under qemu + gdb
On Mon, May 14, 2012 at 12:57:09AM -0400, Jon Masters wrote: On 05/13/2012 04:34 PM, Michael Hope wrote: On 12 May 2012 22:12, Richard W.M. Jones rjo...@redhat.com wrote: On Fri, May 11, 2012 at 01:41:43PM -0700, Brendan Conoboy wrote: On 05/11/2012 01:04 PM, Richard W.M. Jones wrote: Has anyone tried to debug our Fedora/arm kernels under qemu-system-arm? (In this case, the host is also arm, but I don't think that matters.) Richard, FYI, we as of a few hours ago have nearly-official F17-beta images for versatile express on the following page: http://scotland.proximity.on.ca/arm-nightlies/ There's a link for vexpress and vexpress+x rootfs images. A second link provides a kernel, initramfs, and script for starting qemu. Note that vexpress is much faster than versatile and allows more ram (1GB). Recommend you try this out! So one issue appears to be lack of PCI support (according to Linaro's notes: https://wiki.linaro.org/PeterMaydell/QemuVersatileExpress). Unfortunately all of the virtio hardware is PCI-based, so it doesn't seem like this is going to work for the virt tools :-( Hi Richard. The plan is to use virtio-mmio and use Device Tree to set where the virtio devices are. virtio-mmio is in the mainline kernel and in the queue for QEMU. Note, we're not using dtb (device tree) yet in the qemu kernel. To do that properly, we'll need to get a qemu that works with U-Boot, etc. I know Linaro have put such a combination together, right? Should any of that be working with upstream bits yet? Brendan mentioned he'd tried poking briefly at the U-Boot that Linaro put together but it apparently didn't boot on our qemu (I know upstream was missing e.g. the model instantiation for the hardware memory controller, etc. in vexpress). There's something quite broken about our qemu package. I haven't looked at what it is yet, but at the moment I'm using qemu built from upstream git for all testing. I'll try to look at what's going on with the qemu package later. I guess I could/should ping Peter Maydell? Is he the best contact? Jon. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Debugging our kernels under qemu + gdb
On 14 May 2012 16:57, Jon Masters j...@redhat.com wrote: On 05/13/2012 04:34 PM, Michael Hope wrote: On 12 May 2012 22:12, Richard W.M. Jones rjo...@redhat.com wrote: On Fri, May 11, 2012 at 01:41:43PM -0700, Brendan Conoboy wrote: On 05/11/2012 01:04 PM, Richard W.M. Jones wrote: Has anyone tried to debug our Fedora/arm kernels under qemu-system-arm? (In this case, the host is also arm, but I don't think that matters.) Richard, FYI, we as of a few hours ago have nearly-official F17-beta images for versatile express on the following page: http://scotland.proximity.on.ca/arm-nightlies/ There's a link for vexpress and vexpress+x rootfs images. A second link provides a kernel, initramfs, and script for starting qemu. Note that vexpress is much faster than versatile and allows more ram (1GB). Recommend you try this out! So one issue appears to be lack of PCI support (according to Linaro's notes: https://wiki.linaro.org/PeterMaydell/QemuVersatileExpress). Unfortunately all of the virtio hardware is PCI-based, so it doesn't seem like this is going to work for the virt tools :-( Hi Richard. The plan is to use virtio-mmio and use Device Tree to set where the virtio devices are. virtio-mmio is in the mainline kernel and in the queue for QEMU. Note, we're not using dtb (device tree) yet in the qemu kernel. To do that properly, we'll need to get a qemu that works with U-Boot, etc. I know Linaro have put such a combination together, right? Should any of that be working with upstream bits yet? Brendan mentioned he'd tried poking briefly at the U-Boot that Linaro put together but it apparently didn't boot on our qemu (I know upstream was missing e.g. the model instantiation for the hardware memory controller, etc. in vexpress). I guess I could/should ping Peter Maydell? Is he the best contact? Peter or Ricardo, yip. Ricardo has a good handle on the integration work that's been done and the upstream versions. We've standardised on the vexpress-a9 model as it's neutral, has a good amount of RAM, and has good support in upstream QEMU and mainline Linux. The vexpress-a15 isn't as tested but goes up to 2 GB and will probably be the base model for KVM. -- Michael ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Debugging our kernels under qemu + gdb
On 12 May 2012 22:12, Richard W.M. Jones rjo...@redhat.com wrote: On Fri, May 11, 2012 at 01:41:43PM -0700, Brendan Conoboy wrote: On 05/11/2012 01:04 PM, Richard W.M. Jones wrote: Has anyone tried to debug our Fedora/arm kernels under qemu-system-arm? (In this case, the host is also arm, but I don't think that matters.) Richard, FYI, we as of a few hours ago have nearly-official F17-beta images for versatile express on the following page: http://scotland.proximity.on.ca/arm-nightlies/ There's a link for vexpress and vexpress+x rootfs images. A second link provides a kernel, initramfs, and script for starting qemu. Note that vexpress is much faster than versatile and allows more ram (1GB). Recommend you try this out! So one issue appears to be lack of PCI support (according to Linaro's notes: https://wiki.linaro.org/PeterMaydell/QemuVersatileExpress). Unfortunately all of the virtio hardware is PCI-based, so it doesn't seem like this is going to work for the virt tools :-( Hi Richard. The plan is to use virtio-mmio and use Device Tree to set where the virtio devices are. virtio-mmio is in the mainline kernel and in the queue for QEMU. -- Michael ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Debugging our kernels under qemu + gdb
On 05/13/2012 04:34 PM, Michael Hope wrote: On 12 May 2012 22:12, Richard W.M. Jones rjo...@redhat.com wrote: On Fri, May 11, 2012 at 01:41:43PM -0700, Brendan Conoboy wrote: On 05/11/2012 01:04 PM, Richard W.M. Jones wrote: Has anyone tried to debug our Fedora/arm kernels under qemu-system-arm? (In this case, the host is also arm, but I don't think that matters.) Richard, FYI, we as of a few hours ago have nearly-official F17-beta images for versatile express on the following page: http://scotland.proximity.on.ca/arm-nightlies/ There's a link for vexpress and vexpress+x rootfs images. A second link provides a kernel, initramfs, and script for starting qemu. Note that vexpress is much faster than versatile and allows more ram (1GB). Recommend you try this out! So one issue appears to be lack of PCI support (according to Linaro's notes: https://wiki.linaro.org/PeterMaydell/QemuVersatileExpress). Unfortunately all of the virtio hardware is PCI-based, so it doesn't seem like this is going to work for the virt tools :-( Hi Richard. The plan is to use virtio-mmio and use Device Tree to set where the virtio devices are. virtio-mmio is in the mainline kernel and in the queue for QEMU. Note, we're not using dtb (device tree) yet in the qemu kernel. To do that properly, we'll need to get a qemu that works with U-Boot, etc. I know Linaro have put such a combination together, right? Should any of that be working with upstream bits yet? Brendan mentioned he'd tried poking briefly at the U-Boot that Linaro put together but it apparently didn't boot on our qemu (I know upstream was missing e.g. the model instantiation for the hardware memory controller, etc. in vexpress). I guess I could/should ping Peter Maydell? Is he the best contact? Jon. ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Debugging our kernels under qemu + gdb
On Fri, May 11, 2012 at 01:41:43PM -0700, Brendan Conoboy wrote:
On 05/11/2012 01:04 PM, Richard W.M. Jones wrote:
Has anyone tried to debug our Fedora/arm kernels under qemu-system-arm?
(In this case, the host is also arm, but I don't think that matters.)
Richard,
FYI, we as of a few hours ago have nearly-official F17-beta images
for versatile express on the following page:
http://scotland.proximity.on.ca/arm-nightlies/
There's a link for vexpress and vexpress+x rootfs images. A second
link provides a kernel, initramfs, and script for starting qemu.
Note that vexpress is much faster than versatile and allows more ram
(1GB). Recommend you try this out!
So one issue appears to be lack of PCI support (according to Linaro's
notes: https://wiki.linaro.org/PeterMaydell/QemuVersatileExpress).
Unfortunately all of the virtio hardware is PCI-based, so it doesn't
seem like this is going to work for the virt tools :-(
Having said that, virtio with the current Fedora ARM kernels doesn't
work either (see below). This appears to be a problem with qemu not
setting up the PCI config space sensibly. I'm still investigating.
Rich.
[...]
[2.360640] PCI core found (slot 11)
[2.367644] PCI host bridge to bus :00
[2.371144] pci_bus :00: root bus resource [io 0x4400-0x4fff]
[2.374066] pci_bus :00: root bus resource [mem 0x5000-0x5fff]
[2.374751] pci_bus :00: root bus resource [mem 0x6000-0x6fff
pref]
[2.393315] PCI: bus0: Fast back to back transfers disabled
[2.405189] PCI map irq: slot 0, pin 1, devslot 12, irq: 27
[2.406686] PCI map irq: slot 0, pin 1, devslot 13, irq: 27
[2.407285] PCI map irq: slot 0, pin 1, devslot 14, irq: 27
[2.413989] pci :00:0d.0: BAR 0: assigned [io 0x4400-0x443f]
[2.416204] pci :00:0e.0: BAR 0: assigned [io 0x4440-0x447f]
[2.417221] pci :00:0c.0: BAR 0: assigned [io 0x4480-0x449f]
[2.516011] bio: create slab bio-0 at 0
[2.538118] vgaarb: loaded
[2.551547] SCSI subsystem initialized
[2.566105] usbcore: registered new interface driver usbfs
[2.568507] usbcore: registered new interface driver hub
[2.573128] usbcore: registered new device driver usb
[2.623191] NetLabel: Initializing
[2.623702] NetLabel: domain hash size = 128
[2.624088] NetLabel: protocols = UNLABELED CIPSOv4
[2.631777] NetLabel: unlabeled traffic allowed by default
[2.646556] Switching to clocksource timer3
[3.165976] NET: Registered protocol family 2
[3.176617] IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
[3.201008] TCP established hash table entries: 8192 (order: 4, 65536 bytes)
[3.203937] TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
[3.206121] TCP: Hash tables configured (established 8192 bind 8192)
[3.207154] TCP reno registered
[3.208383] UDP hash table entries: 256 (order: 0, 4096 bytes)
[3.212231] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
[3.224129] NET: Registered protocol family 1
[3.252885] Unpacking initramfs...
[3.363403] Freeing initrd memory: 764K
[3.389097] audit: initializing netlink socket (disabled)
[3.393720] type=2000 audit(1.860:1): initialized
[4.616872] VFS: Disk quotas dquot_6.5.2
[4.623368] Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[4.671602] msgmni has been set to 492
[4.782151] alg: No test for stdrng (krng)
[4.783383] NET: Registered protocol family 38
[4.792118] Block layer SCSI generic (bsg) driver version 0.4 loaded (major
253)
[4.796706] io scheduler noop registered
[4.797383] io scheduler deadline registered
[4.800773] io scheduler cfq registered (default)
[4.824787] PCI: enabling device :00:0c.0 (0100 - 0101)
[4.829857] Unable to handle kernel paging request at virtual address
4492
[4.830565] pgd = c0004000
[4.830912] [4492] *pgd=
[4.833083] Internal error: Oops: 805 [#1]
[4.834016] Modules linked in:
[4.835742] CPU: 0Not tainted (3.3.4-4.fc17.armv7hl #1)
[4.838653] PC is at vp_reset+0x14/0x64
The code is:
static void vp_reset(struct virtio_device *vdev)
{
struct virtio_pci_device *vp_dev = to_vp_device(vdev);
/* 0 status means a reset. */
iowrite8(0, vp_dev-ioaddr + VIRTIO_PCI_STATUS); --- here
// 0x4480 + 0x12 == 0x4492
[4.840141] LR is at register_virtio_device+0x4c/0x94
[4.841896] pc : [c0230a08]lr : [c022fa48]psr: 8013
[4.841996] sp : cf82feb0 ip : fp :
[4.845557] r10: c0649084 r9 : r8 :
[4.847277] r7 : cf889860 r6 : r5 : cfa50608 r4 : cfa50600
[4.849447] r3 : 4480 r2 : r1 : 002f r0 : cfa50600
[4.851825] Flags: Nzcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment
kernel
[4.854292] Control: 00093177 Table: 4000 DAC:
[fedora-arm] Debugging our kernels under qemu + gdb
Has anyone tried to debug our Fedora/arm kernels under qemu-system-arm?
(In this case, the host is also arm, but I don't think that matters.)
After a lot of effort, I've managed to get to the point where prints
this on the serial port:
Uncompressing Linux... done, booting the kernel.
and then hangs. Under gdb the hang is in this code:
= 0x005fe934: nop ; (mov r0, r0)
0x005fe938: b 0x5fe934
It seems it's meant to be an infinite loop (ie. panic) because
something previously has failed.
However I can't get gdb to make sense of the symbols in the
kernel-debuginfo package, so I've really no idea where to start
looking for this ... The symbols refer to addresses 0xcxxx, but
there's no code at those addresses, just zeroes. Unless it's so early
in the boot that pagetables need to be setup or code needs to be
copied around -- anyone know how all this works on arm?
Rich.
kernel=3.3.4-4.fc17.armv7hl
# Homebrew qemu because qemu from Fedora package doesn't work at all.
QEMUDIR=$HOME/d/qemu
$QEMUDIR/arm-softmmu/qemu-system-arm \
-s \
-M versatilepb \
-cpu cortex-a9 \
-nodefaults \
-nographic \
-serial stdio \
-m 256 \
-kernel /boot/vmlinuz-$kernel \
-initrd /boot/initramfs-$kernel.img \
-append 'console=ttyAMA0'
# Invocation of gdb.
$ gdb
GNU gdb (GDB) Fedora (7.4.50.20120120-42.fc17)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type show copying
and show warranty for details.
This GDB was configured as armv7hl-redhat-linux-gnueabi.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
(gdb) file /usr/lib/debug/lib/modules/3.3.4-4.fc17.armv7hl/vmlinux
Reading symbols from
/usr/lib/debug/lib/modules/3.3.4-4.fc17.armv7hl/vmlinux...done.
(gdb) target remote :1234
Remote debugging using :1234
0x0066904c in ?? ()
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x005fe934 in ?? ()
(gdb) bt
#0 0x005fe934 in ?? ()
#1 0x800c in ?? ()
#2 0x800c in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) disassemble 0x005fe934
No function contains specified address.
(gdb) disassemble 0x005fe934,+0x100
Dump of assembler code from 0x5fe934 to 0x5fea34:
= 0x005fe934:nop; (mov r0, r0)
0x005fe938:b 0x5fe934
0x005fe93c:movr0, #0
0x005fe940:bx lr
0x005fe944:push {r0, r1, r2, r4, r5, r6, r7, r8, r9, r10,
r11, lr}
0x005fe948:ldrr3, [pc, #22576152] ; 0x5feb04
0x005fe94c:ldrr4, [r3]
0x005fe950:cmpr4, #0
0x005fe954:ldrne r0, [pc, #23115160]; 0x5feb08
0x005fe958:bne0x5fe97c
0x005fe95c:ldrr3, [pc, #22576152]; 0x5feb0c
0x005fe960:ldrr4, [r3]
0x005fe964:ldrb r2, [r3, #3118736]
0x005fe968:cmpr4, #0
0x005fe96c:beq0x5fe984
0x005fe970:cmpr2, #0
0x005fe974:bne0x5fea9c
0x005fe978:ldrr0, [pc, #23115160]; 0x5feb10
0x005fe97c:bl 0x429448
0x005fe980:b 0x5fea9c
0x005fe984:cmpr2, #0
0x005fe988:bne0x5fe99c
---Type return to continue, or q return to quit---q
Quit
--
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines. Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v
___
arm mailing list
arm@lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Debugging our kernels under qemu + gdb
On Fri, May 11, 2012 at 09:04:40PM +0100, Richard W.M. Jones wrote: = 0x005fe934: nop ; (mov r0, r0) 0x005fe938: b 0x5fe934 Well I guess there's more than one way to skin this cat. I disassembled the whole of vmlinux and found the code: c05fe934 __error: c05fe934: e1a0nop ; (mov r0, r0) c05fe938: eafdb c05fe934 __error Of the two call sites, it seems most likely to be a failure in __lookup_processor_type. Unfortunately the exact error message is not shown, but at least I can adjust the addresses and breakpoint it now. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Debugging our kernels under qemu + gdb
... and setting the right emulated -cpu, it boots! For reference (mainly mine), here is the qemu command line that works: kernel=3.3.4-4.fc17.armv7hl QEMUDIR=$HOME/d/qemu exec \ $QEMUDIR/arm-softmmu/qemu-system-arm \ -s \ -M versatilepb \ -cpu arm926 \ -nodefaults \ -nographic \ -serial stdio \ -m 256 \ -kernel /boot/vmlinuz-$kernel \ -initrd /boot/initramfs-$kernel.img \ -append 'console=ttyAMA0' Note that I built this qemu from today's upstream qemu.git, because the one we currently have compiled in Fedora fails early with this error: /builddir/build/BUILD/qemu-kvm-1.0/tcg/arm/tcg-target.c:891: tcg fatal error Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Debugging our kernels under qemu + gdb
On Fri, May 11, 2012 at 01:41:43PM -0700, Brendan Conoboy wrote: On 05/11/2012 01:04 PM, Richard W.M. Jones wrote: Has anyone tried to debug our Fedora/arm kernels under qemu-system-arm? (In this case, the host is also arm, but I don't think that matters.) Richard, FYI, we as of a few hours ago have nearly-official F17-beta images for versatile express on the following page: http://scotland.proximity.on.ca/arm-nightlies/ There's a link for vexpress and vexpress+x rootfs images. A second link provides a kernel, initramfs, and script for starting qemu. Note that vexpress is much faster than versatile and allows more ram (1GB). Recommend you try this out! Thanks, I will. Are these going to replace the current Fedora kernel config at some point? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming blog: http://rwmj.wordpress.com Fedora now supports 80 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
Re: [fedora-arm] Debugging our kernels under qemu + gdb
On 05/11/2012 01:50 PM, Richard W.M. Jones wrote: Thanks, I will. Are these going to replace the current Fedora kernel config at some point? Yes, in a few days I would expect the default Fedora ARM kernel to be vexpress oriented. We're still working on integrating the necessary vexpress configuration options. -- Brendan Conoboy / Red Hat, Inc. / b...@redhat.com ___ arm mailing list arm@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/arm
