Re: [Qemu-devel] RFH: difference in read-only mapped bios.bin - memory corruption?
* Philipp Hahn (h...@univention.de) wrote: > Hello, > > Am 15.08.2017 um 13:25 schrieb Laszlo Ersek: > > On 08/14/17 20:39, Dr. David Alan Gilbert wrote: > >> * Philipp Hahn (h...@univention.de) wrote: > >>> I'm currently investigating a problem, were a Linux VM does not reboot > >>> and gets stuck in the SeaBIOS reboot code: > >>> > >>> I'm using SeaBIOS-1.7 from Debian with a more modern qemu-2.8 > ...>>> If I dump both regions and compare them, I get a difference: > ...>> You might want seabios commit c68aff5 and b837e6 that got fixed after > >> I tracked down some reboot hangs - although they were rare, not every > >> time. c68aff5 did certainly cause a corruption, and the address of that > >> corruption was determined at link time and could overlay random useful > >> bits of code if you were unlucky. > > Thanks you for the commit IDs - to me this looks like they fixed the > problem. Testing with seabios-1.10 does not show any reboot problem so far. > > >>> 1. How can it be, that the low-mem ROM mapping is modified? > >> > >> I can't remember all the details, but PC ROM is shadowed and mapped over > >> with RAM at various times, > > > > Right. I don't remember for sure, but I believe the state of the PAM > > registers doesn't only affect what the VCPUs see in that address range, > > but also what your monitor commands will dump. (This would be the > > logical choice -- make the monitor output what the VCPUs see anyway, at > > the moment, dependent on the PAM settings.) > > That makes sense. > Do you know by change what change in Qemu triggered that bug, as I've > never seen any reboot problem with qemu-1.1.2, but only since switching > to qemu-2.8? I didn't go back as far as 1.1.2, but I tried bisecting around 2.4/2.6 before I understood the failure and the bisect was very flaky; I think in the end it's a timing race where it comes down to the exact corrupt value; going back to ancient qemu might be taking some other path through seabios but I ddin't investigate. Dave > Thanks again for your excellent help. > > Philipp -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] RFH: difference in read-only mapped bios.bin - memory corruption?
Hello, Am 15.08.2017 um 13:25 schrieb Laszlo Ersek: > On 08/14/17 20:39, Dr. David Alan Gilbert wrote: >> * Philipp Hahn (h...@univention.de) wrote: >>> I'm currently investigating a problem, were a Linux VM does not reboot >>> and gets stuck in the SeaBIOS reboot code: >>> >>> I'm using SeaBIOS-1.7 from Debian with a more modern qemu-2.8 ...>>> If I dump both regions and compare them, I get a difference: ...>> You might want seabios commit c68aff5 and b837e6 that got fixed after >> I tracked down some reboot hangs - although they were rare, not every >> time. c68aff5 did certainly cause a corruption, and the address of that >> corruption was determined at link time and could overlay random useful >> bits of code if you were unlucky. Thanks you for the commit IDs - to me this looks like they fixed the problem. Testing with seabios-1.10 does not show any reboot problem so far. >>> 1. How can it be, that the low-mem ROM mapping is modified? >> >> I can't remember all the details, but PC ROM is shadowed and mapped over >> with RAM at various times, > > Right. I don't remember for sure, but I believe the state of the PAM > registers doesn't only affect what the VCPUs see in that address range, > but also what your monitor commands will dump. (This would be the > logical choice -- make the monitor output what the VCPUs see anyway, at > the moment, dependent on the PAM settings.) That makes sense. Do you know by change what change in Qemu triggered that bug, as I've never seen any reboot problem with qemu-1.1.2, but only since switching to qemu-2.8? Thanks again for your excellent help. Philipp
Re: [Qemu-devel] RFH: difference in read-only mapped bios.bin - memory corruption?
On 08/14/17 20:39, Dr. David Alan Gilbert wrote:
> * Philipp Hahn (h...@univention.de) wrote:
>> Hello,
>>
>> I'm currently investigating a problem, were a Linux VM does not reboot
>> and gets stuck in the SeaBIOS reboot code:
>>
>> I'm using SeaBIOS-1.7 from Debian with a more modern qemu-2.8
>>
>>> virsh # qemu-monitor-command --hmp ucs41-414 info roms
>>> fw=genroms/kvmvapic.bin size=0x002400 name="kvmvapic.bin"
>>> addr=fffe size=0x02 mem=rom name="bios.bin"
>>
>> which (to my understanding) is mapped at two physical locations:
>>> virsh # qemu-monitor-command --hmp ucs41-414 info mtree
>> ...> memory-region: system
>> ...> 000e-000f (prio 1, R-): alias
>> isa-bios @pc.bios -0001
>>> fffe- (prio 0, R-): pc.bios
>>
>> If I dump both regions and compare them, I get a difference:
>>> virsh # qemu-monitor-command --pretty --domain ucs41-414
>>> '{"execute":"pmemsave","arguments":{"val":917504,"size":131072,"filename":"/tmp/bios-low.dump"}}'
>>> virsh # qemu-monitor-command --pretty --domain ucs41-414
>>> '{"execute":"pmemsave","arguments":{"val":4294836224,"size":131072,"filename":"/tmp/bios-high.dump"}}'
>>> # diff --suppress-common-lines -y <(od -Ax -tx1 -w1 -v /tmp/bios-low.dump)
>>> <(od -Ax -tx1 -w1 -v /tmp/bios-high.dump)
>>> 00f798 fa | 00f798 80
>>> 00f799 7a | 00f799 89
>>> 00f79a f4 | 00f79a f2
>>> 016d40 00 | 016d40 ff
>>> 016d41 00 | 016d41 ff
>>> 016d42 00 | 016d42 ff
>>> 016d43 00 | 016d43 ff
>>
>> The high address dump is the same as the original:
>
> You might want seabios commit c68aff5 and b837e6 that got fixed after
> I tracked down some reboot hangs - although they were rare, not every
> time. c68aff5 did certainly cause a corruption, and the address of that
> corruption was determined at link time and could overlay random useful
> bits of code if you were unlucky.
>
>>> # cmp -l /tmp/bios-high.dump /usr/share/seabios/bios.bin
>>
>>> virsh # qemu-monitor-command --hmp ucs41-414 x/6i 0x000ef78f
>>> 0x000ef78f: mov$0xcf8,%esi
>>> 0x000ef794: mov$0xfa00,%eax
>>> 0x000ef799: jp 0xef78f
>>^^ BUG: endless loop
>>> 0x000ef79b: out%eax,(%dx)
>>> 0x000ef79c: mov$0xfe,%dl
>>> 0x000ef79e: in (%dx),%ax
>>
>>> virsh # qemu-monitor-command --hmp ucs41-414 xp/6i 0xfffef78f
>>> 0xfffef78f: mov$0xcf8,%esi
>>> 0xfffef794: mov$0x8000,%eax
>>> 0xfffef799: mov%esi,%edx
>> CORRECT original code
>>> 0xfffef79b: out%eax,(%dx)
>>> 0xfffef79c: mov$0xfe,%dl
>>> 0xfffef79e: in (%dx),%ax
>>
>> (That's some code from seabios-1.7.0/src/pci.c)
>>
>> I had exactly the same run some weeks ago, but I also get different
>> patterns:
>>> # diff --suppress-common-lines -y <(od -Ax -tx1 -w1 -v /tmp/bios2.dump)
>>> <(od -Ax -tx1 -w1 -v bios.bin)
>>> 00f798 f0 | 00f798 80
>>> 00f799 8d | 00f799 89
>>> 00f79a f3 | 00f79a f2
>>> 016d40 00 | 016d40 ff
>>> 016d41 00 | 016d41 ff
>>> 016d42 00 | 016d42 ff
>>> 016d43 00 | 016d43 ff
>>
>> Not all runs lead to reboot problems, but I don't know if any other
>> corruption happened there.
>>
>> I had a similar problem with OVMF back in June
>> ,
>> which I "solved" by upgrading the OVMF version: I have not seen the
>> problem there since than, but this problems looks very similar.
>>
>>
>> 1. How can it be, that the low-mem ROM mapping is modified?
>
> I can't remember all the details, but PC ROM is shadowed and mapped over
> with RAM at various times,
Right. I don't remember for sure, but I believe the state of the PAM
registers doesn't only affect what the VCPUs see in that address range,
but also what your monitor commands will dump. (This would be the
logical choice -- make the monitor output what the VCPUs see anyway, at
the moment, dependent on the PAM settings.)
Thanks
Laszlo
> and the bioses play lots of silly tricks of
> being copied and then reusing bits of the copied space as temporaries
> and. oh it's just a mess.
>
>
Re: [Qemu-devel] RFH: difference in read-only mapped bios.bin - memory corruption?
* Philipp Hahn (h...@univention.de) wrote:
> Hello,
>
> I'm currently investigating a problem, were a Linux VM does not reboot
> and gets stuck in the SeaBIOS reboot code:
>
> I'm using SeaBIOS-1.7 from Debian with a more modern qemu-2.8
>
> > virsh # qemu-monitor-command --hmp ucs41-414 info roms
> > fw=genroms/kvmvapic.bin size=0x002400 name="kvmvapic.bin"
> > addr=fffe size=0x02 mem=rom name="bios.bin"
>
> which (to my understanding) is mapped at two physical locations:
> > virsh # qemu-monitor-command --hmp ucs41-414 info mtree
> ...> memory-region: system
> ...> 000e-000f (prio 1, R-): alias
> isa-bios @pc.bios -0001
> > fffe- (prio 0, R-): pc.bios
>
> If I dump both regions and compare them, I get a difference:
> > virsh # qemu-monitor-command --pretty --domain ucs41-414
> > '{"execute":"pmemsave","arguments":{"val":917504,"size":131072,"filename":"/tmp/bios-low.dump"}}'
> > virsh # qemu-monitor-command --pretty --domain ucs41-414
> > '{"execute":"pmemsave","arguments":{"val":4294836224,"size":131072,"filename":"/tmp/bios-high.dump"}}'
> > # diff --suppress-common-lines -y <(od -Ax -tx1 -w1 -v /tmp/bios-low.dump)
> > <(od -Ax -tx1 -w1 -v /tmp/bios-high.dump)
> > 00f798 fa | 00f798 80
> > 00f799 7a | 00f799 89
> > 00f79a f4 | 00f79a f2
> > 016d40 00 | 016d40 ff
> > 016d41 00 | 016d41 ff
> > 016d42 00 | 016d42 ff
> > 016d43 00 | 016d43 ff
>
> The high address dump is the same as the original:
You might want seabios commit c68aff5 and b837e6 that got fixed after
I tracked down some reboot hangs - although they were rare, not every
time. c68aff5 did certainly cause a corruption, and the address of that
corruption was determined at link time and could overlay random useful
bits of code if you were unlucky.
> > # cmp -l /tmp/bios-high.dump /usr/share/seabios/bios.bin
>
> > virsh # qemu-monitor-command --hmp ucs41-414 x/6i 0x000ef78f
> > 0x000ef78f: mov$0xcf8,%esi
> > 0x000ef794: mov$0xfa00,%eax
> > 0x000ef799: jp 0xef78f
>^^ BUG: endless loop
> > 0x000ef79b: out%eax,(%dx)
> > 0x000ef79c: mov$0xfe,%dl
> > 0x000ef79e: in (%dx),%ax
>
> > virsh # qemu-monitor-command --hmp ucs41-414 xp/6i 0xfffef78f
> > 0xfffef78f: mov$0xcf8,%esi
> > 0xfffef794: mov$0x8000,%eax
> > 0xfffef799: mov%esi,%edx
> CORRECT original code
> > 0xfffef79b: out%eax,(%dx)
> > 0xfffef79c: mov$0xfe,%dl
> > 0xfffef79e: in (%dx),%ax
>
> (That's some code from seabios-1.7.0/src/pci.c)
>
> I had exactly the same run some weeks ago, but I also get different
> patterns:
> > # diff --suppress-common-lines -y <(od -Ax -tx1 -w1 -v /tmp/bios2.dump)
> > <(od -Ax -tx1 -w1 -v bios.bin)
> > 00f798 f0 | 00f798 80
> > 00f799 8d | 00f799 89
> > 00f79a f3 | 00f79a f2
> > 016d40 00 | 016d40 ff
> > 016d41 00 | 016d41 ff
> > 016d42 00 | 016d42 ff
> > 016d43 00 | 016d43 ff
>
> Not all runs lead to reboot problems, but I don't know if any other
> corruption happened there.
>
> I had a similar problem with OVMF back in June
> ,
> which I "solved" by upgrading the OVMF version: I have not seen the
> problem there since than, but this problems looks very similar.
>
>
> 1. How can it be, that the low-mem ROM mapping is modified?
I can't remember all the details, but PC ROM is shadowed and mapped over
with RAM at various times, and the bioses play lots of silly tricks of
being copied and then reusing bits of the copied space as temporaries
and. oh it's just a mess.
Dave
> 2. Can I tell QEMU or gdb to trap any modification of that 128 KiB area?
>
> I'll try to get http://rr-project.org/ running, but any help is appreciated.
>
> Philipp
> --
> Philipp Hahn
> Open Source Software Engineer
>
> Univention GmbH
> be open.
> Mary-Somerville-Str. 1
> D-28359 Bremen
> Tel.: +49 421 22232-0
> Fax : +49 421 22232-99
> h...@univention.de
>
> http://www.univention.de/
> Geschäftsführer: Peter H. Ganten
>
