Re: "Instant clone" with Qemu?

[Philipp Hahn]

Hello Stefan,

Am 15.12.23 um 16:21 schrieb Stefan Hajnoczi:

Am 05.12.23 um 15:44 schrieb Stefan Hajnoczi:

On Tue, 5 Dec 2023 at 04:53, Philipp Hahn  wrote:

  >

by accident I stumbled over "VMware Instant Clone" ¹, which allows
cloning of running VMs by copy-on-write-sharing the disk images and
memory content; the network MAC address gets changed (or a different
bridge is used?).
I wonder if something similar can also be done with Qemu? My current
solution would be to:
- start and install the VM
- create a live-snapshot into the qcow2 file
- clone the disk image, e.g. put a qcow2 overlay on it per clone
- start and restore the clones from that live-snapshot
- put the clones in individual bridges and let the host do some network
address translation (NAT) to give each clone a unique external IP address.

Has someone done something similar or is there even a better alternative?

Background: our test suite currently provisions a set of multiple VMs,
which are dependent on each other. Provisioning them takes sometimes
many hours. After that the test suite runs inside these VMs and again
takes many hours.
I'd like to speed that up by parallelizing these tests, e.g.
1. setup the VM environment once
2. clone the VM environments as the resources allow
3. distribute tests over these environments to run in parallel and to
allow running flaky tests multiple times from a clean clone again


It would be simplest to use qcow2 backing files and boot each new
instance from scratch. This involves setting up a master image and
then "qemu-img create -f qcow2 -b master.img vm001.qcow2" to create
the instance image. You may be able to use systemd or your distro's
"first boot" functionality to recreate unique IDs and cryptographic
keys when the new instance boots.


Actually I do not want to modify the clones at all: While the machine ID
is probably less interesting to others, I can even live with re-using
the SSH keys as this is only for *internal* testing: I can tell `ssh` to
not check the keys as I can control all the networking, so security is
of little concern here.


If you really want to use a RAM snapshot then I suggest creating a
qcow2 master image with the savevm command and using "cp
--reflink=always master.qcow2 vm001.qcow2" to create an efficient copy
of the qcow2 file. You'll need some custom scripts to recreate unique
IDs and cryptographic keys inside the new instance after loadvm.


Is there a major difference between doing a "savevm" to an external file
and doing a live snapshot, which stores the "savevm" inside the qcow2
file itself. The later has the benefit for me, that I only have to
handle one file; I could even store it for later use if needed.


With the reflink approach you still snapshot the VM into the original
qcow2 file (with the "savevm" command), not into an external file. The
reflink creates an efficient copy of the file for each instance of the
VM that you with to clone. But since you've said you don't want to
modify the clones at all, maybe this approach is overkill because you
have to manage these new qcow2 files.

Have you tried the -snapshot command-line option?


Thank you for the reminder, I already used it in other contexts but 
forgot it here. I'll add it to my toolshed.


Regarding reflink: that requires FS support, which AFAIKS is only 
supported by BTRFS and XFS (and OCFS2). So using `-snapshot` or an 
explicit `qemu-img create -f qcow2 -b $base.qcow2 $overlay.qcow2` works 
on any FS.



My main problem currently is cloning the MAC address: As our product is
an operating system the MAC addresses of the involved systems is stored
in some databases; while in most cases they are not required, I do not
want to hunt for these in all kind of different locations and change
them to some cloned MAC address.
I already had a look at "Virtual Routing and Forwarding"², which allows
me to resue the same MAC addresses in different network bridge
interfaces, but what I did not yet get to work is the "routing" between
them. I found some very nice articles³⁴ on how to do NAT with VRF, but
it is not yet working.


I'm not knowledgeable about VRF. You could also use -netdev
user,hostfwd=tcp::$VM_SSH_NAT_PORT-:22 where VM_SSH_NAT_PORT is a
unique port assigned by the script that launches the guest. That way
each guest can have the same MAC address and IP address but receive
incoming SSH connections.


Good to know, thank you. My problem is that I have to clone multiple VMs 
belonging together, they must be able to communicate with each other 
using their unmodified IP addresses; I only need to connect to one of 
them from the outside; something like a "jump host".


Thanks again
Philipp

Re: "Instant clone" with Qemu?

[Philipp Hahn]

Hello Stefan,

thank you for your kind reply.

Am 05.12.23 um 15:44 schrieb Stefan Hajnoczi:

On Tue, 5 Dec 2023 at 04:53, Philipp Hahn  wrote:

>

by accident I stumbled over "VMware Instant Clone" ¹, which allows
cloning of running VMs by copy-on-write-sharing the disk images and
memory content; the network MAC address gets changed (or a different
bridge is used?).
I wonder if something similar can also be done with Qemu? My current
solution would be to:
- start and install the VM
- create a live-snapshot into the qcow2 file
- clone the disk image, e.g. put a qcow2 overlay on it per clone
- start and restore the clones from that live-snapshot
- put the clones in individual bridges and let the host do some network
address translation (NAT) to give each clone a unique external IP address.

Has someone done something similar or is there even a better alternative?

Background: our test suite currently provisions a set of multiple VMs,
which are dependent on each other. Provisioning them takes sometimes
many hours. After that the test suite runs inside these VMs and again
takes many hours.
I'd like to speed that up by parallelizing these tests, e.g.
1. setup the VM environment once
2. clone the VM environments as the resources allow
3. distribute tests over these environments to run in parallel and to
allow running flaky tests multiple times from a clean clone again


It would be simplest to use qcow2 backing files and boot each new
instance from scratch. This involves setting up a master image and
then "qemu-img create -f qcow2 -b master.img vm001.qcow2" to create
the instance image. You may be able to use systemd or your distro's
"first boot" functionality to recreate unique IDs and cryptographic
keys when the new instance boots.


Actually I do not want to modify the clones at all: While the machine ID 
is probably less interesting to others, I can even live with re-using 
the SSH keys as this is only for *internal* testing: I can tell `ssh` to 
not check the keys as I can control all the networking, so security is 
of little concern here.



If you really want to use a RAM snapshot then I suggest creating a
qcow2 master image with the savevm command and using "cp
--reflink=always master.qcow2 vm001.qcow2" to create an efficient copy
of the qcow2 file. You'll need some custom scripts to recreate unique
IDs and cryptographic keys inside the new instance after loadvm.


Is there a major difference between doing a "savevm" to an external file 
and doing a live snapshot, which stores the "savevm" inside the qcow2 
file itself. The later has the benefit for me, that I only have to 
handle one file; I could even store it for later use if needed.



My main problem currently is cloning the MAC address: As our product is 
an operating system the MAC addresses of the involved systems is stored 
in some databases; while in most cases they are not required, I do not 
want to hunt for these in all kind of different locations and change 
them to some cloned MAC address.
I already had a look at "Virtual Routing and Forwarding"², which allows 
me to resue the same MAC addresses in different network bridge 
interfaces, but what I did not yet get to work is the "routing" between 
them. I found some very nice articles³⁴ on how to do NAT with VRF, but 
it is not yet working.


Philipp

¹<https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-853B1E2B-76CE-4240-A654-3806912820EB.html>
²<https://www.kernel.org/doc/html/latest/networking/vrf.html>
³<https://blog.oddbit.com/post/2023-02-19-vrf-and-nat/>
⁴<https://www.linuxquestions.org/questions/linux-newbie-8/iptables-nat-for-vrf-4175721876/>

"Instant clone" with Qemu?

[Philipp Hahn]

Hello,

by accident I stumbled over "VMware Instant Clone" ¹, which allows 
cloning of running VMs by copy-on-write-sharing the disk images and 
memory content; the network MAC address gets changed (or a different 
bridge is used?).
I wonder if something similar can also be done with Qemu? My current 
solution would be to:

- start and install the VM
- create a live-snapshot into the qcow2 file
- clone the disk image, e.g. put a qcow2 overlay on it per clone
- start and restore the clones from that live-snapshot
- put the clones in individual bridges and let the host do some network 
address translation (NAT) to give each clone a unique external IP address.


Has someone done something similar or is there even a better alternative?

Background: our test suite currently provisions a set of multiple VMs, 
which are dependent on each other. Provisioning them takes sometimes 
many hours. After that the test suite runs inside these VMs and again 
takes many hours.

I'd like to speed that up by parallelizing these tests, e.g.
1. setup the VM environment once
2. clone the VM environments as the resources allow
3. distribute tests over these environments to run in parallel and to 
allow running flaky tests multiple times from a clean clone again


¹<https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-853B1E2B-76CE-4240-A654-3806912820EB.html>
--
Philipp Hahn
Open Source Software Engineer

Univention GmbH
Mary-Somerville-Str. 1
28359 Bremen
Germany | Deutschland
Phone: +49 (0)421 22232-0 | E-Mail: i...@univention.de

https://www.univention.de | https://www.univention.com

Managing Directors: Peter H. Ganten, Stefan Gohmann
Local court: Amtsgericht Bremen
HRB 20755 | Ust-ID: DE220051310

Re: VNC clipboard support

[Philipp Hahn]

Hello,

Am 29.04.23 um 17:29 schrieb octavef...@outlook.fr:

I'm trying to use the copy/paste with VNC.

I'm launching qemu with:

$ qemu-system-x86_64 -hda debiandisk.img vnc :1

I'm using tightvncviewer which has support for copy/paste.

I try to copy text between guest and host.

It doesn't work. Neither from host to guest or guest to host.

As far as I know, there is clipboard support in VNC (ui/vnc-clipboard.c 
and so on).

With wireshark, I can see that tightvncviewer send clipboard requests.

Am I missing some configuration?


Clipboard via VNC to QEMU does not work: Basically the VNC protocol was 
originally used for remote-access to X11 application. There you work on 
a high-level were XKeySyms are used and where clipboard data can be 
exchanged.


In contrast with QEMU you work on a much lower level as you do hardware 
emulation: The operating system inside your QEMU process expects 
low-level hardware events like a USB or PS2 keyboard key Up / Down events.
As the VNC protocol only exchanges "XKeySyms" QEMU has to translate them 
back to those low-level USB/PS2 events expected by the OS.
This back-translation is not unique: For example most full-sized 
keyboards have the number block, so there are 2 keys to enter a digit.


Therefore QEMU added the "extended key event"-extension, which adds the 
low-level "KeyCode" in addition to the "XKeySym" to the VNC protocol: 
When you press a key in your VNC browser the X-Server luckily also gets 
the low-level "KeyCode", but the original VNC protocol did not include 
that information on the protocol level. By including that information 
QEMU no longer has to "fake" it and simply pass that information to the 
inside OS.


Even when your clipboards only contains text (it could also have a 
bitmap picture or some other rich data), you are basically back in the 
situation where you have to translate each character to one (or 
multiple) key-press/release events, again with the same issue mention 
above: You do not have the KeyCode and someone has to fake them.


Therefore it will not work: Albeit some VNC browsers or libraries 
implement that approach, but their result is unreliable, especially if 
the keyboard layout on your client computer does not match the keyboard 
layout configured in QEMU for the backward translation does not match 
the keyboard layout of your operating system running inside of QEMU.


If you want to know more, read this still excellent blog-post from 
Daniel Berrangé:
- 
https://www.berrange.com/posts/2010/07/04/more-than-you-or-i-ever-wanted-to-know-about-virtual-keyboard-handling/
- 
https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#qemu-extended-key-event-message


Philipp

Re: [Qemu-devel] storing machine data in qcow images?

[Philipp Hahn]

Hi,

Am 18.05.2018 um 17:30 schrieb Michael S. Tsirkin:
> Unfortunately this means that it's no longer possible
> to more or less reliably boot a VM just given a disk image,
> even if you select the correct QEMU binary:
...
> Would it be reasonable to support storing this information in the qcow
> image itself?  For example, I can see it following immediately the
> backing file path within the image.

- This looks like a layering violation
- what happens when you have multiple (conflicting) images like a VM
with 2 image files?

Philipp

PS: this is even more an issue for restoring snapshots as you the must
launch a new QEMU process with the exact layout of the saving QEMU
process - otherwise LoadVM will just fail.
PPS: I'm afraid of someone suggesting such an abomination as those self
extracting archives using shell scripts at the beginning and an
compressed archive BLOB at the end of the same file.

Re: [Qemu-devel] [PATCH 1/2] docs: Add image locking subsection

[Philipp Hahn]

Hello,

Am 23.11.2017 um 14:59 schrieb Fam Zheng:
> diff --git a/docs/qemu-block-drivers.texi b/docs/qemu-block-drivers.texi
> index 1cb1e55686..fa2e90d15f 100644
> --- a/docs/qemu-block-drivers.texi
> +++ b/docs/qemu-block-drivers.texi
> @@ -785,6 +785,42 @@ warning: ssh server @code{ssh.example.com:22} does not 
> support fsync
...
> +To check if image locking is active, check the output of "lslocks" command on
> +host and see if there are locks held by QEMU process on the image file. More
> +than one bytes could be locked by a QEMU instance, each byte of which 
> reflects

"one byte", without the 's'?

> +a perticular permission that are acquired or protected by the running block

"particular", 'e' -> 'a'

Philipp

Re: [Qemu-devel] [libvirt] How to best handle the reoccurring of rom changes breaking cross version migrations?

[Philipp Hahn]

Hello

Am 03.11.2017 um 08:30 schrieb Christian Ehrhardt:
> On Thu, Nov 2, 2017 at 4:34 PM, Daniel P. Berrange  
> wrote:
>>
>> On Thu, Nov 02, 2017 at 04:14:06PM +0100, Christian Ehrhardt wrote:
>>> Ping - since there wasn't any reply so far - any best practices one could
>>> share?
>>>
>>> Let me add a TL;DR:
>>> - bump of ipxe rom versions change the size of virtio-net-pci.rom
>>> - that breaks on migration "Length mismatch"
>>>
>>> I'd guess the size of that rom has to be fixed up on the fly, but if that
>>> is really ok and how/where is the question.
>>
>> The actual ROM contents will be transferred in the migration stream, so
>> the fact that the target host has ROMs with different content is not
>> important. The key thing that matters is that QEMU the target host loads
>> the ROMs at the same location, so that when the ROM contents is overwritten
>> with data from the incoming migration scheme, it all ends up at the same
>> place as it was on the source.
> 
> 
> Thanks Daniel for your answer, although you try to kill my remaining
> hopes of a better solution :-)
> But if the actual ROM content is migrated over anyway "all I'd have to
> do" is to make clear that the newer system (qemu) knows that the
> incoming rom has a different size than it thinks.
> I thought that was what the machine types and their mechanisms were
> for, but so far I only scratched like 10% of those - maybe they don't
> cover these rom sizes?

The problem is that libvirt launches a qemu process without knowing
those details.
Later when you try to restore an old snapshot or try to do a live
migration, the newly spawned qemu process must have the same layout to
be able to load the old VMState.

Qemu has gone through several such changes and is worsened by the fact,
that distributions like Debian use their own ROM files, which might be
different from the sizes shipped with Qemu.
For that reason we (Univention) still ship the old ROM images next to
the new ones and carry the attached patch to switch between them based
on the model. Maybe that works for you, too.

One more warning (we experiences): Even if the sizes are the same, live
migration can still fail afterwards: We have observed several cases,
where a migrated VM carried an old version of SeaBIOS, which fails when
you reboot the VM. We traced that down to Qemu implementing a new
feature (SMM), which was not supported by the old SeaBIOS and then got
it wrong after reboot only.

Hope that helps.

Philipp
Bug #24702: Fix PXE ROM size mismatch

For pc-0.14 and older use the PXE ROMs from the deprecated etherboot-qemu
package, which have different sizes. Without the patch the virtual PC gets
build with the current ROM sizes. Loading then fails because the stored PCI ROM
BAR size mismatches the configured size.
--- a/hw/pc_piix.c
+++ b/hw/pc_piix.c
@@ -433,6 +433,30 @@ static QEMUMachine pc_machine_v0_15 = {
 .driver   = "virtio-balloon-pci",\
 .property = "event_idx",\
 .value= "off",\
+},{\
+.driver   = "ne2000",\
+.property = "romfile",\
+.value   = "/usr/lib/etherboot/rtl8029.rom",\
+},{\
+.driver   = "rtl8139",\
+.property = "romfile",\
+.value   = "/usr/lib/etherboot/rtl8139.rom",\
+},{\
+.driver   = "e1000",\
+.property = "romfile",\
+.value   = "/usr/lib/etherboot/e1000-82540em.rom",\
+},{\
+.driver   = "pcnet",\
+.property = "romfile",\
+.value   = "/usr/lib/etherboot/pcnet32.rom",\
+},{\
+.driver   = "ne2k-isa",\
+.property = "romfile",\
+.value   = "/usr/lib/etherboot/ne.rom",\
+},{\
+.driver   = "virtio-net-pci",\
+.property = "romfile",\
+.value   = "/usr/lib/etherboot/virtio-net.rom",\
 }
 
 static QEMUMachine pc_machine_v0_14 = {

Re: [Qemu-devel] Why got no response of vnc?

[Philipp Hahn]

Hello,

Am 25.08.2017 um 08:31 schrieb Sam:
> I'm starting vm using:
> 
> kvm]$ sudo /usr/local/bin/qemu-system-x86_64 -m 256 -hda test.qcow2 -cdrom
>> CentOS-7-x86_64-DVD-1503-01.iso -boot d
>> VNC server running on '127.0.0.1:5900'
  ^
This a a *local* host socket,

> then I start vnc client in another PC to connect 10.253.23.24:11800 and
> 10.253.23.24:5900, I got time out, why?

to which you can't connect from any *other* host

$ man qemu-system-x86_64
...
> -vnc display[,option[,option[,...]]]

So try to add '-vnc 10.253.23.24:5900' to your command line to make Qemu
to listen on that IP address of your host you're running the VM on.

Philipp

Re: [Qemu-devel] RFH: difference in read-only mapped bios.bin - memory corruption?

[Philipp Hahn]

Hello,

Am 15.08.2017 um 13:25 schrieb Laszlo Ersek:
> On 08/14/17 20:39, Dr. David Alan Gilbert wrote:
>> * Philipp Hahn (h...@univention.de) wrote:
>>> I'm currently investigating a problem, were a Linux VM does not reboot
>>> and gets stuck in the SeaBIOS reboot code:
>>>
>>> I'm using SeaBIOS-1.7 from Debian with a more modern qemu-2.8
...>>> If I dump both regions and compare them, I get a difference:
...>> You might want seabios commit c68aff5 and b837e6 that got fixed after
>> I tracked down some reboot hangs - although they were rare, not every
>> time.  c68aff5 did certainly cause a corruption, and the address of that
>> corruption was determined at link time and could overlay random useful
>> bits of code if you were unlucky.

Thanks you for the commit IDs - to me this looks like they fixed the
problem. Testing with seabios-1.10 does not show any reboot problem so far.

>>> 1. How can it be, that the low-mem ROM mapping is modified?
>>
>> I can't remember all the details, but PC ROM is shadowed and mapped over
>> with RAM at various times,
> 
> Right. I don't remember for sure, but I believe the state of the PAM
> registers doesn't only affect what the VCPUs see in that address range,
> but also what your monitor commands will dump. (This would be the
> logical choice -- make the monitor output what the VCPUs see anyway, at
> the moment, dependent on the PAM settings.)

That makes sense.
Do you know by change what change in Qemu triggered that bug, as I've
never seen any reboot problem with qemu-1.1.2, but only since switching
to qemu-2.8?

Thanks again for your excellent help.

Philipp

[Qemu-devel] RFH: difference in read-only mapped bios.bin - memory corruption?

[Philipp Hahn]

Hello,

I'm currently investigating a problem, were a Linux VM does not reboot
and gets stuck in the SeaBIOS reboot code:

I'm using SeaBIOS-1.7 from Debian with a more modern qemu-2.8

> virsh # qemu-monitor-command --hmp ucs41-414 info roms
> fw=genroms/kvmvapic.bin size=0x002400 name="kvmvapic.bin"
> addr=fffe size=0x02 mem=rom name="bios.bin"

which (to my understanding) is mapped at two physical locations:
> virsh # qemu-monitor-command --hmp ucs41-414 info mtree
...> memory-region: system
...>   000e-000f (prio 1, R-): alias
isa-bios @pc.bios -0001
>   fffe- (prio 0, R-): pc.bios

If I dump both regions and compare them, I get a difference:
> virsh # qemu-monitor-command --pretty --domain ucs41-414 
> '{"execute":"pmemsave","arguments":{"val":917504,"size":131072,"filename":"/tmp/bios-low.dump"}}'
> virsh # qemu-monitor-command --pretty --domain ucs41-414 
> '{"execute":"pmemsave","arguments":{"val":4294836224,"size":131072,"filename":"/tmp/bios-high.dump"}}'
> # diff --suppress-common-lines -y <(od -Ax -tx1 -w1 -v /tmp/bios-low.dump) 
> <(od -Ax -tx1 -w1 -v /tmp/bios-high.dump)
> 00f798 fa | 00f798 80
> 00f799 7a | 00f799 89
> 00f79a f4 | 00f79a f2
> 016d40 00 | 016d40 ff
> 016d41 00 | 016d41 ff
> 016d42 00 | 016d42 ff
> 016d43 00 | 016d43 ff

The high address dump is the same as the original:
> # cmp -l /tmp/bios-high.dump /usr/share/seabios/bios.bin

> virsh # qemu-monitor-command --hmp ucs41-414 x/6i 0x000ef78f
> 0x000ef78f:  mov$0xcf8,%esi
> 0x000ef794:  mov$0xfa00,%eax
> 0x000ef799:  jp 0xef78f
   ^^ BUG: endless loop
> 0x000ef79b:  out%eax,(%dx)
> 0x000ef79c:  mov$0xfe,%dl
> 0x000ef79e:  in (%dx),%ax

> virsh # qemu-monitor-command --hmp ucs41-414 xp/6i 0xfffef78f
> 0xfffef78f:  mov$0xcf8,%esi
> 0xfffef794:  mov$0x8000,%eax
> 0xfffef799:  mov%esi,%edx
    CORRECT original code
> 0xfffef79b:  out%eax,(%dx)
> 0xfffef79c:  mov$0xfe,%dl
> 0xfffef79e:  in (%dx),%ax

(That's some code from seabios-1.7.0/src/pci.c)

I had exactly the same run some weeks ago, but I also get different
patterns:
> # diff --suppress-common-lines -y <(od -Ax -tx1 -w1 -v /tmp/bios2.dump) <(od 
> -Ax -tx1 -w1 -v bios.bin)
> 00f798 f0 | 00f798 80
> 00f799 8d | 00f799 89
> 00f79a f3 | 00f79a f2
> 016d40 00 | 016d40 ff
> 016d41 00 | 016d41 ff
> 016d42 00 | 016d42 ff
> 016d43 00 | 016d43 ff

Not all runs lead to reboot problems, but I don't know if any other
corruption happened there.

I had a similar problem with OVMF back in June
<https://lists.nongnu.org/archive/html/qemu-devel/2017-06/msg03940.html>,
which I "solved" by upgrading the OVMF version: I have not seen the
problem there since than, but this problems looks very similar.


1. How can it be, that the low-mem ROM mapping is modified?

2. Can I tell QEMU or gdb to trap any modification of that 128 KiB area?

I'll try to get http://rr-project.org/ running, but any help is appreciated.

Philipp
-- 
Philipp Hahn
Open Source Software Engineer

Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
h...@univention.de

http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876

Re: [Qemu-devel] [PATCH 01/11] qemu.py: Pylint/style fixes

[Philipp Hahn]

Hi,

Am 20.07.2017 um 18:28 schrieb Lukáš Doktor:
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
...
> @@ -64,16 +79,16 @@ class QEMUMachine(object):
>  if self._socket_scm_helper is None:
>  print >>sys.stderr, "No path to socket_scm_helper set"
>  return -1
> -if os.path.exists(self._socket_scm_helper) == False:
> +if os.path.exists(self._socket_scm_helper) is False:

if not os.path.exists(...):

Philipp

Re: [Qemu-devel] [RFH] qemu-2.6 memory corruption with OVMF and linux-4.9

[Philipp Hahn]

Hello,

Am 18.06.2017 um 20:22 schrieb Philipp Hahn:
> Am 17.06.2017 um 18:51 schrieb Laszlo Ersek:
>> (I also recommend using the "vbindiff" tool for such problems, it is
>> great for picking out patterns.)
>>
>>   ** ** ** ** ** ** ** **   8  9 ** ** ** 13 14 15
>>   -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- --
>>   01 e8 00 00 00 00 00 00  8c 5e 00 00 00 10 ff f1
>> 0010  5b 78 8a 3e 00 00 00 00  00 00 00 00 00 00 00 00
>> 0020  8c 77 00 00 00 12 00 02  18 f0 00 00 00 00 00 00
>> 0030  00 1e 00 00 00 00 00 00  8c 8c 00 00 00 12 00 02
>> 0040  07 70 00 00 00 00 00 00  00 14 00 00 00 00 00 00
>> 0050  8c 9c 00 00 00 12 00 02  22 00 00 00 00 00 00 00
>> 0060  00 40 00 00 00 00 00 00  8c ac 00 00 00 10 ff f1
>>
>>   01 e8 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
>> 0010  5b 78 8a 3e 00 00 00 00  00 3c 00 00 00 07 00 00
>> 0020  8c 77 00 00 00 12 00 02  00 3c 00 00 00 07 00 00
>> 0030  00 1e 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
>> 0040  07 70 00 00 00 00 00 00  00 3c 00 00 00 07 00 00
>> 0050  8c 9c 00 00 00 12 00 02  00 3c 00 00 00 07 00 00
>> 0060  00 40 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
>>   -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- --
>>   ** ** ** ** ** ** ** **   8  9 ** ** ** 13 14 15
>>
>> The columns that I marked with "**" are identical between "good" and
>> "bad". (These are columns 0-7, 10-12.)
>>
>> Column 8 is overwritten by zeros (every 16th byte).
>>
>> Column 9 is overwritten by 0x3c (every 16th byte).
>>
>> Column 13 is super interesting. The most significant nibble in that
>> column is not disturbed. And, in the least significant nibble, the least
>> significant three bits are turned on. Basically, the corruption could be
>> described, for this column (i.e., every 16th byte), as
>>
>>   bad = good | 0x7
>>
>> Column 14 is overwritten by zeros (every 16th byte).
>>
>> Column 15 is overwritten by zeros (every 16th byte).
>>
>> My take is that your host machine has faulty RAM. Please run memtest86+
>> or something similar.
> 
> I will do so, but for me very unlikely:
> - it never happens with BIOS, only with OVMF
> - for each test I start q new QEMU process, which should use a different
> memory region
> - it repeatedly hits e1000 or libata.ko

Okay: memtest+-5.01 run for 8h and did not find any errors in those 3
passes.

> After updating from OVMF to 0~20161202.7bbe0b3e-1 from
> (0~20160813.de74668f-2 it has not yet happened again.

Anyway, thank you for your help.

Philipp

Re: [Qemu-devel] [RFH] qemu-2.6 memory corruption with OVMF and linux-4.9

[Philipp Hahn]

Am 18.06.2017 um 20:27 schrieb Dr. David Alan Gilbert:
> * Philipp Hahn (h...@univention.de) wrote:
>> Hello,
>>
>> Am 17.06.2017 um 18:51 schrieb Laszlo Ersek:
>>> (I also recommend using the "vbindiff" tool for such problems, it is
>>> great for picking out patterns.)
>>>
>>>   ** ** ** ** ** ** ** **   8  9 ** ** ** 13 14 15
>>>   -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- --
>>>   01 e8 00 00 00 00 00 00  8c 5e 00 00 00 10 ff f1
>>> 0010  5b 78 8a 3e 00 00 00 00  00 00 00 00 00 00 00 00
>>> 0020  8c 77 00 00 00 12 00 02  18 f0 00 00 00 00 00 00
>>> 0030  00 1e 00 00 00 00 00 00  8c 8c 00 00 00 12 00 02
>>> 0040  07 70 00 00 00 00 00 00  00 14 00 00 00 00 00 00
>>> 0050  8c 9c 00 00 00 12 00 02  22 00 00 00 00 00 00 00
>>> 0060  00 40 00 00 00 00 00 00  8c ac 00 00 00 10 ff f1
>>>
>>>   01 e8 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
>>> 0010  5b 78 8a 3e 00 00 00 00  00 3c 00 00 00 07 00 00
>>> 0020  8c 77 00 00 00 12 00 02  00 3c 00 00 00 07 00 00
>>> 0030  00 1e 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
>>> 0040  07 70 00 00 00 00 00 00  00 3c 00 00 00 07 00 00
>>> 0050  8c 9c 00 00 00 12 00 02  00 3c 00 00 00 07 00 00
>>> 0060  00 40 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
>>>   -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- --
>>>   ** ** ** ** ** ** ** **   8  9 ** ** ** 13 14 15
>>>
>>> The columns that I marked with "**" are identical between "good" and
>>> "bad". (These are columns 0-7, 10-12.)
>>>
>>> Column 8 is overwritten by zeros (every 16th byte).
>>>
>>> Column 9 is overwritten by 0x3c (every 16th byte).
>>>
>>> Column 13 is super interesting. The most significant nibble in that
>>> column is not disturbed. And, in the least significant nibble, the least
>>> significant three bits are turned on. Basically, the corruption could be
>>> described, for this column (i.e., every 16th byte), as
>>>
>>>   bad = good | 0x7
>>>
>>> Column 14 is overwritten by zeros (every 16th byte).
>>>
>>> Column 15 is overwritten by zeros (every 16th byte).
>>>
>>> My take is that your host machine has faulty RAM. Please run memtest86+
>>> or something similar.
>>
>> I will do so, but for me very unlikely:
>> - it never happens with BIOS, only with OVMF
>> - for each test I start q new QEMU process, which should use a different
>> memory region
>> - it repeatedly hits e1000 or libata.ko
>>
>> After updating from OVMF to 0~20161202.7bbe0b3e-1 from
>> (0~20160813.de74668f-2 it has not yet happened again.
>>
>> Anyway, thank you for your help.
> 
> What host CPU are you using?

Everything is amd64:
> processor   : 3
> vendor_id   : GenuineIntel
> cpu family  : 6
> model   : 58
> model name  : Intel(R) Core(TM) i5-3337U CPU @ 1.80GHz
> stepping: 9
> microcode   : 0x19
> cpu MHz : 2591.015
> cache size  : 3072 KB
> physical id : 0
> siblings: 4
> core id : 1
> cpu cores   : 2
> apicid  : 3
> initial apicid  : 3
> fpu : yes
> fpu_exception   : yes
> cpuid level : 13
> wp  : yes
> flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx 
> rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology 
> nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est 
> tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer 
> aes xsave avx f16c rdrand lahf_lm epb tpr_shadow vnmi flexpriority ept vpid 
> fsgsbase smep erms xsaveopt dtherm ida arat pln pts
> bugs:
> bogomips: 3592.75
> clflush size: 64
> cache_alignment : 64
> address sizes   : 36 bits physical, 48 bits virtual
> power management:

Philipp

Re: [Qemu-devel] [RFH] qemu-2.6 memory corruption with OVMF and linux-4.9

[Philipp Hahn]

Hello,

Am 17.06.2017 um 18:51 schrieb Laszlo Ersek:
> (I also recommend using the "vbindiff" tool for such problems, it is
> great for picking out patterns.)
> 
>   ** ** ** ** ** ** ** **   8  9 ** ** ** 13 14 15
>   -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- --
>   01 e8 00 00 00 00 00 00  8c 5e 00 00 00 10 ff f1
> 0010  5b 78 8a 3e 00 00 00 00  00 00 00 00 00 00 00 00
> 0020  8c 77 00 00 00 12 00 02  18 f0 00 00 00 00 00 00
> 0030  00 1e 00 00 00 00 00 00  8c 8c 00 00 00 12 00 02
> 0040  07 70 00 00 00 00 00 00  00 14 00 00 00 00 00 00
> 0050  8c 9c 00 00 00 12 00 02  22 00 00 00 00 00 00 00
> 0060  00 40 00 00 00 00 00 00  8c ac 00 00 00 10 ff f1
> 
>   01 e8 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
> 0010  5b 78 8a 3e 00 00 00 00  00 3c 00 00 00 07 00 00
> 0020  8c 77 00 00 00 12 00 02  00 3c 00 00 00 07 00 00
> 0030  00 1e 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
> 0040  07 70 00 00 00 00 00 00  00 3c 00 00 00 07 00 00
> 0050  8c 9c 00 00 00 12 00 02  00 3c 00 00 00 07 00 00
> 0060  00 40 00 00 00 00 00 00  00 3c 00 00 00 17 00 00
>   -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- --
>   ** ** ** ** ** ** ** **   8  9 ** ** ** 13 14 15
> 
> The columns that I marked with "**" are identical between "good" and
> "bad". (These are columns 0-7, 10-12.)
> 
> Column 8 is overwritten by zeros (every 16th byte).
> 
> Column 9 is overwritten by 0x3c (every 16th byte).
> 
> Column 13 is super interesting. The most significant nibble in that
> column is not disturbed. And, in the least significant nibble, the least
> significant three bits are turned on. Basically, the corruption could be
> described, for this column (i.e., every 16th byte), as
> 
>   bad = good | 0x7
> 
> Column 14 is overwritten by zeros (every 16th byte).
> 
> Column 15 is overwritten by zeros (every 16th byte).
> 
> My take is that your host machine has faulty RAM. Please run memtest86+
> or something similar.

I will do so, but for me very unlikely:
- it never happens with BIOS, only with OVMF
- for each test I start q new QEMU process, which should use a different
memory region
- it repeatedly hits e1000 or libata.ko

After updating from OVMF to 0~20161202.7bbe0b3e-1 from
(0~20160813.de74668f-2 it has not yet happened again.

Anyway, thank you for your help.

Philipp

[Qemu-devel] [RFH] qemu-2.6 memory corruption with OVMF and linux-4.9

[Philipp Hahn]

dev file,id=charserial1,path=/tmp/uefi.log -device 
> isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -device 
> cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device 
> intel-hda,id=sound0,bus=pci.0,addr=0x4 -device 
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device 
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -global 
> isa-debugcon.iobase=0x402 -debugcon file:/tmp/ovmf.log -msg timestamp=on


Has someone seen a similar issue or is this even a known issue?


I will try a newer version of QEMU and OVMF next.

Thank you in advance for your input.

Philipp
-- 
Philipp Hahn
Open Source Software Engineer

Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
h...@univention.de

http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876

Re: [Qemu-devel] [PATCH 1/2] acpi_piix4: fix migration of gpe fields

[Philipp Hahn]

Hello Marcelo,

Am 21.03.2017 um 00:20 schrieb Marcelo Tosatti:
> On Mon, Mar 20, 2017 at 01:02:10PM +0100, Philipp Hahn wrote:
>> Hello Marcelo, cc:qemu,
>>
>> Sorry for re-using this old thread, but I have a problem loading some
>> saved state from qemu-kvm-1.1.2, which fails for piix4_pm.
>>
>> You following patch was committed as
>> <http://git.qemu-project.org/?p=qemu.git;a=commitdiff;h=b0b873a07872f7ab7f66f259c73fb9dd42aa66a9>:
>>
>> Am 15.11.2012 um 01:11 schrieb Marcelo Tosatti:
>>> Migrate 16 bytes for en/sts fields (which is the correct size),
>>> increase version to 3, and document how to support incoming
>>> migration from qemu-kvm 1.2.
>>
>> I my case qemu-kvm-1.1.2/hw/acpi_piix4.c:284
>> | VMSTATE_STRUCT(ar.gpe, PIIX4PMState, 2, vmstate_gpe, ACPIGPE),
>> only saves 4 bytes, not 16 bytes.
> 
> IIRC qemu-1.1.2 saved 16 bytes, which is the correct size, not 4 bytes.

I found 23910d3f669d46073b403876e30a7314599633af in qemu, which changed
"gpe" to be an array[4] instead of single "struct gpe_regs" without
changing the version number. So the incompatibility was introduced there.

> So while merging from qemu-kvm -> qemu, it was decided to maintain
> backwards compability with qemu, and not qemu-kvm.
> 
> Is there any way to differentiate between the two (qemu vs qemu-kvm, 
> perhaps via some other field not in the VMState of ACPI PIIX4), so
> your patch can be integrated upstream?

I had a look and haven't found any reliable indicator to distinguish
between "qemu" and "qemu-kvm" yet. I'll have another look if my time
permits, but as I know "qemu-kvm" was used in my case, I will hard code
that knowledge for now in my private build.

Thank you four sharing your knowledge.

Philipp

Re: [Qemu-devel] [PATCH 1/2] acpi_piix4: fix migration of gpe fields

[Philipp Hahn]

Hello Marcelo, cc:qemu,

Sorry for re-using this old thread, but I have a problem loading some
saved state from qemu-kvm-1.1.2, which fails for piix4_pm.

You following patch was committed as
<http://git.qemu-project.org/?p=qemu.git;a=commitdiff;h=b0b873a07872f7ab7f66f259c73fb9dd42aa66a9>:

Am 15.11.2012 um 01:11 schrieb Marcelo Tosatti:
> Migrate 16 bytes for en/sts fields (which is the correct size),
> increase version to 3, and document how to support incoming
> migration from qemu-kvm 1.2.

I my case qemu-kvm-1.1.2/hw/acpi_piix4.c:284
| VMSTATE_STRUCT(ar.gpe, PIIX4PMState, 2, vmstate_gpe, ACPIGPE),
only saves 4 bytes, not 16 bytes.

> Acked-by: Paolo Bonzini 
> Signed-off-by: Marcelo Tosatti 
> ---
>  hw/acpi_piix4.c |   50 ++
>  1 files changed, 46 insertions(+), 4 deletions(-)
> 
> diff --git a/hw/acpi_piix4.c b/hw/acpi_piix4.c
> index 15275cf..519269a 100644
> --- a/hw/acpi_piix4.c
> +++ b/hw/acpi_piix4.c
> @@ -235,10 +235,9 @@ static int vmstate_acpi_post_load(void *opaque, int 
> version_id)
>   {   \
>   .name   = (stringify(_field)),  \
>   .version_id = 0,\
> - .num= GPE_LEN,  \
>   .info   = &vmstate_info_uint16, \
>   .size   = sizeof(uint16_t), \
> - .flags  = VMS_ARRAY | VMS_POINTER,  \
> + .flags  = VMS_SINGLE | VMS_POINTER, \
>   .offset = vmstate_offset_pointer(_state, _field, uint8_t),  \
>   }
>  
> @@ -267,11 +266,54 @@ static const VMStateDescription vmstate_pci_status = {
>  }
>  };

On load qemu-2.8 uses this function

> +static int acpi_load_old(QEMUFile *f, void *opaque, int version_id)
> +{
> +PIIX4PMState *s = opaque;
> +int ret, i;
> +uint16_t temp;
> +
> +ret = pci_device_load(&s->dev, f);
> +if (ret < 0) {
> +return ret;
> +}
> +qemu_get_be16s(f, &s->ar.pm1.evt.sts);
> +qemu_get_be16s(f, &s->ar.pm1.evt.en);
> +qemu_get_be16s(f, &s->ar.pm1.cnt.cnt);
> +
> +ret = vmstate_load_state(f, &vmstate_apm, opaque, 1);
> +if (ret) {
> +return ret;
> +}
> +
> +qemu_get_timer(f, s->ar.tmr.timer);
> +qemu_get_sbe64s(f, &s->ar.tmr.overflow_time);
> +
> +qemu_get_be16s(f, (uint16_t *)s->ar.gpe.sts);
> +for (i = 0; i < 3; i++) {
> +qemu_get_be16s(f, &temp);
> +}

and the loop loads 6 extra bytes here

> +
> +qemu_get_be16s(f, (uint16_t *)s->ar.gpe.en);
> +for (i = 0; i < 3; i++) {
> +qemu_get_be16s(f, &temp);
> +}

and 6 here.
So in total 12 bytes are read too much and qemu_loadvm_state() ends
somewhere in the middle of the next device state section. (patch 0001
helped me to identify that offset)

If I apply the attached 0002 patch, I can load the old VM state.

> +
> +ret = vmstate_load_state(f, &vmstate_pci_status, opaque, 1);
> +return ret;
> +}
> +
> +/* qemu-kvm 1.2 uses version 3 but advertised as 2
> + * To support incoming qemu-kvm 1.2 migration, change version_id
> + * and minimum_version_id to 2 below (which breaks migration from
> + * qemu 1.2).
> + *
> + */
>  static const VMStateDescription vmstate_acpi = {
>  .name = "piix4_pm",
> -.version_id = 2,
> -.minimum_version_id = 1,
> +.version_id = 3,
> +.minimum_version_id = 3,
>  .minimum_version_id_old = 1,
> +.load_state_old = acpi_load_old,
>  .post_load = vmstate_acpi_post_load,
>  .fields  = (VMStateField []) {
>  VMSTATE_PCI_DEVICE(dev, PIIX4PMState),
> 

Do you remember why 16 bytes were saved in your case?

Thank you in advance.

Philipp
From 4e43999b0f56975dbbd528e4956231b2ee64c071 Mon Sep 17 00:00:00 2001
Message-Id: <4e43999b0f56975dbbd528e4956231b2ee64c071.1490011194.git.h...@univention.de>
In-Reply-To: <1313efdc0ac664169c5c271f832264efe6659743.1490011194.git.h...@univention.de>
References: <1313efdc0ac664169c5c271f832264efe6659743.1490011194.git.h...@univention.de>
From: Philipp Hahn 
Date: Mon, 20 Mar 2017 12:36:53 +0100
Subject: [PATCH 2/2] 0007-Bug-38877-fix-qemu-kvm-1.1-piix4_pm-incompatibility
Organization: Univention GmbH, Bremen, Germany
To: qemu-devel@nongnu.org

qemu-kvm-1.1 only saves 2*uint16, but qemu-2.8 tries to load 2*4*uint16.

The code was added by b0b873a07872f7ab7f66f259c73fb9dd42aa66a9 and the
commit message claims 16 bytes are the supposed size, but as UCS never
used qemu-k

[Qemu-devel] RFC: Qemu/SeaBIOS/VGA/PXE upgrades vs. longterm-snapshot/migration

[Philipp Hahn]

Hello,

If my understanding of migration/snapshots is correct, the
target/loading VM must be a clone of the source/saving VM, that is have
the same devices, RAM/PCI layout, etc. In the past I have had several
issues with Qemu, when the distribution (Debian) updates their packaging
of SeaBIOS/iPXE/VgaROM, which changes the the next/previous power-of-2
size and thus changes the PCI layout, preventing me from loading old
snapshots.

As the BIOS file is provided externally, it is prone to such
(accidental) updates. As (AFAIK) libvirt does not track the Qemu
version, BIOS files inside its XML data, there's always the danger of
making snapshots invalid by updating Qemu and the BIOS files.

Some questions:

1. Does Qemu use ROM shadowing, that is copying the ROM to RAM? This
makes sense on real HW as ROMs are a low slower than RAM. In that case
the content of the "ROM" would already be contained inside the VM-State
and it would be enough to provide an "empty ROM file of the right size".

2. If Qemu does no ROM shadowing, what happens if I suspend a VM while
executing ROM or SMM code and then doing an SeaBIOS update? (my
DOS-assembler knowledge is quiet old nowadays)

3. How do others handle long-term snapshots? Just say "good-bye" to your
old snapshots when upgrading to a newer Qemu version or keeping the old,
probably unmaintained and vulnerable Qemu/BIOS binaries until the
end-of-time?

Thanks in advance
Philipp
-- 
Philipp Hahn
Open Source Software Engineer

Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
h...@univention.de

http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876

[Qemu-devel] virnet: page allocation failure: order:0

[Philipp Hahn]

Hello,

this Sunday one of our virtual servers running linux-4.1.16 inside
OpenStack using qemu "crashed" while doing a backup using rsync to a
slow NFS server.
Crash here means that the server became unresponsive to network traffic:
- it was no longer able to contact the two LDAP servers
- no ssh login was possible
- the backup got stuck
- crond was still running and added process after process, leading to
~1.5k processes running after one day.

What I don't know is if the network problem was the cause or the
consequence. Because of that I want to understand why the follwoing
order=0 allocation failed:

> swapper/0: page allocation failure: order:0, mode:0x120

 4KiB
src/extern/linux/include/linux/gfp.h:
  18 #define ___GFP_HIGH»»···0x20u
  21 #define ___GFP_COLD»»···0x100u
  72 #define __GFP_HIGH»·((__force gfp_t)___GFP_HIGH)»···/* Should
access emergency pools? */


  75 #define __GFP_COLD»·((__force gfp_t)___GFP_COLD)»···/* Cache-cold
page required */



> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.0-ucs190-amd64 #1 Debian
4.1.6-1.190.201604142226

The kernel is 4.1.16 with some patches from Debian and some others.

> Hardware name: OpenStack Foundation OpenStack Nova, BIOS Bochs 01/01/2011
>    81597807 0120
>  8116b6e9 00018200 88021fffbb00 
>  88021fffbb08  88021fffab50 818164e0
> Call Trace:
>[] ? dump_stack+0x40/0x50
>  [] ? warn_alloc_failed+0xf9/0x150
>  [] ? __alloc_pages_nodemask+0x65a/0x9d0
>  [] ? alloc_pages_current+0xa4/0x120
>  [] ? skb_page_frag_refill+0xb7/0xe0
>  [] ? try_fill_recv+0x31b/0x610 [virtio_net]
>  [] ? virtnet_receive+0x580/0x890 [virtio_net]

Received network packet, but failed to copy from VirtIO to local kernel
memory.

>  [] ? virtnet_poll+0x26/0x90 [virtio_net]
>  [] ? net_rx_action+0x159/0x330
>  [] ? __do_softirq+0xde/0x260
>  [] ? irq_exit+0x95/0xa0
>  [] ? do_IRQ+0x64/0x110
>  [] ? common_interrupt+0x6e/0x6e
>[] ? mwait_idle+0x150/0x150
>  [] ? native_safe_halt+0x2/0x10
>  [] ? default_idle+0x1c/0xb0
>  [] ? cpu_startup_entry+0x314/0x3e0
>  [] ? start_kernel+0x48d/0x498
>  [] ? set_init_arg+0x50/0x50
>  [] ? early_idt_handler_array+0x117/0x120
>  [] ? early_idt_handler_array+0x117/0x120
>  [] ? x86_64_start_kernel+0x14a/0x159
> Mem-Info:
> active_anon:34861 inactive_anon:3120 isolated_anon:0
>  active_file:941577 inactive_file:946489 isolated_file:0
>  unevictable:974 dirty:2953 writeback:282206 unstable:1277
>  slab_reclaimable:55077 slab_unreclaimable:31365
>  mapped:3979 shmem:3986 pagetables:3163 bounce:0
>  free:9437 free_pcp:740 free_cma:0

Looks like enough memory is free

> Node 0 DMA free:15908kB min:20kB low:24kB high:28kB active_anon:0kB
inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB
isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB
pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> lowmem_reserve[]: 0 3492 7971 7971
> Node 0 DMA32 free:19532kB min:4968kB low:6208kB high:7452kB
active_anon:62328kB inactive_anon:5336kB active_file:1659704kB
inactive_file:1671236kB unevictable:2056kB isolated(anon):0kB
isolated(file):0kB present:3653624kB managed:3578476kB mlocked:2056kB
dirty:10472kB writeback:479692kB mapped:11892kB shmem:7132kB
slab_reclaimable:79832kB slab_unreclaimable:46784kB kernel_stack:2000kB
pagetables:4660kB unstable:2596kB bounce:0kB free_pcp:1080kB
local_pcp:120kB free_cma:0kB writeback_tmp:0kB pages_scanned:0
all_unreclaimable? no
> lowmem_reserve[]: 0 0 4478 4478
> Node 0 Normal free:2308kB min:6372kB low:7964kB high:9556kB
active_anon:77116kB inactive_anon:7144kB active_file:2106604kB
inactive_file:2114720kB unevictable:1840kB isolated(anon):0kB
isolated(file):0kB present:4718592kB managed:4586204kB mlocked:1840kB
dirty:1340kB writeback:649132kB mapped:4024kB shmem:8812kB
slab_reclaimable:140476kB slab_unreclaimable:78676kB kernel_stack:1888kB
pagetables:7992kB unstable:2512kB bounce:0kB free_pcp:1880kB
local_pcp:164kB free_cma:0kB writeback_tmp:0kB pages_scanned:768
all_unreclaimable? no
> lowmem_reserve[]: 0 0 0 0

Show also enough pages being available

> Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U)
1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (EM) = 15908kB
> Node 0 DMA32: 4632*4kB (UEM) 81*8kB (UMR) 4*16kB (R) 1*32kB (R) 0*64kB
1*128kB (R) 1*256kB (R) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 19656kB
> Node 0 Normal: 315*4kB (UM) 1*8kB (R) 1*16kB (R) 1*32kB (R) 1*64kB (R)
1*128kB (R) 2*256kB (R) 1*512kB (R) 0*1024kB 0*2048kB 0*4096kB = 2532kB
> Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0
hugepages_size=2048kB

Here too.

> 1892811 total pagecache pages
> 0 pages in swap cache
> Swap cache stats: add 0, delete 0, find 0/0
>

Re: [Qemu-devel] [libvirt] [PATCH v2 3/8] Add support for fetching statistics of completed jobs

[Philipp Hahn]

Hi,

FYI as tracking down that failure did cost me some time and hopefully
this summary will help other to avoid this situation:

Am 09.09.2014 um 11:54 schrieb Jiri Denemark:
> virDomainGetJobStats gains new VIR_DOMAIN_JOB_STATS_COMPLETED flag that
> can be used to fetch statistics of a completed job rather than a
> currently running job.
> 
> Signed-off-by: Jiri Denemark 
...
> --- a/src/qemu/qemu_monitor_json.c
> +++ b/src/qemu/qemu_monitor_json.c
> @@ -2500,7 +2500,8 @@ qemuMonitorJSONGetMigrationStatusReply(virJSONValuePtr 
> reply,
...
> -if (status->status == QEMU_MONITOR_MIGRATION_STATUS_ACTIVE) {
> +if (status->status == QEMU_MONITOR_MIGRATION_STATUS_ACTIVE ||
> +status->status == QEMU_MONITOR_MIGRATION_STATUS_COMPLETED) {
>  virJSONValuePtr ram = virJSONValueObjectGet(ret, "ram");
>  if (!ram) {
>  virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
>_("migration was active, but RAM 'transferred' "
>  "data was missing"));

This change from libvirt v1.2.9-rc1~203 breaks migration with
qemu-kvm-1.1.2, as that ancient implementation does *not* export
transfer statistics for completed jobs:

> qemuMonitorJSONCommandWithFd:286 : Send command 
> '{"execute":"query-migrate","id":"libvirt-41"}' for write with FD -1
> qemuMonitorJSONIOProcessLine:179 : Line [{"return": {"status": "active", 
> "ram": {"total": 2164654080, "remaining": 22474752, "transferred": 
> 175117413}}, "id": "libvirt-41"}]
...
> qemuMonitorJSONCommandWithFd:286 : Send command 
> '{"execute":"query-migrate","id":"libvirt-42"}' for write with FD -1
> qemuMonitorJSONIOProcessLine:179 : Line [{"return": {"status": "completed"}, 
> "id": "libvirt-42"}]

As you can see, there is not "ram" section and the migration is aborted
with the message:
> internal error: migration was active, but no RAM info was set

qemu-kvm/qmp-commands.hx even states this:
> - "ram": only present if "status" is "active", it is a json-object with the
>   following RAM information (in bytes):
but the example some lines below is wrong:
> 2. Migration is done and has succeeded
> 
> -> { "execute": "query-migrate" }
> <- { "return": {
> "status": "completed",
> "ram":{
That example has been updated by v1.2.0-rc0~29^2~2, but forgot to update
the specification above.


The attached hack for libvirt makes migration work again for me by
making "ram" optional in case of "completed".

Philipp
From c4f6dfb25042f420fdd1728865686552d928d90e Mon Sep 17 00:00:00 2001
Message-Id: 
From: Philipp Hahn 
Date: Mon, 9 May 2016 10:52:09 +0200
Subject: [PATCH] Bug #40318 libvirt: Handle qemu-kvm-1.1.2 migration
 incompatibility
Organization: Univention GmbH, Bremen, Germany
To: libvir-l...@redhat.com

The change from libvirt v1.2.9-rc1~203 breaks migration with qemu-kvm-1.1.2, as
that ancient implementation does *not* export transfer statistics for completed
jobs:

> qemuMonitorJSONCommandWithFd:286 : Send command '{"execute":"query-migrate","id":"libvirt-41"}' for write with FD -1
> qemuMonitorJSONIOProcessLine:179 : Line [{"return": {"status": "active", "ram": {"total": 2164654080, "remaining": 22474752, "transferred": 175117413}}, "id": "libvirt-41"}]
...
> qemuMonitorJSONCommandWithFd:286 : Send command '{"execute":"query-migrate","id":"libvirt-42"}' for write with FD -1
> qemuMonitorJSONIOProcessLine:179 : Line [{"return": {"status": "completed"}, "id": "libvirt-42"}]

As you can see, there is not "ram" section and the migration is aborted with
the message:
> internal error: migration was active, but no RAM info was set

qemu-kvm/qmp-commands.hx even states this:
> - "ram": only present if "status" is "active", it is a json-object with the
>   following RAM information (in bytes):
but the example some lines below is wrong:
> 2. Migration is done and has succeeded
>
> -> { "execute": "query-migrate" }
> <- { "return": {
> "status": "completed",
> "ram":{
That example has been updated by v1.2.0-rc0~29^2~2, but forgot to update the
s

Re: [Qemu-devel] how to setup a watchdog?

[Philipp Hahn]

Hi,

Am 18.01.2016 um 12:43 schrieb lejeczek:
> I'm trying Qemu's watchdog.
> My understanding was that hardware (here qemu's watchdog) would take
> action, eg. cold reboot the system if there is no ping from the OS
> watchdog, so I
> thought stopping watchdog service in VM should be a quick test, right?
> 
> I have this in the guest:
> 
> 
>function='0x0'/>
> 
> 
> and I see /dev/watchdog in my guest. Yet nothing happens, guest(linux)
> runs uninterrupted.
> I must be missing something, an expert said it's config problem, is it
> really is?

Probably reading linux/Documentation/watchdog/watchdog-api.txt should
give you some extra hints.

AFAIK you need to open that device file at least once, either with the
sample program included in the Linux source tree, or by another program
like (Debian) watchdog or systemd (man 5 systemd-system.conf ->
RuntimeWatchdogSec)

  cat /dev/watchdog # will print an error about the read(), but the
important thing (opening the file) has been done
  dmesg | grep watchdog # will print "i6300esb: Unexpected close, not
stopping watchdog!" to show you that the watchdog is still running and
will trigger the reboot, as it has gone through the shutdown protocol,
which can be used to disable the watchdog again

After ~1m my test VM rebooted.

Philipp

Re: [Qemu-devel] VirtIO windows driver: viostor.sys not post-installable

[Philipp Hahn]

Hello,

On 31.05.2015 12:58, Vadim Rozenfeld wrote:
> On Sun, 2015-05-31 at 11:26 +0300, Yan Vugenfirer wrote:
>>> On May 29, 2015, at 5:43 PM, Philipp Hahn 
>>> wrote:
...
>>> we tried to migrate some Windows 2008 and 2012 VMs from Xen to KVM,
>>> but
>>> installing the VirtIO viostor.sys driver fails, because the
>>> signature of
>>> the driver doesn't seem to match what's stored in the
>>> corresponding .cat
>>> file.
...
>> Can you send the error message you are getting from Windows? If
>> possible attach setupapi.log as well (search your system
>> for setupapi.*, the location might be different for different OS
>> versions).

I attached them to <https://bugzilla.redhat.com/show_bug.cgi?id=1226928>

> Do you use the same installation media for v2v and a fresh install, or
> vfd for a fresh install and iso for v2v conversion? 

If the drivers are installed during the initial setup, both ISO and VFD
work - after navigating into the right sub directory and enabling "show
not matching drivers". No BSOD.

Using the same ISO after doing the install using the emulated IDE, the
drivers are rejected with:

> Processing inf :vioscsi.inf
> Adding the driver package failed : The hash for the file is not present in 
> the specified catalog file. The file is likely corrupt or the victim of 
> tampering.
> 
> Processing inf :viostor.inf
> Adding the driver package failed : The hash for the file is not present in 
> the specified catalog file. The file is likely corrupt or the victim of 
> tampering.


>>> Are there some Linux tools to work with the .cat files and
>>> signatures to make sure they match?

Answering my own question: .crt files are DER encoded. "dumpasn1" shows
them to contain PKCS#7 data.

>> Vadim and I monitor qemu-devel, you can also open bug in
>> bugzilla.redhat.com for virtio-win component or report an issue
>> here: https://github.com/YanVugenfirer/kvm-guest-drivers-windows/issues

I filed <https://bugzilla.redhat.com/show_bug.cgi?id=1226928>. If
anything is still missing, just ask.

Thanks for the fast help and for any more help in advance
Philipp Hahn

[Qemu-devel] VirtIO windows driver: viostor.sys not post-installable

[Philipp Hahn]

Hello,

we tried to migrate some Windows 2008 and 2012 VMs from Xen to KVM, but
installing the VirtIO viostor.sys driver fails, because the signature of
the driver doesn't seem to match what's stored in the corresponding .cat
file.

On the other hand installing the drivers during a fresh install from the
beginning never had any problems.


We use
<https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.104-1/virtio-win-0.1.104.iso>
but also tried "virtio-win-0.1.103.iso" and "virtio-win-0.1-81.iso".

Running the following command on 0.1.104 prints (among others) the
following sha1hash:
> "C:\Program Files (x86)\Windows Kits\8.1\bin\x86\signtool.exe" /verify
/v /kp E:\NetKVM\2k12\amd64\netkvm.sys
...
> Hash of file (sha1): 135E3AA23217610AEE8046F68550B0BA86F4EAE6

> "C:\Program Files (x86)\Windows Kits\8.1\bin\x86\signtool.exe" /verify
/v /kp E:\viostor\2k12\amd64\viostor.sys
...
> Hash of file (sha1): EF11F5E539EEE0A9DB6DF3710A0DAA35066C5607

Looking into the corresponding .cat "Security Catalog File"
- netkvm.cat contains the above given hash for netkvm.sys,
- viostor.cat contains 55FC4DA2EE96ECC3FD4865680436DCDA6B8C6BDD instead!

Running "sha1sum" on Linux print some completely different hashes, so I
don't know what the Microsoft tool actually hash:

> # sha1sum /cdrom/NetKVM/2k12/amd64/netkvm.sys 
> /cdrom/viostor/2k12/amd64/viostor.sys 
> 1aa91c8e1d7680457d92c1875810a79f68af536d  /cdrom/NetKVM/2k12/amd64/netkvm.sys
> f39bc2b561091addfcac30e370227c91700d2698  
> /cdrom/viostor/2k12/amd64/viostor.sys

Is this a known issue?

Are there some (working) alternatives?

Are there some Linux tools to work with the .cat files and signatures to
make sure they match?

Is there some better mailing list for VirtIO Windows driver issues?


Some more background for our migration procedure:

- The VM was installed some years are on Xen.
- The GPLPV drivers were added afterwards.
- For the migration the GPLPV drivers were disabled and then removed.
- A 2nd VirtIO hard-disk was added in KVM to trigger Windows to request
the virstor driver.

If you need any more data, just ask.

Thanks in advance
Philipp Hahn

PS: data was copied by hand from Windows, so it might contains
copy-paste-errors.
-- 
Philipp Hahn
Open Source Software Engineer

Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
h...@univention.de

http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876

Re: [Qemu-devel] [PATCH v10] Support vhd type VHD_DIFFERENCING

[Philipp Hahn]

Hello,

On 11.03.2015 07:22, Xiaodong Gong wrote:
>> Hope that clarified things.
...
> first，your patch is very clear，a good sample.
> 
> store ascii code in kernel that I said before is a mistake，I mean the
> glibc need the input of arguments of fuction such as fopen（path）is
> ascii code

No:
 ASCII = *7* bit, see "man 7 ascii".

 Kernel = *8* bit, that is the kernel doesn't care if you use
ISO-8859-1, UTF-8, BIG5, GB2312, or any other encoding you can get by
running "iconv --list".
For the kernel only '\0'=0x00 and '/'=0x2f are special, all other
characters the kernel doesn't care for and passes them in and out
unmodified.

Most character sets have the ASCII alphabet in their character range
0x00-0x7f, which solved the '\0' and '/' issue nicely.

So again:
- If you use opendir() and readdir(), the kernel returns to you a 8 bit
byte sequence.
- To convert that into a character sequence, you must know which
encoding was used when the file was created. This information is not
stored explicitly in the file-system, file-name, or anywhere else.

- The only hint you get is LC_CTYPE, which is set by the user to tell
you which encoding should be used to convert a byte-stream into a
character-string.

- If I create a new file on the Linux text console, where I did a
"unicode_start", I get an UTF-8 byte sequence from the input layer,
which is passed unmodified through the getty and by shell to the
create() call. You don't need to know the encoding, you just pass the
data in and out unmodified.

- When typing "ls" the kernel again return that byte sequence, which
gets passed through the shell to the Linux frame buffer, which
translates that UTF-8 sequence to a character and picks the right glyph
for being displayed on the screen.

- If I don't switch the Linux console to Unicode mode, I get a different
byte sequence from the input layer. That different byte sequence would
be stored on the disk when creating a file. (This translation is
installed by running the "loadkeys" command.)

- If I do the same in X11, the translation from key-codes to characters
and back is done by the terminal (or application). See "man 1 xterm" for
"-lc" and "-u8".

- BUT when you want to create a specific character encoding, you MUST
know from which encoding you start. Assuming ASCII or UTF-8 is wrong,
you MUST check LC_ALL/LC_CTYPE/LANG by querying nl_langinfo(CODESET)).

So if I would give you a disk containing a file with the name "\xa1",
depending on the locale you would see a different glyph:
$ for ((c=1;c<=15;c++));do printf '\xa1'|recode
ISO-8859-$c..dump-with-names 2>/dev/null|tail -n 1;done
00A1   !Iinverted exclamation mark
0104   A;latin capital letter a with ogonek
0126   H/latin capital letter h with stroke
0104   A;latin capital letter a with ogonek
0401   IOcyrillic capital letter io
201B   9'single high-reversed-9 quotation mark
00A1   !Iinverted exclamation mark
0104   A;latin capital letter a with ogonek
00A1   !Iinverted exclamation mark
00A1   !Iinverted exclamation mark
1E02   B.latin capital letter b with dot above
00A1   !Iinverted exclamation mark

In an UTF-8 environment you would get an error, as "\xa1" is not a valid
UTF-8 byte sequence.

Read "man 7 unicode", especially the section "Unicode Under Linux" or
"man 7 charsets".

> I think：

Any program basically has two options:

1. The program does not care about different character sets ans just
passes in file-names and data in and out as byte streams. That is
perfectly okay and most UNIX shell commands work just fine that way.

2. The program is encoding aware, as for example it works on characters
instead of bytes (like "wc --bytes" vs. "wc --chars") or does need to
perform a conversion between encodings. Then the sanest thing is to
- query the encoding of the environment once (or per network connection),
- convert any input data from that encoding into a (fixed) internal
format like wchar/utf-16/utf-32 including file-names, file-content, etc.
- convert the internal data back into the right format on output, which
also includes calling APIs like open().

Otherwise you always have to remember if your char[] buffer contains
"some byte stream, which needs to be decoded() before being used" or
"already decoded character string". That is why most libraries and
frame-works contain wrappers for the file, input- and output, as they
internally use one data type for characters consistently and hide all
the explicit conversion from you by providing wrappers.

> icovn_open（utf16le，ascii）in encode
> icovn_open（ascii，utf16le）in decode
> icovn_open（codeset，ascii）in show

That would be correct ONLY if you store the file-name internally as
ASCII, which would prevent you from handling file-names containing any
character outside the ASCII codeset.
You should use "UTF-8" instead of "ascii", as that allows you to handle
file-names containing any valid characters.
This would make the conversion in show() trivial when codeset="UTF-8",
as

Re: [Qemu-devel] [PATCH v10] Support vhd type VHD_DIFFERENCING

[Philipp Hahn]

Hello,

On 08.03.2015 02:53, Xiaodong Gong wrote:
> the encoding type of parent location is must be utf 8，utf16e，according
> to the draft

Yes, the SPEC for VPC/VHD specifies the character encoding to use, which
is good for being portable.

> ascii is the encoding type to store the string of parent location in
> memery and to use fopen（）

No: For the (Linux) kernel the filename is a sequence of 8 bit bytes,
where only '\0'=end_of_string and '/'=path_separator are handled
specially. All other bytes have no special meaning and are passed in and
out as is.

Only the applications are doing the character encoding. Normally this is
not a problem as you setup your system once with one encoding (nowadays
UTF-8) and use that consistently: If you enter ä on the keyboard,
the kernels input layer returns \u00E4 as the two-byte UTF-8 sequence
> $ echo -n ä | xxd -g 1
> 000: c3 a4
Any application can either just pass the byte sequence around as a CLOB
(or use any other encoding internally - but then it must know that the
input-encoding is UTF-8), but when again doing any system call, they
will again pass that same byte sequence as the file-name, which the
kernel will store on disk.
If you take that disk to another computer, which does NOT use Unicode,
you have a problem: If, for example, that one is still using the old
ISO-8859-1 encoding used in western Europe, you file will be named
differently:
> $ echo -n ä | iconv -f ISO-8859-1 -t UTF-8
> ä

(The reverse is even more painful, as not any ISO-8859-1 character
sequence is a valid UTF-8 byte sequence - several years back when I
moved from my old ISO-8859-1 to a more modern UTF-8 setup, I had to
rename lots of files to be readable again)

You can even test that locally on one system by creating a file
containing an umlaut in its name and then to display that in a non-UTF-8
terminal / environment:
> $ touch ä
> $ LANG=C ls -NQ
> "\303\244"

> ascii need to translate to other encoding type according to LANG when to
> show the information of the vhd file using the qemu-info and so on

No: your assumption that ASCII is used is IMHO wrong: ASCII is only 7
bit, but the kernel interface is 8 bit. The terminal input- and output
layer nowadays are UTF-8, so as long as you're working on the console
everything is fine. If you mix in GUIs and libraries doing their own
encoding/decoding, things get more interesting.

But when you do explicit character conversion like you do for VHD, you
must honor the user configured character encoding of the environment
yourself, that is use LC_CTYPE for any conversion from input, for output
which includes file names.

I checked xen/tools/blktap2/vpc/lib/libvhd.c #
vhd_initialize_header_parent_name()
which also (wrongly) assumes ASCII. Because of the creating a snapshot
using vhd-utils is also broken:

> $ /usr/bin/vhd-util create -n ä.vhd -s 1
> $ /usr/bin/vhd-util snapshot -n snap.vhd -p ä.vhd ; echo $?
> 84

Next I checked

to create a VHD using umlauts with Windows 7:

> cmd # as Admin
> diskpart
> create vdisk file="C:\ä.vhd" maximum=2000 type=expandable
> create vdisk file="C:\snap.vhd" parent="C:\ä.vhd"

But vhd-utils from Xen is broken:

> $ /usr/bin/vhd-util read -n snap.vhd -p
> VHD Header Summary:
...
> Parent name : failed to read name
...
> VHD Parent Locators:
> 
> locator:: 0

> failed to read parent name

With the attached patch it works:

> VHD Header Summary:
> ---
...
> Parent name : /ä.vhd
...
> VHD Parent Locators:
> 
> locator:: 0
>code : PLAT_CODE_W2KU
...
>decoded name : /ä.vhd
> 
> locator:: 1
>code : PLAT_CODE_W2RU
...
>decoded name : ./ä.vhd

Hope that clarified things.

Philipp
conectix
ŽÓ½win 
Wi2k}}ß?ÿÿì×» ÄMéN¿å¦Ÿ¢I"ºcxsparse
è ÿÿ㈚(ƒûøG’ā¨~nÇ5ŽÓ5C:\ä.vhdW2kuW2ru


Re: [Qemu-devel] [PATCH v10] Support vhd type VHD_DIFFERENCING

[Philipp Hahn]

Hi,

On 04.03.2015 15:18, Xiaodong Gong wrote:
> @@ -157,6 +178,224 @@ static int vpc_probe(const uint8_t *buf, int buf_size, 
> const char *filename)
..
> +static int vpc_decode_maxc_loc(BlockDriverState *bs, uint32_t data_length)
...
> +cd = g_iconv_open("ASCII", "UTF8");
...
> +static int vpc_decode_w2u_loc(BlockDriverState *bs, uint32_t data_length)
...
> +cd = g_iconv_open("ASCII", "UTF-16LE");

Please correct me if my understanding is wrong, but a hard-coded "ASCII"
is AFAIK wrong, as it only contains the 7-bit characters.

For the Linux kernel the file name is just a string of bytes, but when
it gets displayed to the user, the bytes are converted to characters.
The conversion depends on the locale used, which now-adays is most often
UTF-8 (LANG=de_DE.UTF-8, or more specifically LC_CTYPE), but some years
back it was ISO-8859-1 (or what-ever).

So if I create a backing file with some non-ASCII umlauts, the
conversion will break, as ß = ß = \uc39f = ISO-8859-1(0xdf)

AFAIK using nl_langinfo(CODESET) would return the codeset previously set
by setlocale(LC_ALL, ""), which any main program would need to do.

Am I missing something?

Sincerely
Philipp

[Qemu-devel] [PATCH v2] hw/dma/i8257: Silence phony error message

[Philipp Hahn]

Convert into trace event. Otherwise the message
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=1
gets printed every time and fills up the log-file with 50 MiB / minute.

Signed-off-by: Philipp Hahn 
---
 v2:
   Convert into trace event instead of tracking once per static bitmap.
---
 hw/dma/i8257.c | 4 ++--
 trace-events   | 3 +++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/hw/dma/i8257.c b/hw/dma/i8257.c
index dd370ed..a414029 100644
--- a/hw/dma/i8257.c
+++ b/hw/dma/i8257.c
@@ -24,6 +24,7 @@
 #include "hw/hw.h"
 #include "hw/isa/isa.h"
 #include "qemu/main-loop.h"
+#include "trace.h"
 
 /* #define DEBUG_DMA */
 
@@ -473,8 +474,7 @@ static void dma_reset(void *opaque)
 
 static int dma_phony_handler (void *opaque, int nchan, int dma_pos, int 
dma_len)
 {
-dolog ("unregistered DMA channel used nchan=%d dma_pos=%d dma_len=%d\n",
-   nchan, dma_pos, dma_len);
+trace_i8257_unregistered_dma(nchan, dma_pos, dma_len);
 return dma_pos;
 }
 
diff --git a/trace-events b/trace-events
index 03ac5d2..c12afc0 100644
--- a/trace-events
+++ b/trace-events
@@ -1318,3 +1318,6 @@ mhp_pc_dimm_assigned_address(uint64_t addr) "0x%"PRIx64
 # target-s390x/kvm.c
 kvm_enable_cmma(int rc) "CMMA: enabling with result code %d"
 kvm_clear_cmma(int rc) "CMMA: clearing with result code %d"
+
+# hw/dma/i8257.c
+i8257_unregistered_dma(int nchan, int dma_pos, int dma_len) "unregistered DMA 
channel used nchan=%d dma_pos=%d dma_len=%d"
-- 
1.9.1

Re: [Qemu-devel] [PATCH v2] Support vhd type VHD_DIFFERENCING

[Philipp Hahn]

Hello,

I'm no qemu-devel expert, but as I tried myself at adding VHD_DIFF
support some time ago, here are some comments:

On 08.09.2014 16:41, Xiaodong Gong wrote:
> diff --git a/block/vpc.c b/block/vpc.c
...
> +/* Read backing file location from dyn header table */
> +if (dyndisk_header->parent_name[0] || 
> dyndisk_header->parent_name[1]) {
> +for (i = 0; i < PARENT_LOCATOR_NUM; i++) {
...
> +if (MACX == platform) {

Most images I have luckily have the MACX entry, but from reading the
spec I think this is not guaranteed. What about the other more
Windows-agnostic types?

There's also dyndisk_header->parent_name (which uses UTF-16-BE).

> +ret = bdrv_pread(bs->file, data_offset + 
> PARENT_PREFIX_LEN,
> +bs->backing_file, data_length - PARENT_PREFIX_LEN);

You assume that the system locale is UTF-8 (which MACX uses as the
encoding). I don't know how QEMU handles that internally, but AFAIK for
correctness you would need to convert that from UTF-8 to $LC_CTYPE.
Using iconv() is currently is not possible, since it requires calling
setlocale() to be called from all main programs using that code.

>  static int vpc_write(BlockDriverState *bs, int64_t sector_num,
...
> +bool isdiff = true;
...
> +if (true == isdiff) {

 if (isdiff) {
is enough - no need to add any confusing ==true IMHO.


Other notes:
- Using backing files requires CONFIG_UUID. I once created a VHD file
using qemu-img, which set UUID:=0. This lead to Xen handling the image
as a raw-file instead of a vhd file instead.

Otherwise thank you for working on VHD support.

Sincerely
Philipp

[Qemu-devel] [PATCH] hw/dma: Print error message only once

[Philipp Hahn]

otherwise the message
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=1
gets printed every time and fills up the log-file with 50 MiB / minute.

Signed-off-by: Philipp Hahn 
---
 hw/dma/i8257.c | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/hw/dma/i8257.c b/hw/dma/i8257.c
index dd370ed..9673ab6 100644
--- a/hw/dma/i8257.c
+++ b/hw/dma/i8257.c
@@ -473,8 +473,14 @@ static void dma_reset(void *opaque)
 
 static int dma_phony_handler (void *opaque, int nchan, int dma_pos, int 
dma_len)
 {
-dolog ("unregistered DMA channel used nchan=%d dma_pos=%d dma_len=%d\n",
-   nchan, dma_pos, dma_len);
+static int once;
+int mask = 1 << nchan;
+
+if (0 == (once & mask)) {
+once |= mask;
+dolog("unregistered DMA channel used nchan=%d dma_pos=%d dma_len=%d\n",
+  nchan, dma_pos, dma_len);
+}
 return dma_pos;
 }
 
-- 
1.9.1

Re: [Qemu-devel] Differential VHD

[Philipp Hahn]

Hello,

On 15.05.2014 13:54, Ankur Srivastava wrote:
> What all steps to be taken for differential vhd image? i created one
> differential image using vhd-util

Last time I checked it it wass not implemented in QEMU.
I started working on it in my spare time, but never completed it.

Sincerely
Philipp

[Qemu-devel] [BUG] 50MB/min logspam: dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=1

[Philipp Hahn]

Hello,

a Xen-4.1-3 host system filled its log file with the message
 dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=1

The happened two times was a Windows Server 2003 with GPLPV 0.11.0.372:
one VM was migrated to that host and one was directly started there.
After a restart of the VM the message no longer gets printed.

Looking at both xen-4.1.3/tools/ioemu-qemu-xen/hw/dma.c and
qemu-git/hw/dma/i8257.c the message is printed by dma_phony_handler(),
which is the default is no other transfer_handler is registered.

The function gets run through DMA_run_bh -> DMA_run -> channel_run.
In DMA_run() the code checks both the 'mask' and 'status' registers:
383   if ((0 == (d->mask & mask)) && (0 != (d->status & (mask << 4 {
384   channel_run (icont, ichan);
385   rearm = 1;
386}

So the messages is printed with maximum frequency, as the BH gets
executed each time leading to massive log spam.

Looking further I analyzed the following case, which *might* be responsible:

While 'mask' is included in the VMStateDescription, 'status' is not.
Before the conversion to VMSTATE* the code was commented out from day-one:
512 /* qemu_put_8s (f, &d->status); */
540 /* qemu_get_8s (f, &d->status); */

1. Might it be that 'status' is uninitialized after migration?
2. Has someone other seen that message?
3. Could the rate of the message please be limited? See attached patch.
4. Some other ideas or comments?

Sincerely
Philipp
-- 
Philipp Hahn
Open Source Software Engineer

Univention GmbH
be open.
Mary-Somerville-Str. 1
D-28359 Bremen
Tel.: +49 421 22232-0
Fax : +49 421 22232-99
h...@univention.de

http://www.univention.de/
Geschäftsführer: Peter H. Ganten
HRB 20755 Amtsgericht Bremen
Steuer-Nr.: 71-597-02876
>From 5e54adc8f53df4b8c8e8b802049964b2054ad829 Mon Sep 17 00:00:00 2001
Message-Id: <5e54adc8f53df4b8c8e8b802049964b2054ad829.1399625721.git.h...@univention.de>
From: Philipp Hahn 
Date: Fri, 9 May 2014 10:53:57 +0200
Subject: [PATCH] hw/dma: Print error message only once
Organization: Univention GmbH, Bremen, Germany
To: qemu-devel@nongnu.org

otherwise the message
	dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=1
gets printed every time and fills up the log-file with 50 MiB / minute.

Signed-off-by: Philipp Hahn 
---
 hw/dma/i8257.c | 10 --
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/hw/dma/i8257.c b/hw/dma/i8257.c
index 4490372..cd710a9 100644
--- a/hw/dma/i8257.c
+++ b/hw/dma/i8257.c
@@ -473,8 +473,14 @@ static void dma_reset(void *opaque)
 
 static int dma_phony_handler (void *opaque, int nchan, int dma_pos, int dma_len)
 {
-dolog ("unregistered DMA channel used nchan=%d dma_pos=%d dma_len=%d\n",
-   nchan, dma_pos, dma_len);
+static int once = 0;
+int mask = 1 << nchan;
+
+if (0 == (once & mask)) {
+once |= mask;
+dolog ("unregistered DMA channel used nchan=%d dma_pos=%d dma_len=%d\n",
+   nchan, dma_pos, dma_len);
+}
 return dma_pos;
 }
 
-- 
1.9.1

[Qemu-devel] VHD suport in QEMU?

[Philipp Hahn]

Hello,

sorry for high-jacking this thread, but I have a questing regarding VHD 
(without the X): Xen-4.3 switched to upstream QEMUs support for VHD. The 
dropped their own VHD implementation in blktap2, which supported backing 
chains, which QEMU currently doesn't. The Xen implementation 
(xen/tools/blktap2/vhd/lib/ is BSD 3 clause) also supports some kind of journal.
1. Does someone know if the journal support for VHD is somehow related to VHDX?
2. Has somewone put some work into supporting backing chains? I started to have 
a look myself some time ago, but had to do other work since then.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/
Director:Peter H. Ganten   HRB 20755 Amtsgericht Bremen   UID:DE 220 051 310

Re: [Qemu-devel] Qemu Boot failure with /dev/ram0

[Philipp Hahn]

Hello,

On Monday 16 September 2013 07:40:08 Saptarshi Sen wrote:
>I want to use a ram disk as a block device for my kernel.( i am
> measuring file io on ram )
> 
>I am trying to boot a kernel from using /dev/ram0. However I am getting
> "No root Device found"
> 
>qemu-system-x86_64 -m  8G -hda disk.img -kernel vmlinuz-3.10.0-rc6
> -initrd initramfs-3.10.0-rc6.img -append "root=/dev/ram0 selinux=0
> enforcing=0 rw" -vnc :0

Perhaps you're mixing "Initial*RamFS*" and "Initial*RootDisk*"? Your filename 
"initramfs" indicates you're going for the former, but your "root=/dev/ram0" is 
required fror InitRD.

InitRamFS is a (compressed) CPIO archive, which is extracted to a dynamically 
growing "tmpfs" by the kernel.
InitRD is a (compressed) ext2/3/4/jfs/xfs/whatever filesystem, which is backed 
by a fixed portion of you RAM; you probably need to explicitly specify a 
"ramdisk_size=25" parameter to be large enougth to contain the uncompressed 
file-system. If the size is too small, you will have a corrupt FS where you 
also get the error you mention above.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/
Director:Peter H. Ganten   HRB 20755 Amtsgericht Bremen   UID:DE 220 051 310

[Qemu-devel] backing file support for vpc/vhd ?

[Philipp Hahn]

Hello,

while working on a Xen-VM I noticed that qemu-img does not support the backing 
file mechanism of vpc/vhd files, but Xens blktap2 does; it has a libvhd and 
some tools to create and modify vpc/vhd files:

tools/blktap2/vhd/lib/atomicio.c: BSD (2 clause)
tools/blktap2/vhd/lib/vhd-util-check.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-modify.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-uuid.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-scan.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-revert.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-snapshot.c: BSD (3 clause)
tools/blktap2/vhd/lib/libvhd.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-query.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-set-field.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-repair.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-fill.c: BSD (3 clause)
tools/blktap2/vhd/lib/libvhd-journal.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-read.c: BSD (3 clause)
tools/blktap2/vhd/lib/relative-path.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-coalesce.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-create.c: BSD (3 clause)
tools/blktap2/vhd/lib/vhd-util-resize.c: BSD (3 clause)
tools/blktap2/vhd/vhd-update.c: BSD (3 clause)
tools/blktap2/vhd/vhd-util.c: BSD (3 clause)

Is the lack of support for backing files in vpc/vjd in qemu just a question of 
time / interest, are there technical problems, or are there some political 
issues?

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/

[Qemu-devel] RFH: boot from virtio cdrom?

[Philipp Hahn]

Hello,

I'm using libvirt to manage my VMs and configured one VM to boot from a CDROM 
connected via virtio.
This does neither work with QEMU-1.1.2 nor with QEMU-1.5; neither with SeaBIOS 
is 1.7.0 nor 1.7.2.

/root/qemu-1.5.0+dfsg/qemu-build/x86_64-softmmu/qemu-system-x86_64 \
 -M pc-1.1 \
 -enable-kvm \
 -m 1024 \
 -smp 1,sockets=1,cores=1,threads=1 \
 -name ucs31-64-xxx \
 -uuid 1dae3236-23e8-c87f-ec7b-9f82060fcc72 \
 -nodefconfig -nodefaults \
 -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/ucs31-64-xxx.monitor,server,nowait
 \
 -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc \
 -no-shutdown \
 -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
 -drive 
file=/var/lib/libvirt/images/ucs31-64-xxx-0.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=unsafe
 \
 -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
 \
 -drive 
file=/root/ucs-3.2-0/./virtualization/univention-kvm-virtio/virtio-win-0.1-30.iso,if=none,media=cdrom,id=drive-virtio-disk1,readonly=on,format=raw
 \
 -device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1,bootindex=1
 \
 -device usb-tablet,id=input0 \
 -vga cirrus \
 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
 -sdl

Is this supposed to work or must a bootable CDROM use the ide bus?
I ask because if this is not supposed to work, I need to modify our admin to to 
prevent CDROMs from being added through virtio by default; later on when Linux 
is installed, the CDROM works fine through virtio.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/

Re: [Qemu-devel] [PATCH for-1.4 0/2] fix migration failure from 1.3 due to SeaBIOS size change

[Philipp Hahn]

Hello,

On Wednesday 06 February 2013 01:48:24 Michael Roth wrote:
> Migration from 1.3 currently fails due to a mismatch between the expected
> size of 256KB and the received size of 128KB for seabios.
...
> 2) We can begin checking in past versions of SeaBIOS and other roms and
> loading those according to selected machine type. This would be similar to
> be how we've dealt with similar issues regarding the default vram size
> changing for VGA, but might not scale well with ISPs.

We currenty are carrying the attached patch with our Debian based 
distribution, which fixes a very similar issue with the PXE-BIOSs, because 
Debian provides it's own compilation of etherboot and iPXE.

As each upgrade threatens to break all your previous snapshots, would it help 
to include the BIOSs within the saved state instead of letting qemu pickup the 
current external file?

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/
diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/patches/romfile-compatibility.diff qemu-kvm-1.1.2+dfsg/debian/patches/romfile-compatibility.diff
--- qemu-kvm-1.1.2+dfsg.orig/debian/patches/romfile-compatibility.diff	1970-01-01 01:00:00.0 +0100
+++ qemu-kvm-1.1.2+dfsg/debian/patches/romfile-compatibility.diff	2012-10-08 14:25:09.225656404 +0200
@@ -0,0 +1,39 @@
+Bug #24702: Fix PXE ROM size mismatch
+
+For pc-0.14 and older use the PXE ROMs from the deprecated etherboot-qemu
+package, which have different sizes. Without the patch the virtual PC gets
+build with the current ROM sizes. Loading then fails because the stored PCI ROM
+BAR size mismatches the configured size.
+--- a/hw/pc_piix.c
 b/hw/pc_piix.c
+@@ -433,6 +433,30 @@ static QEMUMachine pc_machine_v0_15 = {
+ .driver   = "virtio-balloon-pci",\
+ .property = "event_idx",\
+ .value= "off",\
++},{\
++.driver   = "ne2000",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/rtl8029.rom",\
++},{\
++.driver   = "rtl8139",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/rtl8139.rom",\
++},{\
++.driver   = "e1000",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/e1000-82540em.rom",\
++},{\
++.driver   = "pcnet",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/pcnet32.rom",\
++},{\
++.driver   = "ne2k-isa",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/ne.rom",\
++},{\
++.driver   = "virtio-net-pci",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/virtio-net.rom",\
+ }
+ 
+ static QEMUMachine pc_machine_v0_14 = {
diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/patches/series qemu-kvm-1.1.2+dfsg/debian/patches/series
--- qemu-kvm-1.1.2+dfsg.orig/debian/patches/series	2012-09-10 12:14:05.0 +0200
+++ qemu-kvm-1.1.2+dfsg/debian/patches/series	2012-10-08 14:24:51.497155189 +0200
@@ -8,3 +8,4 @@
 
 net-add--netdev-options-to-man-page.patch
 revert-serial-fix-retry-logic.patch
+romfile-compatibility.diff
diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/control qemu-kvm-1.1.2+dfsg/debian/control
--- qemu-kvm-1.1.2+dfsg.orig/debian/control	2012-09-11 08:29:43.0 +0200
+++ qemu-kvm-1.1.2+dfsg/debian/control	2012-10-08 15:06:39.201155102 +0200
@@ -37,7 +37,8 @@
  seabios (>> 1.7.0~), vgabios (>= 0.6c-3~),
  qemu-keymaps, qemu-utils (>> 0.14.0),
  ipxe-qemu | ipxe (<< 1.0.0+git-20120202.f6840ba-2)
-Recommends: bridge-utils, iproute
+Recommends: bridge-utils, iproute,
+ etherboot-qemu
 Suggests: debootstrap, vde2, samba
 Provides: kvm
 Conflicts: kvm-source (<= 18-1), kvm-data (<= 66+dfsg-1.1), kvm (<< 1:0)

Re: [Qemu-devel] [PATCH] vmdk: Allow space in file name

[Philipp Hahn]

Hello,

On Tuesday 29 January 2013 22:50:31 Philipp Hahn wrote:
> The previous scanf() format string stopped parsing the file name on the
> first white white space, which seems to be allowed at least by VMware
> Wokrstation.
>
> Change the format string to collect everything between the first and
> second quote as the file name, disallowing line breaks.
>
> Signed-off-by: Philipp Hahn 
Forgot to explicitly cc: the peaople from the first round.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [BUG, RFC] block/vmdk.c: File name with space fails to open

[Philipp Hahn]

Hello,

On Friday 25 January 2013 09:37:39 Markus Armbruster wrote:
> Suggest to include '\n' in the stop set, like \"%511[^\"\n]\", to better
> detect malformed input.

Will do, also \r.

> > I don't know how portable %[ together with a maximum width is, because
> > the manual page for sscanf() doesn't mention "max width" for "%[", but it
> > works with Debian/GNU Linux Squeeze.
>
> It's fine according to my reading of C89.

Thank you for lloking this up.

> I'm afraid your patch is flawed.  For
>
> RW 1048576 FLAT ""test-f001.vmdk"" 0

You seem to assume " is allowed in the file name, I've assumed that " is not 
allowed, since we don't know the quoting rules for vmdk files, if there are 
any.
That's why I checked our old VMware workstation, which refused to create 
volumes containing !#%^&*><:;'"<>/?
Should we print a warning or error out if a " is detected?

> fname is now "test-f001.vmdk" instead of "\"test-f001.vmdk\"".  That's
> because you change sscanf() to ignore the double-quotes without dropping
> the quote stripping code below.

I'll remove the stripping code.

> Care to post a fixed up patch?

Will do so.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

[Qemu-devel] [PATCH] vmdk: Allow space in file name

[Philipp Hahn]

The previous scanf() format string stopped parsing the file name on the
first white white space, which seems to be allowed at least by VMware
Wokrstation.

Change the format string to collect everything between the first and
second quote as the file name, disallowing line breaks.

Signed-off-by: Philipp Hahn 
---
 block/vmdk.c |   10 +-
 1 files changed, 1 insertions(+), 9 deletions(-)

[V2] Also remove " striping code.
 Add \n\r to stop character set.

diff --git a/block/vmdk.c b/block/vmdk.c
index 19298c2..20ad646 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -641,7 +641,7 @@ static int vmdk_parse_extents(const char *desc, 
BlockDriverState *bs,
  * RW [size in sectors] SPARSE "file-name.vmdk"
  */
 flat_offset = -1;
-ret = sscanf(p, "%10s %" SCNd64 " %10s %511s %" SCNd64,
+ret = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\n\r\"]\" %" SCNd64,
 access, §ors, type, fname, &flat_offset);
 if (ret < 4 || strcmp(access, "RW")) {
 goto next_line;
@@ -653,14 +653,6 @@ static int vmdk_parse_extents(const char *desc, 
BlockDriverState *bs,
 return -EINVAL;
 }
 
-/* trim the quotation marks around */
-if (fname[0] == '"') {
-memmove(fname, fname + 1, strlen(fname));
-if (strlen(fname) <= 1 || fname[strlen(fname) - 1] != '"') {
-return -EINVAL;
-}
-fname[strlen(fname) - 1] = '\0';
-}
 if (sectors <= 0 ||
 (strcmp(type, "FLAT") && strcmp(type, "SPARSE")) ||
 (strcmp(access, "RW"))) {
-- 
1.7.1

[Qemu-devel] [BUG, RFC] block/vmdk.c: File name with space fails to open

[Philipp Hahn]

Hello,

I tried to open a "twoGbMaxExtentSparse" VMDK file, which uses spaces in its 
own and for the referenced file names. This breaks in line 646 of 
block/vmdk.c because "%511s" stops at the first space and thus fname is 
incomplete:
ret = sscanf(p, "%10s %" SCNd64 " %10s %511s %" SCNd64,
access, §ors, type, fname, &flat_offset);

I've only checked with our very old VMware workstation version, which refuses 
to create new images with unsupported characters with the following message:
> The characters !#%^&*><:;'"<>/? cannot be used.
So it looks like spaces are valid, at least we have several VMs with spaces in 
their name.

If the quotes around the file name are required, the simpliest solution would 
be to change %511s to "%511[^"]":

diff --git a/block/vmdk.c b/block/vmdk.c
index 19298c2..045f6a1 100644
--- a/block/vmdk.c
+++ b/block/vmdk.c
@@ -641,7 +641,7 @@ static int vmdk_parse_extents(const char *desc, 
BlockDriverState *bs,
  * RW [size in sectors] SPARSE "file-name.vmdk"
  */
 flat_offset = -1;
-ret = sscanf(p, "%10s %" SCNd64 " %10s %511s %" SCNd64,
+ret = sscanf(p, "%10s %" SCNd64 " %10s \"%511[^\"]\" %" SCNd64,
 access, §ors, type, fname, &flat_offset);
 if (ret < 4 || strcmp(access, "RW")) {
 goto next_line;

I don't know how portable %[ together with a maximum width is, because the 
manual page for sscanf() doesn't mention "max width" for "%[", but it works 
with Debian/GNU Linux Squeeze.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [BUG] qemu-1.1.2 [FIXED-BY] qcow2: Fix avail_sectors in cluster allocation code

[Philipp Hahn]

Hello Kevin, hello Michael,

On Wednesday 12 December 2012 17:54:58 Kevin Wolf wrote:
> Am 12.12.2012 15:09, schrieb Philipp Hahn:
> > Am Mittwoch 12 Dezember 2012 14:41:49 schrieb Kevin Wolf:
> >> As you can see in the commit message of that patch I was convinced that
> >> no bug did exist in practice and this was only dangerous with respect to
> >> future changes. Therefore my first question is if you're using an
> >> unmodified upstream qemu or if some backported patches are applied to
> >> it? If it's indeed unmodified, we should probably review the code once
> >> again to understand why it makes a difference.
> >
> > This were all unmodified versions directly from git between
> > "qemu-kvm-1.1.0" and "qemu-kvm-1.2.0"
> >
> > "git checkout b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c" works,
> > "git checkout b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c~1" is broken.
> > "git checkout qemu-kvm-1.1.2"  is broken,
> > "git checkout qemu-kvm-1.1.2 ; git cherry-pick
> > b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c"  works
>
> Ok, thanks for clarifying. Then I must have missed some interesting case
> while doing the patch.

I think I found your missing link:
After filling in "QCowL2Meta *m", that request ist queued:
  QLIST_INSERT_HEAD(&s->cluster_allocs, m, next_in_flight);
do prevent double allocating the same cluster for overlapping requests, which 
is checked in do_alloc_cluster_offset().

I guess that since the sector count was wrong, the overlap detection didn't 
work and the two concurrent write requests to the same cluster overwrote each 
other.

> Ideally we would find a sequence of qemu-io commands to reliably
> reproduce this.

You're the block guru, so I leave that to you (or anybody else who knows more 
about the working of qemu-io.) ;-)

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [BUG] qemu-1.1.2 [FIXED-BY] qcow2: Fix avail_sectors in cluster allocation code

[Philipp Hahn]

Hello Kevin,

On Wednesday 12 December 2012 18:29:48 Philipp Hahn wrote:
> I just re-run my "git bisect run ~/bisect.sh"  case, but it again arrived
> at that patch. I just queued another run for tonight so make sure the test
> is reliable:

The run from last night again arrived at the refecenced patch.

> > Ideally we would find a sequence of qemu-io commands to reliably
> > reproduce this. First thing worth trying would be running the current
> > qemu-iotests suite on the old versions. If we don't find it this way, I
> > guess we need to catch it with code review. I'm not sure if I can get to
> > it this week, and starting next week I'll be on vacation, so any help
> > with finding a reproducer would be appreciated.

I took a closer look at what gets corrupted; I've attached my notes.
Please notice that the partitions are not alignd properly.

If you would like to look at the full qcow2_alloc_clusters_offset trace, I can 
provide you with a link to the trace file.

BYtE
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/
# FILE=/var/cache/apt/archives/liblqr-1-0_0.4.1-1.3.201104131631_amd64.deb
# debugfs /dev/vg_ucs/rootfs -R "stat $FILE"
debugfs 1.41.12 (17-May-2010)
Inode: 1385717   Type: regularMode:  0644   Flags: 0x0
Generation: 2854262870Version: 0x
User: 0   Group: 0   Size: 38884
File ACL: 0Directory ACL: 0
Links: 1   Blockcount: 80
Fragment:  Address: 0Number: 0Size: 0
ctime: 0x50979ae9 -- Mon Nov  5 11:54:33 2012
atime: 0x50979b0d -- Mon Nov  5 11:55:09 2012
mtime: 0x4de1dbce -- Sun May 29 07:38:22 2011
Size of extra inode fields: 4
BLOCKS:
(0-9):5728401-5728410
TOTAL: 10

# BSIZE=4096
# BOFFSET=0
# dd bs=$BSIZE count=1 if=$FILE skip=$BOFFSET 2>/dev/null | md5sum
065b19ba6e9153dcc88003ea06076f9f  -

# BLOCK=5728401
# dd bs=$BSIZE count=1 if=/dev/vg_ucs/rootfs skip=$BLOCK 2>/dev/null | md5sum
065b19ba6e9153dcc88003ea06076f9f  -

# dmsetup table
vg_ucs-rootfs: 0 97910784 linear 254:3 384
# LV_SOFFSET=384
# dd bs=512c count=8 if=/dev/vda3 skip=$((BLOCK*8+LV_SOFFSET)) 2>/dev/null | 
md5sum
065b19ba6e9153dcc88003ea06076f9f  -

# fdisk -l -u /dev/vda
/dev/vda3 4739175   10485625450058540   8e  Linux LVM
# PART_SOFFSET=4739175
# dd bs=512c count=8 if=/dev/vda skip=$((BLOCK*8+LV_SOFFSET+PART_SOFFSET)) 
2>/dev/null | md5sum
065b19ba6e9153dcc88003ea06076f9f  -

# debugfs /dev/vg_ucs/rootfs -R "icheck $(seq 5728387 5728403 | tr '\n' ' ')"
debugfs 1.41.12 (17-May-2010)
Block   Inode number
5728387 1385715  
5728388 1385715  
5728389 1385715  
5728390 1385715  
5728391 1385715  
5728392 1385715  
5728393 1385715  
5728394 1385715  
5728395 1385715  
5728396 1385715  
5728397 1385716  
5728398 1385716  
5728399 1385716  
5728400 1385716  
5728401 1385717  
5728402 1385717  
5728403 1385717  

# debugfs /dev/vg_ucs/rootfs -R "ncheck 1385715 1385716 1385717"
debugfs 1.41.12 (17-May-2010)
Inode   Pathname 
1385715 
/var/cache/apt/archives/libhtml-template-perl_2.9-2.7.201104290220_all.deb
1385717 /var/cache/apt/archives/liblqr-1-0_0.4.1-1.3.201104131631_amd64.deb
1385716 
/var/cache/apt/archives/libio-socket-inet6-perl_2.65-1.1.3.201104291113_all.deb

# md5sum 
/var/cache/apt/archives/libhtml-template-perl_2.9-2.7.201104290220_all.deb 
/var/cache/apt/archives/liblqr-1-0_0.4.1-1.3.201104131631_amd64.deb 
/var/cache/apt/archives/libio-socket-inet6-perl_2.65-1.1.3.201104291113_all.deb
123f4c338bd875825d5d762e8bb48b2a  
/var/cache/apt/archives/libhtml-template-perl_2.9-2.7.201104290220_all.deb
eadbf53d7313df2560f4741fbe982008  
/var/cache/apt/archives/liblqr-1-0_0.4.1-1.3.201104131631_amd64.deb
d0c7bf2d62c125409e00e459ac94277a  
/var/cache/apt/archives/libio-socket-inet6-perl_2.65-1.1.3.201104291113_all.deb

# apt-cache show libhtml-template-perl liblqr-1-0 libio-socket-inet6-perl | 
egrep 'MD5sum|Package'
Package: libhtml-template-perl
MD5sum: 123f4c338bd875825d5d762e8bb48b2a
Package: liblqr-1-0
MD5sum: 94ccb97b38bedef97072fdcc7bce1872
Package: libio-socket-inet6-perl
MD5sum: 8f04f2da2a7d2eefb9e77bf87a0b8972



# IMAGE=/var/lib/libvirt/images/stefan_UCS-3.0-2-13.3-Kolab-Slave.qcow2
# BLOCK=5728401 BSIZE=4096 BOFFSET=0 LV_SOFFSET=384 PART_SOFFSET=4739175
# OFFSET=$((BLOCK*BSIZE + LV_SOFFSET*512 + PART_SOFFSET*512))
# echo $((OFFSET >> 16 >> 13)) $(((OFFSET >> 16) & ((1 << 13) - 1))) $(((OFFSET 
>> 0) & ((1 << 16) - 1)))
48 1836 56832
# qemu-io -c "read -v $OFFSET $BSIZE" "$IMAGE" | sed -ne 's/^\([0-9a-f]\+:\) 
/\1/p' | xxd -r -seek -$OFFSET | md5sum
065b19ba6e9153dcc88003ea06076f9

Re: [Qemu-devel] [BUG] qemu-1.1.2 [FIXED-BY] qcow2: Fix avail_sectors in cluster allocation code

[Philipp Hahn]

Hello Kevin,

Am Mittwoch 12 Dezember 2012 17:54:58 schrieb Kevin Wolf:
> Am 12.12.2012 15:09, schrieb Philipp Hahn:
> > Am Mittwoch 12 Dezember 2012 14:41:49 schrieb Kevin Wolf:
> >> As you can see in the commit message of that patch I was convinced that
> >> no bug did exist in practice and this was only dangerous with respect to
> >> future changes. Therefore my first question is if you're using an
> >> unmodified upstream qemu or if some backported patches are applied to
> >> it? If it's indeed unmodified, we should probably review the code once
> >> again to understand why it makes a difference.
> >
> > This were all unmodified versions directly from git between
> > "qemu-kvm-1.1.0" and "qemu-kvm-1.2.0"
> >
> > "git checkout b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c" works,
> > "git checkout b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c~1" is broken.
> > "git checkout qemu-kvm-1.1.2"  is broken,
> > "git checkout qemu-kvm-1.1.2 ; git cherry-pick
> > b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c"  works
>
> Ok, thanks for clarifying. Then I must have missed some interesting case
> while doing the patch.

I just re-run my "git bisect run ~/bisect.sh"  case, but it again arrived at 
that patch. I just queued another run for tonight so make sure the test is 
reliable:

# bad: [4c3e02beed9878a5f760eeceb6cd42c475cf0127] Merge tag 'v1.2.0'
# good: [bd11ac4feb54d32653e5d4eb7994bed18be0609c] fdc: fix implied seek while 
there is no media in drive
git bisect start 'qemu-kvm-1.2.0' 'qemu-kvm-1.1.0'
# good: [15ecf28f39e2b6fba359ed094770c8fa4ad8dc60] Merge tag 'v1.1.0' into 
next
git bisect good 15ecf28f39e2b6fba359ed094770c8fa4ad8dc60
# bad: [2fa5008ffd49e51540756adccf966a2fcde6e6c1] hd-geometry: Factor out 
guess_chs_for_size()
git bisect bad 2fa5008ffd49e51540756adccf966a2fcde6e6c1
# bad: [306a571a2d75e32cd2eae5486c2714b7b7792a63] hw/arm_gic: Add qdev 
property for GIC revision
git bisect bad 306a571a2d75e32cd2eae5486c2714b7b7792a63
# good: [5c6f4f178ba542358c012ca033985f73e61b8ae5] z2: Rename PXA2xxState 
variable
git bisect good 5c6f4f178ba542358c012ca033985f73e61b8ae5
# good: [833e40858cb9501c5e76b3aa345e4bb5be34385a] qcow2: remove a line of 
unnecessary code
git bisect good 833e40858cb9501c5e76b3aa345e4bb5be34385a
# bad: [0b0cb9d310edfe2b2d108f18be4f013a1e552cfd] Merge remote-tracking 
branch 'kwolf/for-anthony' into staging
git bisect bad 0b0cb9d310edfe2b2d108f18be4f013a1e552cfd
# bad: [0446919dcab51e7468f346c0a009a88632c5c5e0] qemu-iotests: COW with many 
AIO requests on the same cluster
git bisect bad 0446919dcab51e7468f346c0a009a88632c5c5e0
# good: [b75a02829dde98723dfe16fa098338cb267b28b9] Prevent disk data loss when 
closing qemu
git bisect good b75a02829dde98723dfe16fa098338cb267b28b9
# good: [c4a248a138028bee63a099410c79b428db0c4779] block: copy 
enable_write_cache in bdrv_append
git bisect good c4a248a138028bee63a099410c79b428db0c4779
# good: [6af4e9ead4ec9491259c9861b1b35f9abee24a66] qcow2: always operate 
caches in writeback mode
git bisect good 6af4e9ead4ec9491259c9861b1b35f9abee24a66
# bad: [b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c] qcow2: Fix avail_sectors in 
cluster allocation code
git bisect bad b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c
# good: [cdba7fee1daa8865bac2d69da288171fe7c21aae] qcow2: Simplify calculation 
for COW area at the end
git bisect good cdba7fee1daa8865bac2d69da288171fe7c21aae

> Ideally we would find a sequence of qemu-io commands to reliably
> reproduce this. First thing worth trying would be running the current
> qemu-iotests suite on the old versions. If we don't find it this way, I
> guess we need to catch it with code review. I'm not sure if I can get to
> it this week, and starting next week I'll be on vacation, so any help
> with finding a reproducer would be appreciated.

I'll have a look at it tommorrow.

Thank you for your fast replies and have a nice vacation in case we don't head 
from each other this week again.

BYtE
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [BUG] qemu-1.1.2 [FIXED-BY] qcow2: Fix avail_sectors in cluster allocation code

[Philipp Hahn]

Hello Kevin, 

Am Mittwoch 12 Dezember 2012 14:41:49 schrieb Kevin Wolf:
> As you can see in the commit message of that patch I was convinced that
> no bug did exist in practice and this was only dangerous with respect to
> future changes. Therefore my first question is if you're using an
> unmodified upstream qemu or if some backported patches are applied to
> it? If it's indeed unmodified, we should probably review the code once
> again to understand why it makes a difference.

This were all unmodified versions directly from git between "qemu-kvm-1.1.0" 
and "qemu-kvm-1.2.0"

"git checkout b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c" works,
"git checkout b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c~1" is broken.
"git checkout qemu-kvm-1.1.2"  is broken,
"git checkout qemu-kvm-1.1.2 ; git cherry-pick 
b7ab0fea37c15ca9e249c42c46f5c48fd1a0943c"  works

> In any case, this is the cluster allocation code. It's probably not
> related to rereading things from disk, but rather to the writeout of the
> page cache.

Yes, the problem is probably write related. But as the write "doens't explode 
with some spectacular error", I only notice the error on the following read 
by comparing md5 sums.
I just re-checked it: After a reboot the md5sums are still invalid, so I guess 
the data is corrupted on writeout.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

[Qemu-devel] [BUG] qemu-1.1.2 [FIXED-BY] qcow2: Fix avail_sectors in cluster allocation code

[Philipp Hahn]

Hello Kevin, hello Michael, hello *,

we noticed a data corruption bug in qemu-1.1.2, which will be shipped by 
Debian and our own Debian based distibution.
The corruption mostly manifests while installing large Debian package files 
and seems to be reladed to memory preasure: As long as the file is still in 
the page cache, everything looks fine, but when the file is re-read from the 
virtual hard disk using a qcow2 file backed by another qcow2 file, the file 
is corrupted: dpkg complains that the .tar.gz file inside the Debian archive 
file is corrupted and the md5sum no longer matches.

I tracked this down using "git bisect" to your patch attached below, which 
fixed this bug, so everything is fine with qemu-kvm-1.2.0.
From my reading this seems to explain our problems, since during my own 
testing during development I never used backing chains and the problem only 
showed up when my collegues started using qemu-kvm-1.1.2 with their VMs using 
backing chains.

@Kevin: Do you thinks that's a valid explanation and your patch should fix 
that problem?
I'd like to get your expertise before filing a bug with Debian and asking 
Michael to include that patch with his next stable update for 1.1.

Thanks in advance.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/
--- Begin Message ---
avail_sectors should really be the number of sectors from the start of
the allocation, not from the start of the write request.

We're lucky enough that this mistake didn't cause any real bug.
avail_sectors is only used in the intialiser of QCowL2Meta:

  .nb_available   = MIN(requested_sectors, avail_sectors),

m->nb_available in turn is only used for COW at the end of the
allocation. A COW occurs only if the request wasn't cluster aligned,
which in turn would imply that requested_sectors was less than
avail_sectors (both in the original and in the fixed version). In this
case avail_sectors is ignored and therefore the mistake doesn't cause
any misbehaviour.

Signed-off-by: Kevin Wolf 
---
 block/qcow2-cluster.c |   10 +-
 1 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c
index 98fba71..d7e0e19 100644
--- a/block/qcow2-cluster.c
+++ b/block/qcow2-cluster.c
@@ -947,8 +947,16 @@ again:
 
 /* save info needed for meta data update */
 if (nb_clusters > 0) {
+/*
+ * requested_sectors: Number of sectors from the start of the first
+ * newly allocated cluster to the end of the (possibly shortened
+ * before) write request.
+ *
+ * avail_sectors: Number of sectors from the start of the first
+ * newly allocated to the end of the last newly allocated cluster.
+ */
 int requested_sectors = n_end - keep_clusters * s->cluster_sectors;
-int avail_sectors = (keep_clusters + nb_clusters)
+int avail_sectors = nb_clusters
 << (s->cluster_bits - BDRV_SECTOR_BITS);
 
 *m = (QCowL2Meta) {
-- 
1.7.6.5


--- End Message ---


signature.asc
Description: This is a digitally signed message part.

[Qemu-devel] [BUG?] vvfat vs. pc-1.1 isa-fdc.check_media_rate=off

[Philipp Hahn]

Hello,

I tried to create a virtual VFAT floppy image on-the-fly with qemi(-kvm)-1.1.1 
using the command line argument
  -drive file=fat:floppy:"$WORK/floppy",if=floppy,index=0

Everything worked fine until I added the additional "-M pc-0.14" command line 
argument: Then I only get an empty floppy image.
I narrowed this down to the PC_COMPAT_1_0 definition in hw/pc_piix.c, which 
added
   .driver   = "isa-fdc",\
.property = "check_media_rate",\
.value= "off",\
to the compat settings. If I also add an
  -global isa-fdc.check_media_rate=on
to explicitly overwrite that setting, vvfat also works with pc-1.0 and 
earlier.

I didn't unsterstand from reading the code what that flag is supposed to do, 
so can somebody give me some help to understand what goes wrong here.

I think vvfat needs to be patched to also work with check_media_rate=off.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] 1.1.1 -> 1.1.2 migrate /managedsave issue

[Philipp Hahn]

Hello Doug,

On Tuesday 23 October 2012 22:38:47 Doug Goldstein wrote:
> >>> qemu: warning: error while loading state for instance 0x0 of device
> >>> 'ram' load of migration failed
...
> >> I diagnosed that using gdb to single step kvm until I found
> >> hw/pci.c#get_pci_config_device() returning -EINVAL.
...
> But yes I can confirm that
> vmstate_load_state() which is calling field->info->get() which is
> calling get_pci_config_device() is returning -EINVAL.
...
> Any recommendations to fix this? Or do I need to kill the saved state
> and start over?

For start try to re-get the old PXE ROM files, that is for Debian re-install 
the old etherboot-qemu or an older Version of the ipxe-qemu package. If that 
then works again, you can go further. Otherwise you need to get the exact 
index "i" where get_pci_config_device() returns -EINVAL and match that with 
the PCI spec. For short see hw/pci_regs.h.


For our Debian based UCS distribution I patched qemu(-kvm) to use the old 
image files (which are still available and installed in parallel to the new 
files) if the pc-level is 0.14 or lower (which is the version we currently 
ship; only our next release will switch to 1.1.2).

I've attached my patch FYI. That is only a work around until QEMU provides a 
real solution. If you've used an e1000, that is not enougth because some more 
PCI configuration bits (PCI_STATUS_CAP_LIST) were changed in an incompatible 
way.

If I remember correctly there was a different bug report, where changed PCI 
bits were also responsible for breaking saved states: Some PCI devices were 
changed to multi-function devices (or the other way around), which also 
breaks loading of previous saved stated. I think it was a bug in libvirt 
which started adding an explicit "multifunction=on", while the save was done 
using an implicit "multifinction=off". 
(<https://forge.univention.org/bugzilla/show_bug.cgi?id=22877#c6> in our 
German BZ)

Sincerely
Philipp Hahn
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/
diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/patches/romfile-compatibility.diff qemu-kvm-1.1.2+dfsg/debian/patches/romfile-compatibility.diff
--- qemu-kvm-1.1.2+dfsg.orig/debian/patches/romfile-compatibility.diff	1970-01-01 01:00:00.0 +0100
+++ qemu-kvm-1.1.2+dfsg/debian/patches/romfile-compatibility.diff	2012-10-08 14:25:09.225656404 +0200
@@ -0,0 +1,39 @@
+Bug #24702: Fix PXE ROM size mismatch
+
+For pc-0.14 and older use the PXE ROMs from the deprecated etherboot-qemu
+package, which have different sizes. Without the patch the virtual PC gets
+build with the current ROM sizes. Loading then fails because the stored PCI ROM
+BAR size mismatches the configured size.
+--- a/hw/pc_piix.c
 b/hw/pc_piix.c
+@@ -433,6 +433,30 @@ static QEMUMachine pc_machine_v0_15 = {
+ .driver   = "virtio-balloon-pci",\
+ .property = "event_idx",\
+ .value= "off",\
++},{\
++.driver   = "ne2000",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/rtl8029.rom",\
++},{\
++.driver   = "rtl8139",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/rtl8139.rom",\
++},{\
++.driver   = "e1000",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/e1000-82540em.rom",\
++},{\
++.driver   = "pcnet",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/pcnet32.rom",\
++},{\
++.driver   = "ne2k-isa",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/ne.rom",\
++},{\
++.driver   = "virtio-net-pci",\
++.property = "romfile",\
++.value   = "/usr/lib/etherboot/virtio-net.rom",\
+ }
+ 
+ static QEMUMachine pc_machine_v0_14 = {
diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/patches/series qemu-kvm-1.1.2+dfsg/debian/patches/series
--- qemu-kvm-1.1.2+dfsg.orig/debian/patches/series	2012-09-10 12:14:05.0 +0200
+++ qemu-kvm-1.1.2+dfsg/debian/patches/series	2012-10-08 14:24:51.497155189 +0200
@@ -8,3 +8,4 @@
 
 net-add--netdev-options-to-man-page.patch
 revert-serial-fix-retry-logic.patch
+romfile-compatibility.diff
diff -urN qemu-kvm-1.1.2+dfsg.orig/debian/control qemu-kvm-1.1.2+dfsg/debian/control
--- qemu-kvm-1.1.2+dfsg.orig/debian/

Re: [Qemu-devel] [PATCH] e1000: Don't set the Capabilities List bit

[Philipp Hahn]

Hello,

On Friday 19 October 2012 11:59:24 Philipp Hahn wrote:
> On Wednesday 21 September 2011 22:06:25 dann frazier wrote:
...
> > -/* TODO: we have no capabilities, so why is this bit set? */
> > -pci_set_word(pci_conf + PCI_STATUS, PCI_STATUS_CAP_LIST);
...
> Since cmask[PCI_STATUS=6] = PCI_STATUS_CAP_LIST=0x10 marks that bit as
> unmodifiable, the functions returns an error and aborts loading the saved
> state.
...
> Has somebody an idea how to fix this issue?

I'm no expert on PCI issues, but since e1000 drivers don't seem to much care 
about the Capability List bit, perhaps the attached patch to ignore the bit 
for e1000 would be in oder?
At least it fixes my problem.
Comments welcomed.

Sincerely
Philipp Hahn
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/
e1000: Ignore the Capabilities List bit

dd8e93799f13ef82d83c185b8e71e049452f7d40 unconditionally removed the
PCI_STATUS_CAP_LIST bit from PCI_STATUS, because the e1000 does not have
capabilities. This breaks upgrades from before qemu-0.15, because there the bit
is still set and get_pci_config_device() refused to load incompatible save
states.

Remove the Capabilities List bit from the list of compatible bits, so it is not
validated for an exact match.

Signed-off-by: Philipp Hahn 

--- a/hw/e1000.c
+++ b/hw/e1000.c
@@ -1224,6 +1224,9 @@ static int pci_e1000_init(PCIDevice *pci_dev)
 
 pci_conf = d->dev.config;
 
+/* Ignore capability bit, which was set until qemu-0.15 */
+d->dev.cmask[PCI_STATUS] &= ~PCI_STATUS_CAP_LIST;
+d->dev.wmask[PCI_STATUS] &= ~PCI_STATUS_CAP_LIST;
 /* TODO: RST# value should be 0, PCI spec 6.2.4 */
 pci_conf[PCI_CACHE_LINE_SIZE] = 0x10;
 


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [PATCH] e1000: Don't set the Capabilities List bit

[Philipp Hahn]

Hello,

On Wednesday 21 September 2011 22:06:25 dann frazier wrote:
> The Capabilities Pointer is NULL, so this bit shouldn't be set. The state
> of this bit doesn't appear to change any behavior on Linux/Windows versions
> we've tested, but it does cause Windows' PCI/PCI Express Compliance Test to
> balk.
...
> +++ b/hw/e1000.c
> @@ -1151,8 +1151,6 @@ static int pci_e1000_init(PCIDevice *pci_dev)
...
> -/* TODO: we have no capabilities, so why is this bit set? */
> -pci_set_word(pci_conf + PCI_STATUS, PCI_STATUS_CAP_LIST);

I think that change introduced a upgrade problem:
We have several VMs initially setup with qemu-0.12 and 0.14. Over the time of 
years we've created lots of snapshots, which we'd like to keep.
Now it's time to upgrade qemu(-kvm) on our hosts to qemu-1.1.2, but trying to 
load the old save states failes with
  qemu: warning: error while loading state for instance 0x0 of 
device ':00:03.0/e1000'

I traced it down to get_pci_config_device() failung for the e1000, because of 
that changed bit:
(gdb) print /x config@256 
$23 = {0x86, 0x80, 0xe, 0x10, 0x7, 0x0, 0x18, 0x0, 0x3, 0x0, 0x0, 0x2, 0x0, 
0x0, 0x0, 0x0, 0x0, 0x0, 0x2, 0xf2, 0x41, 0xc0, 0x0 , 0xf4, 
0x1a, 0x0, 0x11, 0x0,  0x0, 0x4, 0xf2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 
0x0, 0xb, 0x1, 0x0 }
(gdb) print /x *s->config@256 
$25 = {0x86, 0x80, 0xe, 0x10, 0x0, 0x0, 0x0, 0x0, 0x3, 0x0, 0x0, 0x2, 0x0, 
0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0 , 0xf4, 0x1a, 
0x0, 0x11, 0x0 , 0x1, 0x0 }

Since cmask[PCI_STATUS=6] = PCI_STATUS_CAP_LIST=0x10 marks that bit as 
unmodifiable, the functions returns an error and aborts loading the saved 
state.

Is this problem known?
Is such an upgrade of qemu supposed to work?
Has somebody an idea how to fix this issue?

Sincerely
Philipp Hahn

PS: It would be nice if the error message could indicate an error because of 
an incompatible PCI configuration. I had a very similar problem with the 
rtl8139 card, where the ROM size was changed due to the upgrade from 
etherboot to iPXE.
PPS: It's slightly confusing to get a 'warning' message starting with 'error 
loading ...'
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [PATCH] Rate limit vnc_write_pixels_generic

[Philipp Hahn]

Hello Anthony,

Am Montag 07 Mai 2012 16:19:57 schrieb Anthony Liguori:
> On 05/07/2012 08:58 AM, Philipp Hahn wrote:
> > Only print the error message once per change and also include the actual
> > unsupported color depth in bytes per pixel in the error message.
> >
> > Signed-off-by: Philipp Hahn
>
> Under what conditions are you seeing this happen?

Actually the kvm-version I'm seeing this on is a custom build from one of our 
partners with a custom RDP/RemoteFX extension they're working on. That 
versions currently spams my console with the message:

2012-05-07 14:16:21.363: starting up
vnc_write_pixels_generic: VncState color depth not supported
... 767 lines removed ...
2012-05-07 14:16:42.998: shutting down

That's 36 messages per second. So I looked up the source code and found that 
line, where I don't even see which bpp fails. At least that info I would find 
very helpful, the rate limiting was just a bonus.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

[Qemu-devel] [PATCH] Rate limit vnc_write_pixels_generic

[Philipp Hahn]

Only print the error message once per change and also include the actual
unsupported color depth in bytes per pixel in the error message.

Signed-off-by: Philipp Hahn 
---
 ui/vnc.c |8 +++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/ui/vnc.c b/ui/vnc.c
index deb9ecd..9dcff9b 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -641,6 +641,7 @@ static void vnc_write_pixels_generic(VncState *vs, struct 
PixelFormat *pf,
  void *pixels1, int size)
 {
 uint8_t buf[4];
+static uint8_t last_error_bpp = 4; /* any of the supported formats */
 
 if (pf->bytes_per_pixel == 4) {
 uint32_t *pixels = pixels1;
@@ -667,7 +668,12 @@ static void vnc_write_pixels_generic(VncState *vs, struct 
PixelFormat *pf,
 vnc_write(vs, buf, vs->clientds.pf.bytes_per_pixel);
 }
 } else {
-fprintf(stderr, "vnc_write_pixels_generic: VncState color depth not 
supported\n");
+if (last_error_bpp != pf->bytes_per_pixel) {
+last_error_bpp = pf->bytes_per_pixel;
+fprintf(stderr,
+"vnc_write_pixels_generic: VncState color depth %d not 
supported\n",
+pf->bytes_per_pixel);
+}
 }
 }
 
-- 
1.7.1

Re: [Qemu-devel] [PATCH 35/46] qemu-iotests: qcow2.py

[Philipp Hahn]

Hello,

On Thursday 05 April 2012 17:52:13 Kevin Wolf wrote:
> This adds a tool that is meant to inspect and edit qcow2 files in a
> low-level way, that wouldn't be possible with qemu-img/io, for example
> by adding yet unknown extensions or flags. This way we can test whether
> qemu deals properly with future backwards compatible extensions.

Cool. I wrote a very similar tool to dump qcow2 data some time ago when I 
tried to debug a ref-count corruption bug. I'll attach it just FYI.

BYtE
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHbe open.   fon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


qcow2.py
Description: application/python


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] Keysymbol interpretation missing in QEMU's VNC server?

[Philipp Hahn]

Hello Fabian,

On Wednesday 07 March 2012 11:53:57 Fabian Holler wrote:
> I'm not sure if I found a bug in QEMU's VNC keyboard layout mapping or
> if it's a general problem in the implemented VNC server:

Daniel P. Berrangé has written some very good blog entries about the VNC 
problem: 
<http://berrange.com/posts/2010/07/04/more-than-you-or-i-ever-wanted-to-know-about-virtual-keyboard-handling/>
They're quiet technical, but after reading them I at least concluded that VNC 
is normally just broken.
I recommend using virt-viewer, with which I have mostly no such problems.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [PATCH 3/3] stop the periodic RTC update timer

[Philipp Hahn]

Hello,

On Wednesday 11 January 2012 01:56:25 Zhang, Yang Z wrote:
> > -Original Message-
> > From: Paolo Bonzini [mailto:pbonz...@redhat.com]
> > Sent: Tuesday, January 10, 2012 5:25 PM
> >
> > >> Also, I'm not sure if the update in progress flag still works.
> > >> Clients are supposed to wait for UIP=0 before reading the RTC, and an
> > >> update is supposed to be at least 220 microseconds away when UIP=0.
> > >
> > > Hardware need a period time to update clock and it would not provide
> > > the right value during the update. So it uses UIP to notify the
> > > software doesn't believe the value if the UIP is set. For emulation,
> > > you can read RTC at any time and it always gives you the right value.
> > > So there is no need to emulate UIP.
> >
> > This is incorrect, for two reasons.  First, the UIP is in the spec, and
> > we have to implement it.  Second, reading the clock is not atomic, and
> > waiting for UIP=0 gives you 220 microseconds during which you know that
> > the read will appear atomic.
>
> For a simulator, we need to follow the spec strictly and simulate hardware
> as precisely as possible. But QEMU is a generic machine emulator and
> virtualizer. It's not a hardware simulator. If there is an easy way we can
> provide the same function, why we chose the complicated one? Also, is there
> an actual case that break with my patch?

FYI:
But you must not break existing implementations (of any (closed-source) OS), 
which depend on that behaviour of the RTC. Have a look at get_cmos_time() of 
the Xen hypervisor for example (that is the one I have been looking at at the 
past few hours), which explicitliy waits for the falling edge of UIP to get 
sub-second precision. This would break if you no longer simulate UIP.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] Correct syntax for named snapshots

[Philipp Hahn]

Hello Ottaavio,

On Wednesday 26 October 2011 18:24:01 Ottavio wrote:
> According to the latest doc file you create a named snapshot like this:
>
> qemu-img snapshot [-l | -a snapshot | -c snapshot | -d snapshot]
> base-image.img
>
> Is this other following syntax still supported or deprecated:
>
> "qemu-img create -f qcow2 -b base-image.img snapshot.img"  ?

These are two different things:
The first one creates an *internal* snapshot, which is stored internally in a 
*single* qcow2 file.
The second one creates an *external* snapshot, which creates a *new second* 
qcow2 file.

The first functionality requires a lot more work for qcow2, which can easily 
corrupt your qcow2 file if something goes wrong. Because of this QED (one 
follow-up format of Qcow) IMHO dropped support for internal snapshots.
The second variante has the drawback, that for each snapshot your get an 
additional file, which your have to manage. Deleting one file from the middle 
of such a chain breaks all following snapshots, so be careful when you do 
delete files.

And yes, both are still supported.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] Question on kvm_clock working ...

[Philipp Hahn]

Hello Al,

I just debugged a kvmclock bug, so I claim to have some knowledge in this area 
now, but please take my answer with a grain of doubt.

On Monday 12 September 2011 15:21:25 al pat wrote:
> Still seeking your guidance on this. Appreciate any pointers you may have.

You have to distiguish between the real-time-clock (RTC), which in hardware is 
a battery powered clock running even when your PC is powered off. Since it's 
slow to access, most Linux distributions read out its value once during boot 
using "hwclock --hctosys --utc" and than don't care about that clock any more 
until shutdown, when they write back the system time to the RTC 
using "... --systohc ...".
During runtime, other methods are used for time keeping: Either by counting 
regular interrupts, using the ACPI-PM clock, or the High Performance Event 
Timer (HPET), or the Time Stamp Counter (TSC) register, or ...; 
see /sys/devices/system/clocksource/clocksource0/available_clocksource for a 
list of available clock sources.

For virtual machines there is an additional clock source named "kvmclock", 
which uses the host clock and the TSC: The host exports its current system 
time (plus some configurable offset) and a snapshot value of TSC register 
when doing so. Than the guest can interpolate the current time by using the 
exported_system_time + scale * (current_TSC_value-snapshot_TSC_value). This 
kvmclock doesn't have anything to do with the RTC clock as far as I know.

Now to your problem: You should check the value 
of /sys/devices/system/clocksource/clocksource0/current_clocksource in your 
guest. If it is somethong other than kvmclock, you should if 
using "hwclock --hctosys --utc" re-synchronizes your guest clock to the host.

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/

Treffen Sie Univention auf der IT&Business vom 20. bis 22. September 2011
auf dem Gemeinschaftsstand der Open Source Business Alliance in Stuttgart in
Halle 3 Stand 3D27-7.


signature.asc
Description: This is a digitally signed message part.

[Qemu-devel] [PATCH] Fix DEBUG_* compilation of qcow2.

[Philipp Hahn]

By introducing BlockDriverState compiling qcow2 with DEBUG_ALLOC and DEBUG_EXT
defined got broken.
Define a BdrvCheckResult structure locally which is now needed as the second
argument.

Also fix qcow2_read_extensions() needing BDRVQcowState.

Signed-off-by: Philipp Hahn 
---
 block/qcow2-snapshot.c |   15 ---
 block/qcow2.c  |6 +-
 2 files changed, 17 insertions(+), 4 deletions(-)

diff --git a/block/qcow2-snapshot.c b/block/qcow2-snapshot.c
index 74823a5..a6c0861 100644
--- a/block/qcow2-snapshot.c
+++ b/block/qcow2-snapshot.c
@@ -303,7 +303,10 @@ int qcow2_snapshot_create(BlockDriverState *bs, QEMUSnapshotInfo *sn_info)
 if (qcow2_write_snapshots(bs) < 0)
 goto fail;
 #ifdef DEBUG_ALLOC
-qcow2_check_refcounts(bs);
+{
+  BdrvCheckResult result = {0};
+  qcow2_check_refcounts(bs, &result);
+}
 #endif
 return 0;
  fail:
@@ -347,7 +350,10 @@ int qcow2_snapshot_goto(BlockDriverState *bs, const char *snapshot_id)
 goto fail;
 
 #ifdef DEBUG_ALLOC
-qcow2_check_refcounts(bs);
+{
+BdrvCheckResult result = {0};
+qcow2_check_refcounts(bs, &result);
+}
 #endif
 return 0;
  fail:
@@ -384,7 +390,10 @@ int qcow2_snapshot_delete(BlockDriverState *bs, const char *snapshot_id)
 return ret;
 }
 #ifdef DEBUG_ALLOC
-qcow2_check_refcounts(bs);
+{
+BdrvCheckResult result = {0};
+qcow2_check_refcounts(bs, &result);
+}
 #endif
 return 0;
 }
diff --git a/block/qcow2.c b/block/qcow2.c
index 48e1b95..a42ece9 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -87,6 +87,7 @@ static int qcow2_read_extensions(BlockDriverState *bs, uint64_t start_offset,
 while (offset < end_offset) {
 
 #ifdef DEBUG_EXT
+BDRVQcowState *s = bs->opaque;
 /* Sanity check */
 if (offset > s->cluster_size)
 printf("qcow2_read_extension: suspicious offset %lu\n", offset);
@@ -277,7 +278,10 @@ static int qcow2_open(BlockDriverState *bs, int flags)
 }
 
 #ifdef DEBUG_ALLOC
-qcow2_check_refcounts(bs);
+{
+BdrvCheckResult result = {0};
+qcow2_check_refcounts(bs, &result);
+}
 #endif
 return ret;
 

Re: [Qemu-devel] [PATCH v2 0.15.0] qcow2: Fix L1 table size after bdrv_snapshot_goto

[Philipp Hahn]

Hello,

On Thursday 04 August 2011 18:24:59 Kevin Wolf wrote:
> When loading an internal snapshot whose L1 table is smaller than the
> current L1 table, the size of the current L1 would be shrunk to the
> snapshot's L1 size in memory, but not on disk. This lead to incorrect
> refcount updates and eventuelly to image corruption.
>
> Instead of writing the new L1 size to disk, this simply retains the bigger
> L1 size that is currently in use and makes sure that the unused part is
> zeroed.
>
> Signed-off-by: Kevin Wolf 
Tested-by: Philipp Hahn 

> Philipp, I think this should fix your corruption. Please give it a try.
Yes, the patch looks conceptually right and fixes the observed problem.

> Anthony, this must go into 0.15.
That bug is also found in 0.12.4 and 0.14.1, so if there ever shoudl be an 
update to those branches, that fix should be applied there as well.

Thanks for your fast support.

Sincerely
Philipp Hahn
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

[Qemu-devel] [BUG] Qcow2 corruption on snapshot revert

[Philipp Hahn]

Hello,

it seems like I have discovered a bug in qemu-0.14.1 which corrupts Qcow2 
image files when using internal snapshots.
I tied this both on an amd64 host running our Debian bases UCS distribution 
(using pure qemu-kvm_0.14 and pure qemu-kvm_0.14.1) and also on a pure Debian 
i386 sid installation (patched qemu-kvm_0.14.1).
I can reproducte this every time doing the following steps:
0. Create and install VM
1. Shut down VM
2. Create offline snapshot using "qemu-img snapshot -c Off image.qcow2"
3. Start VM
4. Create online snapshot using "savevm On"
5. Shut down VM
6. Revert to snapshot using "qemu-img snapshot -a Off image.qcow2"
Repeat step 6 three or more times and the Image is destroyed.

My guess is that this is a but in qemu-img, which is related to having an 
online snapshot (VM running) while reverting to an offline snapshot (VM 
stopped), because I wasn't able to reproduce this using only offline 
snapshots or only online snapshots.

I have attached a shell script to reproduce this bug on all my runs.
* It both happens with kvm-0.14.1 and also with qemu-0.14.1.
* The cache stragtegy "writethrough" vs. "writeback" is irrelevant.

The script needs an image nameded "pmh_squeeze-0.qcow.bak2" of an VM where you 
can login via serial console using "root" for login and "univention" for 
password. You can use other images as well, as long as you modify the block 
device from within the VM. You get get it from 
<http://download.univention.de/download/temp/qemu-0.14-qcow2/>

There you will also find the beginning of a small Python program 
called "qcow2.py" which can be used to dump the structure of an qcow2 file in 
some more human readable format with some consistency checking.

Related links:
<http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=e11480db7ff15a9e878f6b3cc1199b439bf7c825>
<http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=f0aa7a8b2d518c54430e4382309281b93e51981a>
<http://lists.gnu.org/archive/html/qemu-devel/2011-04/msg01376.html>
<https://forge.univention.org/bugzilla/show_bug.cgi?id=1>

Any help is appreciated.

Sincerely
Philipp Hahn
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


1_qemu_qcow2_bug.test
Description: application/shellscript


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [Request for Help] QEMU 0.15.0 change log

[Philipp Hahn]

Hello,

On Monday 25 July 2011 16:18:07 Anthony Liguori wrote:
> Hands down, 0.14.0 had the best change log of any QEMU release.  A large
> part of the success of the change log was how many people participated
> in creating it.  I'd love for us to go even further with 0.15.0.
>
> I've created a template to start with:
>
> http://wiki.qemu.org/ChangeLog/0.15

I just had to look at the 0.14 (.1) changelog and noticed that most links to 
the git-commits were currently not working, because git.qemu.org is 
(currently?) unreachable.

I took the oportunity to create the Wiki-Template 
<http://wiki.qemu.org/Template:Git> which can be used to create links to git 
commits by just writing
{{git|56a60dd6d619877e9957ba06b92d2f276e3c229d}}
instead of hard-coding the host-name and path to gitweb in each link, so 
update the host and path now can be done easily in one place instead of 
multiple places. It's currently pointing to repo.or.cz.

I already converted the 0.14 and 0.15 ChangeLogs to use that template:
 s!\[http://git\.qemu\.org/qemu\.git/commit/?id=\(\x\+\) (commit)\]!{{git|
\1}}!g  
 
 s!\[http://repo\.or\.cz/w/qemu\.git/commitdiff/\(\x\+\) (commit)\]!{{git|
\1}}!g  
 
 s!\[http://repo\.or\.cz/w/qemu\.git/commit/\(\x\+\) (commit)\]!{{git|
\1}}!g  
         

Sincerely
Philipp
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

Re: [Qemu-devel] [PATCH 00/13] QED image streaming

[Philipp Hahn]

Hello,

hopefully just a quick question...

On Tuesday 14 June 2011 20:18:18 Stefan Hajnoczi wrote:
> This patch series adds image streaming support for QED image files.  Other
> image formats can also be supported in the future.
>
> Image streaming populates the file in the background while the guest is
> running.  This makes it possible to start the guest before its image file
> has been fully provisioned.
...
> When image streaming is enabled, the unallocated regions of the image file
> are populated with the data from the backing file.  This occurs in the
> background and the guest can perform regular I/O in the meantime.  Once the
> entire backing file has been streamed, the image no longer requires a
> backing file and will drop its reference.

Does this also work for intermediate images, that is
- master image on a NFS share,
- local copy on a local drive using CoR-streaming from master,
- local instance based on local copy using CoW.

We normally have mostly independent servers without a shared storage, expect a 
global NFS share for the master images. Currently we have to copy the master 
image to the local server first, before we than can create lots of instances 
from then, which have few changed relative to the master, so we don't want 
the copying to happen there.

If haven't looked at QED yet, so thanks in advance for your answer.

Sincerely
Philipp Hahn
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.

[Qemu-devel] [BUG] Re: [2/6] loadvm: improve tests before bdrv_snapshot_goto()

[Philipp Hahn]

Hello,

Am Dienstag 03 August 2010 06:44:26 schrieb Kevin Wolf:
> From: Miguel Di Ciurcio Filho 
>
> This patch improves the resilience of the load_vmstate() function, doing
> further and better ordered tests.

This patch broke restoring not-running VMs using libvirt-0.8.7 with qemu-0.14: 
When the domain is not running while taking a snpshot, the sn.vm_state_size 
== 0:

2021} else if (sn.vm_state_size == 0) {
(gdb) print sn
$6 = {id_str = "1", '\0' , name = "pre-update-flash", '\0' 
, vm_state_size = 0, date_sec = 1302698007, date_nsec = 
711909000, 
  vm_clock_nsec = 0}

> The [old] process:
...
> - run bdrv_snapshot_goto() on devices
> - if fails, give an warning and goes to the next (not good!)
> - if fails on the VM state device, return zero (not good!)
> - check if the requested snapshot exists on the device that saves the VM
> state and the state is not zero
> - if fails return -error

Previously the qcow2 image was still reverted to the old state, so on the next 
start of the domain the qcow2 image would be in the state of the snapshot

> New behavior:
...
> - check if the requested snapshot exists on the device that saves the VM
> state and the state is not zero
> - if fails return -error
...
> - run snapshot_goto() on devices

Now the qcow2 image is not reverted and when the domain is started, it is NOT 
in the state of the snapshot.

I can't decide if this regression is an Qemu bug or libvirt should be adapted 
to this new behavior.

I found the Bug also reported with Ubuntu and created a Bug in our own German 
bugtracker:
<https://bugs.launchpad.net/qemu/+bug/726619>
<https://forge.univention.org/bugzilla/show_bug.cgi?id=1>

Sincerely
Philipp Hahn
-- 
Philipp Hahn   Open Source Software Engineer  h...@univention.de
Univention GmbHLinux for Your Businessfon: +49 421 22 232- 0
Mary-Somerville-Str.1  D-28359 Bremen fax: +49 421 22 232-99
   http://www.univention.de/


signature.asc
Description: This is a digitally signed message part.