Re: Why does the vmovdqu works for passthrough device but crashes for emulated device with "illegal operand" error (in x86_64 QEMU, -accel = kvm) ?
On Mon, Mar 4, 2024 at 6:11 PM Xu Liu wrote:
>
> Hey Alex and Paolo,
>
> I saw there is some code related to AVX
> https://elixir.bootlin.com/linux/latest/source/arch/x86/kvm/emulate.c#L668
>
> Does that mean in some special cases, kvm supports AVX instructions ?
> I didn’t really know the big picture, so just guess what it is doing .
The Avx bit was added in commit 1c11b37669a5 ("KVM: x86 emulator: add
support for vector alignment"). It is not used.
> Thanks,
> Xu
>
> > On Mar 4, 2024, at 6:39 PM, Paolo Bonzini wrote:
> >
> > !---|
> > This Message Is From an External Sender
> >
> > |---!
> >
> > On 3/4/24 22:59, Alex Williamson wrote:
> >> Since you're not seeing a KVM_EXIT_MMIO I'd guess this is more of a KVM
> >> issue than QEMU (Cc kvm list). Possibly KVM doesn't emulate vmovdqu
> >> relative to an MMIO access, but honestly I'm not positive that AVX
> >> instructions are meant to work on MMIO space. I'll let x86 KVM experts
> >> more familiar with specific opcode semantics weigh in on that.
> >
> > Indeed, KVM's instruction emulator supports some SSE MOV instructions but
> > not the corresponding AVX instructions.
> >
> > Vector instructions however do work on MMIO space, and they are used
> > occasionally especially in combination with write-combining memory. SSE
> > support was added to KVM because some operating systems used SSE
> > instructions to read and write to VRAM. However, so far we've never
> > received any reports of OSes using AVX instructions on devices that QEMU
> > can emulate (as opposed to, for example, GPU VRAM that is passed through).
> >
> > Thanks,
> >
> > Paolo
> >
> >> Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
> >> acquired through pci-sysfs or a userspace vfio-pci driver within the
> >> guest?
> >> In QEMU 4a2e242bbb30 ("memory: Don't use memcpy for ram_device
> >> regions") we resolved an issue[1] where QEMU itself was doing a memcpy()
> >> to assigned device MMIO space resulting in breaking functionality of
> >> the device. IIRC memcpy() was using an SSE instruction that didn't
> >> fault, but didn't work correctly relative to MMIO space either.
> >> So I also wouldn't rule out that the program isn't inherently
> >> misbehaving by using memcpy() and thereby ignoring the nature of the
> >> device MMIO access semantics. Thanks,
> >> Alex
> >> [1]https://bugs.launchpad.net/qemu/+bug/1384892
> >
>
Re: Why does the vmovdqu works for passthrough device but crashes for emulated device with "illegal operand" error (in x86_64 QEMU, -accel = kvm) ?
Hey Alex and Paolo,
I saw there is some code related to AVX
https://elixir.bootlin.com/linux/latest/source/arch/x86/kvm/emulate.c#L668
Does that mean in some special cases, kvm supports AVX instructions ?
I didn’t really know the big picture, so just guess what it is doing .
Thanks,
Xu
> On Mar 4, 2024, at 6:39 PM, Paolo Bonzini wrote:
>
> !---|
> This Message Is From an External Sender
>
> |---!
>
> On 3/4/24 22:59, Alex Williamson wrote:
>> Since you're not seeing a KVM_EXIT_MMIO I'd guess this is more of a KVM
>> issue than QEMU (Cc kvm list). Possibly KVM doesn't emulate vmovdqu
>> relative to an MMIO access, but honestly I'm not positive that AVX
>> instructions are meant to work on MMIO space. I'll let x86 KVM experts
>> more familiar with specific opcode semantics weigh in on that.
>
> Indeed, KVM's instruction emulator supports some SSE MOV instructions but not
> the corresponding AVX instructions.
>
> Vector instructions however do work on MMIO space, and they are used
> occasionally especially in combination with write-combining memory. SSE
> support was added to KVM because some operating systems used SSE instructions
> to read and write to VRAM. However, so far we've never received any reports
> of OSes using AVX instructions on devices that QEMU can emulate (as opposed
> to, for example, GPU VRAM that is passed through).
>
> Thanks,
>
> Paolo
>
>> Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
>> acquired through pci-sysfs or a userspace vfio-pci driver within the
>> guest?
>> In QEMU 4a2e242bbb30 ("memory: Don't use memcpy for ram_device
>> regions") we resolved an issue[1] where QEMU itself was doing a memcpy()
>> to assigned device MMIO space resulting in breaking functionality of
>> the device. IIRC memcpy() was using an SSE instruction that didn't
>> fault, but didn't work correctly relative to MMIO space either.
>> So I also wouldn't rule out that the program isn't inherently
>> misbehaving by using memcpy() and thereby ignoring the nature of the
>> device MMIO access semantics. Thanks,
>> Alex
>> [1]https://bugs.launchpad.net/qemu/+bug/1384892
>
Re: Why does the vmovdqu works for passthrough device but crashes for emulated device with "illegal operand" error (in x86_64 QEMU, -accel = kvm) ?
Hey Paolo,
Thanks for confirming that the AVX is not supported for MMIO space.
So for the emulated device, basically I have to force the compiler avoid using
vmovdqu .
I am curious about how kvm emulates those instructions. Do you mind sharing
some related code pointer ?
Thanks,
Xu
On Mar 4, 2024, at 6:39 PM, Paolo Bonzini wrote:
!---|
This Message Is From an External Sender
|---!
On 3/4/24 22:59, Alex Williamson wrote:
Since you're not seeing a KVM_EXIT_MMIO I'd guess this is more of a KVM
issue than QEMU (Cc kvm list). Possibly KVM doesn't emulate vmovdqu
relative to an MMIO access, but honestly I'm not positive that AVX
instructions are meant to work on MMIO space. I'll let x86 KVM experts
more familiar with specific opcode semantics weigh in on that.
Indeed, KVM's instruction emulator supports some SSE MOV instructions but not
the corresponding AVX instructions.
Vector instructions however do work on MMIO space, and they are used
occasionally especially in combination with write-combining memory. SSE
support was added to KVM because some operating systems used SSE instructions
to read and write to VRAM. However, so far we've never received any reports of
OSes using AVX instructions on devices that QEMU can emulate (as opposed to,
for example, GPU VRAM that is passed through).
Thanks,
Paolo
Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
acquired through pci-sysfs or a userspace vfio-pci driver within the
guest?
In QEMU 4a2e242bbb30 ("memory: Don't use memcpy for ram_device
regions") we resolved an issue[1] where QEMU itself was doing a memcpy()
to assigned device MMIO space resulting in breaking functionality of
the device. IIRC memcpy() was using an SSE instruction that didn't
fault, but didn't work correctly relative to MMIO space either.
So I also wouldn't rule out that the program isn't inherently
misbehaving by using memcpy() and thereby ignoring the nature of the
device MMIO access semantics. Thanks,
Alex
[1]https://bugs.launchpad.net/qemu/+bug/1384892
Re: Why does the vmovdqu works for passthrough device but crashes for emulated device with "illegal operand" error (in x86_64 QEMU, -accel = kvm) ?
Hey Alex,
Thanks for the detailed explanation!
First answer your question:
Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
acquired through pci-sysfs or a userspace vfio-pci driver within the
guest?
My program is using a usersapcee vfio-pci driver for both emulated device and
the assigned device within the guest OS.
The program is in Rust , and I am using std::ptr::write_volatile to do the
memory copy.
I tried "x-no-mmap=on” for the assigned device, as you mentioned it did behave
same as the emulated device.
I am also suspecting this
Possibly KVM doesn't emulate vmovdqu
relative to an MMIO access, but honestly I'm not positive that AVX
instructions are meant to work on MMIO space.
Do you have some suggestions to verify this ? Or some code pointer to check on
this ?
Thanks ,
Xu
On Mar 4, 2024, at 4:59 PM, Alex Williamson wrote:
!---|
This Message Is From an External Sender
|---!
On Sun, 3 Mar 2024 22:20:33 +
Xu Liu mailto:li...@meta.com>> wrote:
Hello,
Recently I am running my programs in QEMU (x86_64) with “-accel=kvm”.
The QEMU version is 6.0.0.
I run my programs in two ways:
1. I pass through my device through vfio-pci to QEMU, this way
works well.
2. I write an emulated PCI device for QEMU, and run my programs on
the emulated PCI device. This crashes when the code try to do memory
copy to PCI device when the data length is longer than 16 bytes.
While the passthrough device works well for the same situation.
After dump the assembly code. I noticed when the data is <= 16
bytes, the mov assembly code is chosen, and it works well.
When the data is > 16 bytes, the vmovdqu assembly code is chosen,
and it crashes with “illegal operand”.
Given the code and data are exactly same for both passthrough device
and emulated device. I am curious about why this happens.
After turn on kernel trace for kvm by echo kvm:*
/sys/kernel/debug/tracing/set_event And rerun the QEMU and my code
for both passthrough device and emulated device, I noticed that:
1) for passthrough device, I didn’t see any trace events related to
my gva and gpa. This makes me think that the memory copy to PCI
device went through different code path . It is handled by the guest
OS without exit to VMX.
2) for emulated device, if I use compiler flag
target-feature=-avx,-avx2 to force compiler use mov assembly code,
I can see the memory copy goes through the KVM_EXIT_MMIO, and
everything works well. if I don’t force the compiler use mov , the
compiler just chooses the vmovdqu , which just crash the programs,
and no KVM_EXIT_MMIO related to my memory copy appears in the trace
events. Looks like the guest OS handles the crash.
Any clue about why the vmovdqu works for passthrough device but not
work for emulated device.
For an assigned device, the device MMIO space will be directly mapped
into the VM address space (assuming the PCI BAR is at least PAGE_SIZE),
so there's no emulation of the access. You can disable this with the
x-no-mmap=on option for the vfio-pci device, where then I'd guess this
behaves the same as your emulated device (assuming we really don't
reach QEMU for the access).
Since you're not seeing a KVM_EXIT_MMIO I'd guess this is more of a KVM
issue than QEMU (Cc kvm list). Possibly KVM doesn't emulate vmovdqu
relative to an MMIO access, but honestly I'm not positive that AVX
instructions are meant to work on MMIO space. I'll let x86 KVM experts
more familiar with specific opcode semantics weigh in on that.
Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
acquired through pci-sysfs or a userspace vfio-pci driver within the
guest?
In QEMU 4a2e242bbb30 ("memory: Don't use memcpy for ram_device
regions") we resolved an issue[1] where QEMU itself was doing a memcpy()
to assigned device MMIO space resulting in breaking functionality of
the device. IIRC memcpy() was using an SSE instruction that didn't
fault, but didn't work correctly relative to MMIO space either.
So I also wouldn't rule out that the program isn't inherently
misbehaving by using memcpy() and thereby ignoring the nature of the
device MMIO access semantics. Thanks,
Alex
[1]https://bugs.launchpad.net/qemu/+bug/1384892
Re: Why does the vmovdqu works for passthrough device but crashes for emulated device with "illegal operand" error (in x86_64 QEMU, -accel = kvm) ?
Hey Alex,
Thanks for the detailed explanation!
First answer your question:
Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
acquired through pci-sysfs or a userspace vfio-pci driver within the
guest?
My program is using a usersapcee vfio-pci driver for both emulated device and
the assigned device within the guest OS.
The program is in Rust , and I am using
https://doc.rust-lang.org/std/ptr/fn.write_volatile.html to do the memory copy.
I tried "x-no-mmap=on” for the assigned device, as you mentioned it did behave
same as the emulated device.
I am also suspecting this
Possibly KVM doesn't emulate vmovdqu
relative to an MMIO access, but honestly I'm not positive that AVX
instructions are meant to work on MMIO space.
Do you have some suggestions to verify this ? Or some code pointer to check on
this ?
Thanks ,
Xu
On Mar 4, 2024, at 4:59 PM, Alex Williamson wrote:
!---|
This Message Is From an External Sender
|---!
On Sun, 3 Mar 2024 22:20:33 +
Xu Liu mailto:li...@meta.com>> wrote:
Hello,
Recently I am running my programs in QEMU (x86_64) with “-accel=kvm”.
The QEMU version is 6.0.0.
I run my programs in two ways:
1. I pass through my device through vfio-pci to QEMU, this way
works well.
2. I write an emulated PCI device for QEMU, and run my programs on
the emulated PCI device. This crashes when the code try to do memory
copy to PCI device when the data length is longer than 16 bytes.
While the passthrough device works well for the same situation.
After dump the assembly code. I noticed when the data is <= 16
bytes, the mov assembly code is chosen, and it works well.
When the data is > 16 bytes, the vmovdqu assembly code is chosen,
and it crashes with “illegal operand”.
Given the code and data are exactly same for both passthrough device
and emulated device. I am curious about why this happens.
After turn on kernel trace for kvm by echo kvm:*
/sys/kernel/debug/tracing/set_event And rerun the QEMU and my code
for both passthrough device and emulated device, I noticed that:
1) for passthrough device, I didn’t see any trace events related to
my gva and gpa. This makes me think that the memory copy to PCI
device went through different code path . It is handled by the guest
OS without exit to VMX.
2) for emulated device, if I use compiler flag
target-feature=-avx,-avx2 to force compiler use mov assembly code,
I can see the memory copy goes through the KVM_EXIT_MMIO, and
everything works well. if I don’t force the compiler use mov , the
compiler just chooses the vmovdqu , which just crash the programs,
and no KVM_EXIT_MMIO related to my memory copy appears in the trace
events. Looks like the guest OS handles the crash.
Any clue about why the vmovdqu works for passthrough device but not
work for emulated device.
For an assigned device, the device MMIO space will be directly mapped
into the VM address space (assuming the PCI BAR is at least PAGE_SIZE),
so there's no emulation of the access. You can disable this with the
x-no-mmap=on option for the vfio-pci device, where then I'd guess this
behaves the same as your emulated device (assuming we really don't
reach QEMU for the access).
Since you're not seeing a KVM_EXIT_MMIO I'd guess this is more of a KVM
issue than QEMU (Cc kvm list). Possibly KVM doesn't emulate vmovdqu
relative to an MMIO access, but honestly I'm not positive that AVX
instructions are meant to work on MMIO space. I'll let x86 KVM experts
more familiar with specific opcode semantics weigh in on that.
Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
acquired through pci-sysfs or a userspace vfio-pci driver within the
guest?
In QEMU 4a2e242bbb30 ("memory: Don't use memcpy for ram_device
regions") we resolved an issue[1] where QEMU itself was doing a memcpy()
to assigned device MMIO space resulting in breaking functionality of
the device. IIRC memcpy() was using an SSE instruction that didn't
fault, but didn't work correctly relative to MMIO space either.
So I also wouldn't rule out that the program isn't inherently
misbehaving by using memcpy() and thereby ignoring the nature of the
device MMIO access semantics. Thanks,
Alex
[1]https://bugs.launchpad.net/qemu/+bug/1384892
Re: Why does the vmovdqu works for passthrough device but crashes for emulated device with "illegal operand" error (in x86_64 QEMU, -accel = kvm) ?
On 3/4/24 22:59, Alex Williamson wrote:
Since you're not seeing a KVM_EXIT_MMIO I'd guess this is more of a KVM
issue than QEMU (Cc kvm list). Possibly KVM doesn't emulate vmovdqu
relative to an MMIO access, but honestly I'm not positive that AVX
instructions are meant to work on MMIO space. I'll let x86 KVM experts
more familiar with specific opcode semantics weigh in on that.
Indeed, KVM's instruction emulator supports some SSE MOV instructions
but not the corresponding AVX instructions.
Vector instructions however do work on MMIO space, and they are used
occasionally especially in combination with write-combining memory. SSE
support was added to KVM because some operating systems used SSE
instructions to read and write to VRAM. However, so far we've never
received any reports of OSes using AVX instructions on devices that QEMU
can emulate (as opposed to, for example, GPU VRAM that is passed through).
Thanks,
Paolo
Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
acquired through pci-sysfs or a userspace vfio-pci driver within the
guest?
In QEMU 4a2e242bbb30 ("memory: Don't use memcpy for ram_device
regions") we resolved an issue[1] where QEMU itself was doing a memcpy()
to assigned device MMIO space resulting in breaking functionality of
the device. IIRC memcpy() was using an SSE instruction that didn't
fault, but didn't work correctly relative to MMIO space either.
So I also wouldn't rule out that the program isn't inherently
misbehaving by using memcpy() and thereby ignoring the nature of the
device MMIO access semantics. Thanks,
Alex
[1]https://bugs.launchpad.net/qemu/+bug/1384892
Usb passthrough for windows host
Hi Everyone, I tried to have usb cam working, however even if the console report it with info usb. Linux quest don't seem to see the web cam Any clues? Command Line: qemu-system-x86_64 -vga qxl -M q35 -device AC97 -smp 2,sockets=2,cores=1,threads=1 -hda .\file\Linux.img -m 2G -usb -device usb-kbd,id=mykbd -usb -device usb-tablet,id=mymouse -usb -device usb-host,vendorid=0x041e,productid=0x401f,id=myusbcam -accel tcg
Re: Why does the vmovdqu works for passthrough device but crashes for emulated device with "illegal operand" error (in x86_64 QEMU, -accel = kvm) ?
On Sun, 3 Mar 2024 22:20:33 +
Xu Liu wrote:
> Hello,
>
> Recently I am running my programs in QEMU (x86_64) with “-accel=kvm”.
> The QEMU version is 6.0.0.
>
> I run my programs in two ways:
>
> 1. I pass through my device through vfio-pci to QEMU, this way
> works well.
>
> 2. I write an emulated PCI device for QEMU, and run my programs on
> the emulated PCI device. This crashes when the code try to do memory
> copy to PCI device when the data length is longer than 16 bytes.
> While the passthrough device works well for the same situation.
>
>
> After dump the assembly code. I noticed when the data is <= 16
> bytes, the mov assembly code is chosen, and it works well.
>
> When the data is > 16 bytes, the vmovdqu assembly code is chosen,
> and it crashes with “illegal operand”.
>
> Given the code and data are exactly same for both passthrough device
> and emulated device. I am curious about why this happens.
>
> After turn on kernel trace for kvm by echo kvm:*
> /sys/kernel/debug/tracing/set_event And rerun the QEMU and my code
> for both passthrough device and emulated device, I noticed that:
>
> 1) for passthrough device, I didn’t see any trace events related to
> my gva and gpa. This makes me think that the memory copy to PCI
> device went through different code path . It is handled by the guest
> OS without exit to VMX.
>
> 2) for emulated device, if I use compiler flag
> target-feature=-avx,-avx2 to force compiler use mov assembly code,
> I can see the memory copy goes through the KVM_EXIT_MMIO, and
> everything works well. if I don’t force the compiler use mov , the
> compiler just chooses the vmovdqu , which just crash the programs,
> and no KVM_EXIT_MMIO related to my memory copy appears in the trace
> events. Looks like the guest OS handles the crash.
>
>
> Any clue about why the vmovdqu works for passthrough device but not
> work for emulated device.
For an assigned device, the device MMIO space will be directly mapped
into the VM address space (assuming the PCI BAR is at least PAGE_SIZE),
so there's no emulation of the access. You can disable this with the
x-no-mmap=on option for the vfio-pci device, where then I'd guess this
behaves the same as your emulated device (assuming we really don't
reach QEMU for the access).
Since you're not seeing a KVM_EXIT_MMIO I'd guess this is more of a KVM
issue than QEMU (Cc kvm list). Possibly KVM doesn't emulate vmovdqu
relative to an MMIO access, but honestly I'm not positive that AVX
instructions are meant to work on MMIO space. I'll let x86 KVM experts
more familiar with specific opcode semantics weigh in on that.
Is your "program" just doing a memcpy() with an mmap() of the PCI BAR
acquired through pci-sysfs or a userspace vfio-pci driver within the
guest?
In QEMU 4a2e242bbb30 ("memory: Don't use memcpy for ram_device
regions") we resolved an issue[1] where QEMU itself was doing a memcpy()
to assigned device MMIO space resulting in breaking functionality of
the device. IIRC memcpy() was using an SSE instruction that didn't
fault, but didn't work correctly relative to MMIO space either.
So I also wouldn't rule out that the program isn't inherently
misbehaving by using memcpy() and thereby ignoring the nature of the
device MMIO access semantics. Thanks,
Alex
[1]https://bugs.launchpad.net/qemu/+bug/1384892
Re: Xilinx zcu102 machine is absent in list of supported machines
On Mon, 4 Mar 2024 at 11:20, Vadim Idelchuk via wrote: > For building the products I run the following: > > ./configure > > make > > > > I tried these sequence number of times with the same result - machine type > xlnx-zcu102 is absent in the list of supported machines What is the exact command line you are running to check that the machine is not present? Are you definitely running qemu-system-aarch64 (not qemu-system-arm !), and is it definitely the version you just built? -- PMM
RE: Xilinx zcu102 machine is absent in list of supported machines
For building the products I run the following: ./configure make I tried these sequence number of times with the same result - machine type xlnx-zcu102 is absent in the list of supported machines Best regards Vadim Idelchuk vadim.idelc...@mail.ru -- This email has been checked for viruses by Avast antivirus software. www.avast.com
Emulating multicore processor on QEMU in a bare metal environment
Hello, As suggested, QEMU supports few dual core machines. So, I want to emulate mps2-an521 Or mps3-an24 on QEMU, and implement inter core communication and synchronization. The application is developed in bare metal ARM environment. What are the approaches to do this? I have a basic knowledge on openMP and openAMP, but I cannot make out if it is useful for my use case. Can you please guide me on how to proceed? Thanks and regards, Ankita Suman
