Re: [Xen-devel] Branch Trace Storage for guests and VPMUinitialization
On 02/26/2015 03:56 AM, Dietmar Hahn wrote: Am Mittwoch 25 Februar 2015, 11:31:31 schrieb Boris Ostrovsky: On 02/25/2015 10:12 AM, kevin.ma...@gdata.de wrote: -Ursprüngliche Nachricht- Von: Boris Ostrovsky [mailto:boris.ostrov...@oracle.com] Gesendet: Dienstag, 24. Februar 2015 18:13 An: Mayer, Kevin; xen-devel@lists.xen.org Betreff: Re: [Xen-devel] Branch Trace Storage for guests and VPMU initialization On 02/24/2015 10:27 AM, kevin.ma...@gdata.de wrote: Hi guys I`m trying to set up the BTS so that I can log the branches taken in the guest using Xen 4.4.1 with a WinXP SP3 guest on a Core i7 Sandy Bridge. I added the vpmu=bts boot parameter to my grub2 configuration and extended the libxl,libxc,domctl,… with an own command so that I can trigger the activation of the BTS whenever I want. I am not sure why you are doing all these changes to Xen code. BTS is supposed to be managed from the guest. For example, a Fedora HVM guest will produce this: [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf record -e branches:u -c 1 -d sleep 1 [ perf record: Woken up 3838 times to write data ] [ perf record: Captured and wrote 0.704 MB perf.data (~30756 samples) ] [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf script -f ip,addr,sym,dso,symoff --show-kernel-path 8167c347 native_irq_return_iret+0x0 (/proc/kcore) = 328c001590 [unknown] (/proc/kcore) 8167c347 native_irq_return_iret+0x0 (/proc/kcore) = 328c001590 [unknown] ([unknown]) 328c001593 [unknown] ([unknown]) = 328c004b70 [unknown] ([unknown]) ... I want to be able to log the taken branches (of the guest) without the need to modify the guest at all. This means I have to do all the logic in the hypervisor, or am I wrong? In that case, yes. But then you have to make sure that at least * you don't load guest's VPMU (or, at least, BTS-related registers) on context switch But you need to modify PMU registers when switching to/from the guest context to get PMU running. I was thinking that all BTS stuff can be controlled from dom0 and so we can use dom0's version of these registers. I didn't realize that DS_AREA would have to be accessed in guest's address space (and that DEBUGCTL is loaded from VMCS). Which is what I think I said in response to this message (which didn't show up on the list because Kevin accidentally dropped xen-devel). -boris I didn't think of using the VPMU stuff with modifying the context from outside the guest. * You don't send the interrupt to the guest (meaning that you will need to somehow inform dom0 of the BTS interrupt) and probably more. Essentially, you want dom0 to profile the guest. I have been working on patches that would allow that but they are still under review. In this command I do the following: I set up the memory region for the BTS Buffer and the DS Buffer Management Area using xzalloc_bytes I don't think you should be allocating BTS buffers in the hypervisor, they are in guest's memory. I agree. As I said I think this is where my main problem is at the moment. Is there any way I can allocate memory in the hypervisor in a way the guest can access it? I am not sure this is what you want since you seem to *not* want the guest to process the samples, right? But yes, you can. E.g. something like what map_vcpu_info() does. (I have no idea how you'd do this from Windows.) The DS buffer has to be mapped within the guests address space so the CPU running in guest context can access this area. Otherwise you get this triple fault. So I would think you need a mixture of writing some stuff in Windows and patching the hypervisor. Dietmar. Of course the guest must not be able to use this memory in its normal operations but just for BTS. Is this even possible? I am rather confused at the moment. :-D Then I write the pointer to the BTS Buffer into the DS Buffer Management Area at +0x0 and +0x8 (BTS Buffer Base and BTS Index) When I use vmx_msr_write_intercept to store the value in MSR_IA32_DS_AREA the host reboots (my idea is he tries to access a vpmu-struct that isn´t there in the current vcpu and panics). Who is trying to write to MSR_IA32_DS_AREA? The guest or dom0? I thought you said that you want dom0 to do sampling. Or are you trying to setup DS area from your guest and control it from dom0? I am somewhat confused. Can you post hypervisor log? (hard to say how helpful it will be without seeing your code changes though) Right after enabling the BTS I get a triple fault. hvm.c:1357:d2 Triple fault on VCPU0 - invoking HVM shutdown action 1. That's not host reboot, this is your guest dying. When I use a modified version of vmx_msr_write_intercept I don’t get any crashes as long as I don’t enable BTS and TR in the GUEST_IA32_DEBUGCTL (BTR works). When I enable the BTS (and TR) the guest crashes. I suppose he gets killed by the hypervisor for accessing forbidden memory. Possibly because DS area point to
Re: [Xen-devel] Branch Trace Storage for guests and VPMUinitialization
Am Mittwoch 25 Februar 2015, 11:31:31 schrieb Boris Ostrovsky: On 02/25/2015 10:12 AM, kevin.ma...@gdata.de wrote: -Ursprüngliche Nachricht- Von: Boris Ostrovsky [mailto:boris.ostrov...@oracle.com] Gesendet: Dienstag, 24. Februar 2015 18:13 An: Mayer, Kevin; xen-devel@lists.xen.org Betreff: Re: [Xen-devel] Branch Trace Storage for guests and VPMU initialization On 02/24/2015 10:27 AM, kevin.ma...@gdata.de wrote: Hi guys I`m trying to set up the BTS so that I can log the branches taken in the guest using Xen 4.4.1 with a WinXP SP3 guest on a Core i7 Sandy Bridge. I added the vpmu=bts boot parameter to my grub2 configuration and extended the libxl,libxc,domctl,… with an own command so that I can trigger the activation of the BTS whenever I want. I am not sure why you are doing all these changes to Xen code. BTS is supposed to be managed from the guest. For example, a Fedora HVM guest will produce this: [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf record -e branches:u -c 1 -d sleep 1 [ perf record: Woken up 3838 times to write data ] [ perf record: Captured and wrote 0.704 MB perf.data (~30756 samples) ] [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf script -f ip,addr,sym,dso,symoff --show-kernel-path 8167c347 native_irq_return_iret+0x0 (/proc/kcore) = 328c001590 [unknown] (/proc/kcore) 8167c347 native_irq_return_iret+0x0 (/proc/kcore) = 328c001590 [unknown] ([unknown]) 328c001593 [unknown] ([unknown]) = 328c004b70 [unknown] ([unknown]) ... I want to be able to log the taken branches (of the guest) without the need to modify the guest at all. This means I have to do all the logic in the hypervisor, or am I wrong? In that case, yes. But then you have to make sure that at least * you don't load guest's VPMU (or, at least, BTS-related registers) on context switch But you need to modify PMU registers when switching to/from the guest context to get PMU running. I didn't think of using the VPMU stuff with modifying the context from outside the guest. * You don't send the interrupt to the guest (meaning that you will need to somehow inform dom0 of the BTS interrupt) and probably more. Essentially, you want dom0 to profile the guest. I have been working on patches that would allow that but they are still under review. In this command I do the following: I set up the memory region for the BTS Buffer and the DS Buffer Management Area using xzalloc_bytes I don't think you should be allocating BTS buffers in the hypervisor, they are in guest's memory. I agree. As I said I think this is where my main problem is at the moment. Is there any way I can allocate memory in the hypervisor in a way the guest can access it? I am not sure this is what you want since you seem to *not* want the guest to process the samples, right? But yes, you can. E.g. something like what map_vcpu_info() does. (I have no idea how you'd do this from Windows.) The DS buffer has to be mapped within the guests address space so the CPU running in guest context can access this area. Otherwise you get this triple fault. So I would think you need a mixture of writing some stuff in Windows and patching the hypervisor. Dietmar. Of course the guest must not be able to use this memory in its normal operations but just for BTS. Is this even possible? I am rather confused at the moment. :-D Then I write the pointer to the BTS Buffer into the DS Buffer Management Area at +0x0 and +0x8 (BTS Buffer Base and BTS Index) When I use vmx_msr_write_intercept to store the value in MSR_IA32_DS_AREA the host reboots (my idea is he tries to access a vpmu-struct that isn´t there in the current vcpu and panics). Who is trying to write to MSR_IA32_DS_AREA? The guest or dom0? I thought you said that you want dom0 to do sampling. Or are you trying to setup DS area from your guest and control it from dom0? I am somewhat confused. Can you post hypervisor log? (hard to say how helpful it will be without seeing your code changes though) Right after enabling the BTS I get a triple fault. hvm.c:1357:d2 Triple fault on VCPU0 - invoking HVM shutdown action 1. That's not host reboot, this is your guest dying. When I use a modified version of vmx_msr_write_intercept I don’t get any crashes as long as I don’t enable BTS and TR in the GUEST_IA32_DEBUGCTL (BTR works). When I enable the BTS (and TR) the guest crashes. I suppose he gets killed by the hypervisor for accessing forbidden memory. Possibly because DS area point to hypervisor memory. Having said all this, I am not sure how well BTS works. You did notice this in the hypervisor log: (XEN) ** (XEN) ** WARNING: Emulation of BTS Feature is switched on ** (XEN) ** Using this processor
Re: [Xen-devel] Branch Trace Storage for guests and VPMUinitialization
-Ursprüngliche Nachricht-
Von: Boris Ostrovsky [mailto:boris.ostrov...@oracle.com]
Gesendet: Dienstag, 24. Februar 2015 18:13
An: Mayer, Kevin; xen-devel@lists.xen.org
Betreff: Re: [Xen-devel] Branch Trace Storage for guests and VPMU
initialization
On 02/24/2015 10:27 AM, kevin.ma...@gdata.de wrote:
Hi guys
I`m trying to set up the BTS so that I can log the branches taken in
the guest using Xen 4.4.1 with a WinXP SP3 guest on a Core i7 Sandy
Bridge.
I added the vpmu=bts boot parameter to my grub2 configuration and
extended the libxl,libxc,domctl,… with an own command so that I can
trigger the activation of the BTS whenever I want.
I am not sure why you are doing all these changes to Xen code. BTS is
supposed to be managed from the guest. For example, a Fedora HVM guest
will produce this:
[root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf record -e
branches:u -c 1 -d sleep 1 [ perf record: Woken up 3838 times to write data ]
[
perf record: Captured and wrote 0.704 MB perf.data (~30756 samples) ]
[root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf script -f
ip,addr,sym,dso,symoff --show-kernel-path
8167c347 native_irq_return_iret+0x0 (/proc/kcore) =
328c001590 [unknown] (/proc/kcore)
8167c347 native_irq_return_iret+0x0 (/proc/kcore) =
328c001590 [unknown] ([unknown])
328c001593 [unknown] ([unknown]) = 328c004b70 [unknown]
([unknown])
...
I want to be able to log the taken branches (of the guest) without the need to
modify the guest at all.
This means I have to do all the logic in the hypervisor, or am I wrong?
In this command I do the following:
I set up the memory region for the BTS Buffer and the DS Buffer
Management Area using xzalloc_bytes
I don't think you should be allocating BTS buffers in the hypervisor, they are
in guest's memory.
I agree. As I said I think this is where my main problem is at the moment.
Is there any way I can allocate memory in the hypervisor in a way the guest can
access it?
Of course the guest must not be able to use this memory in its normal
operations but just for BTS.
Is this even possible? I am rather confused at the moment. :-D
Then I write the pointer to the BTS Buffer into the DS Buffer
Management Area at +0x0 and +0x8 (BTS Buffer Base and BTS Index)
When I use vmx_msr_write_intercept to store the value in
MSR_IA32_DS_AREA the host reboots (my idea is he tries to access a
vpmu-struct that isn´t there in the current vcpu and panics).
Can you post hypervisor log? (hard to say how helpful it will be without
seeing your code changes though)
Right after enabling the BTS I get a triple fault.
hvm.c:1357:d2 Triple fault on VCPU0 - invoking HVM shutdown action 1.
When I use a modified version of vmx_msr_write_intercept I don’t get
any crashes as long as I don’t enable BTS and TR in the
GUEST_IA32_DEBUGCTL (BTR works). When I enable the BTS (and TR) the
guest crashes. I suppose he gets killed by the hypervisor for
accessing forbidden memory.
Possibly because DS area point to hypervisor memory.
Having said all this, I am not sure how well BTS works. You did notice
this in the hypervisor log:
(XEN) **
(XEN) ** WARNING: Emulation of BTS Feature is switched on **
(XEN) ** Using this processor feature in a virtualized **
(XEN) ** environment is not 100% safe. **
(XEN) ** Setting the DS buffer address with wrong values **
(XEN) ** may lead to hypervisor hangs or crashes. **
(XEN) ** It is NOT recommended for production use! **
(XEN) **
Yes, I saw that. It doesn’t state that BTS is not working at all, just that it
is not that safe to use.
As I understand it as long as I set the DS buffer address correctly I should be
fine, right?
Since I don’t want to use for production that is fine with me. At least for now.
Kevin
-boris
The modified version of vmx_msr_write_intercept takes a vcpu-struct as
a parameter and uses this instead of the current vcpu.
Instead of
staticint vmx_msr_write_intercept(unsigned int msr, uint64_t
msr_content)
{
struct vcpu *v = current;
I just have
staticint own_vmx_msr_write_intercept(unsigned int msr, uint64_t
msr_content, struct vcpu *v)
I get this vcpu by d-vcpu[0] as I have limited my guest domain to one
vcpu atm.
Of course I also use similarly modified version of the called
functions(vpmu_do_wrmsr,…).
I´m pretty sure that my problem is with a wrong scope/usage of the
vcpus/memory, but I have no idea how to fix this.
I can see a potential problem with the memory allocation (in the host)
into which the cpu in guest-mode is supposed to write.
Or maybe I got the principle of a vcpu/vpmu all wrong.
Since I couldn’t find any project that uses the BTS for the guest, I
am wondering if anyone has ever
Re: [Xen-devel] Branch Trace Storage for guests and VPMUinitialization
On 02/25/2015 10:12 AM, kevin.ma...@gdata.de wrote: -Ursprüngliche Nachricht- Von: Boris Ostrovsky [mailto:boris.ostrov...@oracle.com] Gesendet: Dienstag, 24. Februar 2015 18:13 An: Mayer, Kevin; xen-devel@lists.xen.org Betreff: Re: [Xen-devel] Branch Trace Storage for guests and VPMU initialization On 02/24/2015 10:27 AM, kevin.ma...@gdata.de wrote: Hi guys I`m trying to set up the BTS so that I can log the branches taken in the guest using Xen 4.4.1 with a WinXP SP3 guest on a Core i7 Sandy Bridge. I added the vpmu=bts boot parameter to my grub2 configuration and extended the libxl,libxc,domctl,… with an own command so that I can trigger the activation of the BTS whenever I want. I am not sure why you are doing all these changes to Xen code. BTS is supposed to be managed from the guest. For example, a Fedora HVM guest will produce this: [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf record -e branches:u -c 1 -d sleep 1 [ perf record: Woken up 3838 times to write data ] [ perf record: Captured and wrote 0.704 MB perf.data (~30756 samples) ] [root@dhcp-burlington7-2nd-B-east-10-152-55-140 ~]# perf script -f ip,addr,sym,dso,symoff --show-kernel-path 8167c347 native_irq_return_iret+0x0 (/proc/kcore) = 328c001590 [unknown] (/proc/kcore) 8167c347 native_irq_return_iret+0x0 (/proc/kcore) = 328c001590 [unknown] ([unknown]) 328c001593 [unknown] ([unknown]) = 328c004b70 [unknown] ([unknown]) ... I want to be able to log the taken branches (of the guest) without the need to modify the guest at all. This means I have to do all the logic in the hypervisor, or am I wrong? In that case, yes. But then you have to make sure that at least * you don't load guest's VPMU (or, at least, BTS-related registers) on context switch * You don't send the interrupt to the guest (meaning that you will need to somehow inform dom0 of the BTS interrupt) and probably more. Essentially, you want dom0 to profile the guest. I have been working on patches that would allow that but they are still under review. In this command I do the following: I set up the memory region for the BTS Buffer and the DS Buffer Management Area using xzalloc_bytes I don't think you should be allocating BTS buffers in the hypervisor, they are in guest's memory. I agree. As I said I think this is where my main problem is at the moment. Is there any way I can allocate memory in the hypervisor in a way the guest can access it? I am not sure this is what you want since you seem to *not* want the guest to process the samples, right? But yes, you can. E.g. something like what map_vcpu_info() does. (I have no idea how you'd do this from Windows.) Of course the guest must not be able to use this memory in its normal operations but just for BTS. Is this even possible? I am rather confused at the moment. :-D Then I write the pointer to the BTS Buffer into the DS Buffer Management Area at +0x0 and +0x8 (BTS Buffer Base and BTS Index) When I use vmx_msr_write_intercept to store the value in MSR_IA32_DS_AREA the host reboots (my idea is he tries to access a vpmu-struct that isn´t there in the current vcpu and panics). Who is trying to write to MSR_IA32_DS_AREA? The guest or dom0? I thought you said that you want dom0 to do sampling. Or are you trying to setup DS area from your guest and control it from dom0? I am somewhat confused. Can you post hypervisor log? (hard to say how helpful it will be without seeing your code changes though) Right after enabling the BTS I get a triple fault. hvm.c:1357:d2 Triple fault on VCPU0 - invoking HVM shutdown action 1. That's not host reboot, this is your guest dying. When I use a modified version of vmx_msr_write_intercept I don’t get any crashes as long as I don’t enable BTS and TR in the GUEST_IA32_DEBUGCTL (BTR works). When I enable the BTS (and TR) the guest crashes. I suppose he gets killed by the hypervisor for accessing forbidden memory. Possibly because DS area point to hypervisor memory. Having said all this, I am not sure how well BTS works. You did notice this in the hypervisor log: (XEN) ** (XEN) ** WARNING: Emulation of BTS Feature is switched on ** (XEN) ** Using this processor feature in a virtualized ** (XEN) ** environment is not 100% safe. ** (XEN) ** Setting the DS buffer address with wrong values ** (XEN) ** may lead to hypervisor hangs or crashes. ** (XEN) ** It is NOT recommended for production use! ** (XEN) ** Yes, I saw that. It doesn’t state that BTS is not working at all, just that it is not that safe to use. As I understand it as long as I set the DS buffer address correctly I should be fine, right? Right. Except that I am not convinced you did set this buffer correctly, which is possibly why your hypervisor crashed (I am not sure I understood under what circumstances though).
