Re: [Xen-devel] [PATCH] vmx/monitor: CPUID events
On Sun, Jul 10, 2016 at 9:00 PM, Tian, Kevinwrote: >> From: Tamas K Lengyel [mailto:tamas.leng...@zentific.com] >> Sent: Friday, July 08, 2016 10:32 AM >> >> This patch implements sending notification to a monitor subscriber when an >> x86/vmx guest executes the CPUID instruction. >> >> Signed-off-by: Tamas K Lengyel > > >> @@ -3525,10 +3527,23 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs) >> hvm_task_switch((uint16_t)exit_qualification, reasons[source], >> ecode); >> break; >> } >> -case EXIT_REASON_CPUID: >> -is_pvh_vcpu(v) ? pv_cpuid(regs) : vmx_do_cpuid(regs); >> -update_guest_eip(); /* Safe: CPUID */ >> +case EXIT_REASON_CPUID: { >> +int rc; >> + >> +if ( is_pvh_vcpu(v) ) >> +{ >> +pv_cpuid(regs); >> +rc = 0; >> +} >> +else >> +rc = vmx_do_cpuid(regs); >> + >> +if ( rc < 0 ) >> +goto exit_and_crash; >> +if ( !rc ) >> +update_guest_eip(); /* Safe: CPUID */ > > favor a simple comment to explain policies on various > rc values, as you have done in other places. Certainly. Thanks, Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH] vmx/monitor: CPUID events
> From: Tamas K Lengyel [mailto:tamas.leng...@zentific.com] > Sent: Friday, July 08, 2016 10:32 AM > > This patch implements sending notification to a monitor subscriber when an > x86/vmx guest executes the CPUID instruction. > > Signed-off-by: Tamas K Lengyel> @@ -3525,10 +3527,23 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs) > hvm_task_switch((uint16_t)exit_qualification, reasons[source], > ecode); > break; > } > -case EXIT_REASON_CPUID: > -is_pvh_vcpu(v) ? pv_cpuid(regs) : vmx_do_cpuid(regs); > -update_guest_eip(); /* Safe: CPUID */ > +case EXIT_REASON_CPUID: { > +int rc; > + > +if ( is_pvh_vcpu(v) ) > +{ > +pv_cpuid(regs); > +rc = 0; > +} > +else > +rc = vmx_do_cpuid(regs); > + > +if ( rc < 0 ) > +goto exit_and_crash; > +if ( !rc ) > +update_guest_eip(); /* Safe: CPUID */ favor a simple comment to explain policies on various rc values, as you have done in other places. Thanks Kevin ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH] vmx/monitor: CPUID events
On 08/07/16 17:59, Tamas K Lengyel wrote: > On Fri, Jul 8, 2016 at 10:49 AM, Andrew Cooper >wrote: >> On 08/07/16 16:44, Tamas K Lengyel wrote: >>> On Fri, Jul 8, 2016 at 3:33 AM, Andrew Cooper >>> wrote: On 08/07/16 03:31, Tamas K Lengyel wrote: > This patch implements sending notification to a monitor subscriber when an > x86/vmx guest executes the CPUID instruction. > > Signed-off-by: Tamas K Lengyel Is it wise having an on/off control without any further filtering? (I suppose that it is at least a fine first start). >>> What type of extra filtering do you have in mind? >> Not sure. What are you intending to use this facility for? > Primarily to detect malware that is fingerprinting it's environment by > looking for hypervisor leafs and/or doing timing based detection by > benchmarking cpuid with rdtsc. > >> Given that the hypervisor is already in complete control of what a guest >> gets to see via cpuid, mutating the results via the monitor framework >> doesn't seem like a useful thing to do. > Indeed, the hypervisor is in control and to a certain extant the user > is via overriding some leafs in the domain config. However, there are > CPUID leafs Xen adds that the user is unable to override with the > domain config. For example in malware analysis it may be very useful > to be able to hide all hypervisor leafs from the guest, which > currently requires us to recompile Xen completely. By being able to > put the monitor system inline of CPUID it can decide which process it > wants to allow to see what leafs and when. It's very handy. Fair enough. For the record, my planned further work for cpuid will make things far more configurable. The current abilities of a toolstack, and the in-hypervisor auditing are woeful. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH] vmx/monitor: CPUID events
On Fri, Jul 8, 2016 at 10:49 AM, Andrew Cooperwrote: > On 08/07/16 16:44, Tamas K Lengyel wrote: >> On Fri, Jul 8, 2016 at 3:33 AM, Andrew Cooper >> wrote: >>> On 08/07/16 03:31, Tamas K Lengyel wrote: This patch implements sending notification to a monitor subscriber when an x86/vmx guest executes the CPUID instruction. Signed-off-by: Tamas K Lengyel >>> Is it wise having an on/off control without any further filtering? (I >>> suppose that it is at least a fine first start). >> What type of extra filtering do you have in mind? > > Not sure. What are you intending to use this facility for? Primarily to detect malware that is fingerprinting it's environment by looking for hypervisor leafs and/or doing timing based detection by benchmarking cpuid with rdtsc. > > Given that the hypervisor is already in complete control of what a guest > gets to see via cpuid, mutating the results via the monitor framework > doesn't seem like a useful thing to do. Indeed, the hypervisor is in control and to a certain extant the user is via overriding some leafs in the domain config. However, there are CPUID leafs Xen adds that the user is unable to override with the domain config. For example in malware analysis it may be very useful to be able to hide all hypervisor leafs from the guest, which currently requires us to recompile Xen completely. By being able to put the monitor system inline of CPUID it can decide which process it wants to allow to see what leafs and when. It's very handy. > >> >>> cpuid is usually the serialising instruction used with rdtsc for timing >>> loops. This is bad enough in VMs because of the VMExit, but becomes >>> even worse if there is a monitor delay as well. >> Yes, going the extra route of sending a monitor event out will add to >> that delay (how much delay will depend on the subscriber and what it >> decides to do with the event). Wouldn't we be able to mask some of >> that with tsc offsetting though? > > I am going to go out on a limb and say that that is a very large can of > worms which you don't want to open. Yea, I'm well aware. However, we might have to go down that rabbit hole eventually.. > > The problem is not that time skews from the point of view of the guest, > but that the timing loop with a fixed number of iterations takes > proportionally longer. > Yes, there is overhead inevitably. For our use-case what would be the goal is make the detection of this overhead as hard as possible so as long as the overhead is reasonable (ie. we don't make network connections drop and such) we can live with the overhead. Cheers, Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH] vmx/monitor: CPUID events
On 08/07/16 16:44, Tamas K Lengyel wrote: > On Fri, Jul 8, 2016 at 3:33 AM, Andrew Cooper> wrote: >> On 08/07/16 03:31, Tamas K Lengyel wrote: >>> This patch implements sending notification to a monitor subscriber when an >>> x86/vmx guest executes the CPUID instruction. >>> >>> Signed-off-by: Tamas K Lengyel >> Is it wise having an on/off control without any further filtering? (I >> suppose that it is at least a fine first start). > What type of extra filtering do you have in mind? Not sure. What are you intending to use this facility for? Given that the hypervisor is already in complete control of what a guest gets to see via cpuid, mutating the results via the monitor framework doesn't seem like a useful thing to do. > >> cpuid is usually the serialising instruction used with rdtsc for timing >> loops. This is bad enough in VMs because of the VMExit, but becomes >> even worse if there is a monitor delay as well. > Yes, going the extra route of sending a monitor event out will add to > that delay (how much delay will depend on the subscriber and what it > decides to do with the event). Wouldn't we be able to mask some of > that with tsc offsetting though? I am going to go out on a limb and say that that is a very large can of worms which you don't want to open. The problem is not that time skews from the point of view of the guest, but that the timing loop with a fixed number of iterations takes proportionally longer. ~Andrew ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH] vmx/monitor: CPUID events
On Fri, Jul 8, 2016 at 3:33 AM, Andrew Cooperwrote: > On 08/07/16 03:31, Tamas K Lengyel wrote: >> This patch implements sending notification to a monitor subscriber when an >> x86/vmx guest executes the CPUID instruction. >> >> Signed-off-by: Tamas K Lengyel > > Is it wise having an on/off control without any further filtering? (I > suppose that it is at least a fine first start). What type of extra filtering do you have in mind? > > cpuid is usually the serialising instruction used with rdtsc for timing > loops. This is bad enough in VMs because of the VMExit, but becomes > even worse if there is a monitor delay as well. Yes, going the extra route of sending a monitor event out will add to that delay (how much delay will depend on the subscriber and what it decides to do with the event). Wouldn't we be able to mask some of that with tsc offsetting though? Thanks, Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH] vmx/monitor: CPUID events
On Fri, Jul 8, 2016 at 1:03 AM, Razvan Cojocaruwrote: > On 07/08/16 05:31, Tamas K Lengyel wrote: >> This patch implements sending notification to a monitor subscriber when an >> x86/vmx guest executes the CPUID instruction. >> >> Signed-off-by: Tamas K Lengyel >> --- >> Cc: Ian Jackson >> Cc: Wei Liu >> Cc: Razvan Cojocaru >> Cc: Jan Beulich >> Cc: Andrew Cooper >> Cc: Jun Nakajima >> Cc: Kevin Tian >> --- >> tools/libxc/include/xenctrl.h | 1 + >> tools/libxc/xc_monitor.c| 13 + >> tools/tests/xen-access/xen-access.c | 33 - >> xen/arch/x86/hvm/monitor.c | 16 >> xen/arch/x86/hvm/vmx/vmx.c | 23 +++ >> xen/arch/x86/monitor.c | 13 + >> xen/include/asm-x86/domain.h| 1 + >> xen/include/asm-x86/hvm/monitor.h | 1 + >> xen/include/asm-x86/monitor.h | 3 ++- >> xen/include/public/domctl.h | 1 + >> xen/include/public/vm_event.h | 8 >> 11 files changed, 107 insertions(+), 6 deletions(-) >> >> diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h >> index 4a85b4a..e904bd5 100644 >> --- a/tools/libxc/include/xenctrl.h >> +++ b/tools/libxc/include/xenctrl.h >> @@ -2167,6 +2167,7 @@ int xc_monitor_guest_request(xc_interface *xch, >> domid_t domain_id, >> bool enable, bool sync); >> int xc_monitor_debug_exceptions(xc_interface *xch, domid_t domain_id, >> bool enable, bool sync); >> +int xc_monitor_cpuid(xc_interface *xch, domid_t domain_id, bool enable); >> /** >> * This function enables / disables emulation for each REP for a >> * REP-compatible instruction. >> diff --git a/tools/libxc/xc_monitor.c b/tools/libxc/xc_monitor.c >> index 264992c..4298813 100644 >> --- a/tools/libxc/xc_monitor.c >> +++ b/tools/libxc/xc_monitor.c >> @@ -172,6 +172,19 @@ int xc_monitor_debug_exceptions(xc_interface *xch, >> domid_t domain_id, >> return do_domctl(xch, ); >> } >> >> +int xc_monitor_cpuid(xc_interface *xch, domid_t domain_id, bool enable) >> +{ >> +DECLARE_DOMCTL; >> + >> +domctl.cmd = XEN_DOMCTL_monitor_op; >> +domctl.domain = domain_id; >> +domctl.u.monitor_op.op = enable ? XEN_DOMCTL_MONITOR_OP_ENABLE >> +: XEN_DOMCTL_MONITOR_OP_DISABLE; >> +domctl.u.monitor_op.event = XEN_DOMCTL_MONITOR_EVENT_CPUID; >> + >> +return do_domctl(xch, ); >> +} >> + >> /* >> * Local variables: >> * mode: C >> diff --git a/tools/tests/xen-access/xen-access.c >> b/tools/tests/xen-access/xen-access.c >> index 02655d5..d525b82 100644 >> --- a/tools/tests/xen-access/xen-access.c >> +++ b/tools/tests/xen-access/xen-access.c >> @@ -337,7 +337,7 @@ void usage(char* progname) >> { >> fprintf(stderr, "Usage: %s [-m] write|exec", progname); >> #if defined(__i386__) || defined(__x86_64__) >> -fprintf(stderr, "|breakpoint|altp2m_write|altp2m_exec|debug"); >> +fprintf(stderr, >> "|breakpoint|altp2m_write|altp2m_exec|debug|cpuid"); >> #endif >> fprintf(stderr, >> "\n" >> @@ -364,6 +364,7 @@ int main(int argc, char *argv[]) >> int shutting_down = 0; >> int altp2m = 0; >> int debug = 0; >> +int cpuid = 1; > > Should this be on by default? All the rest of the options are 0, and ... No, it's just a typo. Thanks, Tamas ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
Re: [Xen-devel] [PATCH] vmx/monitor: CPUID events
On 07/08/16 05:31, Tamas K Lengyel wrote: > This patch implements sending notification to a monitor subscriber when an > x86/vmx guest executes the CPUID instruction. > > Signed-off-by: Tamas K Lengyel> --- > Cc: Ian Jackson > Cc: Wei Liu > Cc: Razvan Cojocaru > Cc: Jan Beulich > Cc: Andrew Cooper > Cc: Jun Nakajima > Cc: Kevin Tian > --- > tools/libxc/include/xenctrl.h | 1 + > tools/libxc/xc_monitor.c| 13 + > tools/tests/xen-access/xen-access.c | 33 - > xen/arch/x86/hvm/monitor.c | 16 > xen/arch/x86/hvm/vmx/vmx.c | 23 +++ > xen/arch/x86/monitor.c | 13 + > xen/include/asm-x86/domain.h| 1 + > xen/include/asm-x86/hvm/monitor.h | 1 + > xen/include/asm-x86/monitor.h | 3 ++- > xen/include/public/domctl.h | 1 + > xen/include/public/vm_event.h | 8 > 11 files changed, 107 insertions(+), 6 deletions(-) > > diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h > index 4a85b4a..e904bd5 100644 > --- a/tools/libxc/include/xenctrl.h > +++ b/tools/libxc/include/xenctrl.h > @@ -2167,6 +2167,7 @@ int xc_monitor_guest_request(xc_interface *xch, domid_t > domain_id, > bool enable, bool sync); > int xc_monitor_debug_exceptions(xc_interface *xch, domid_t domain_id, > bool enable, bool sync); > +int xc_monitor_cpuid(xc_interface *xch, domid_t domain_id, bool enable); > /** > * This function enables / disables emulation for each REP for a > * REP-compatible instruction. > diff --git a/tools/libxc/xc_monitor.c b/tools/libxc/xc_monitor.c > index 264992c..4298813 100644 > --- a/tools/libxc/xc_monitor.c > +++ b/tools/libxc/xc_monitor.c > @@ -172,6 +172,19 @@ int xc_monitor_debug_exceptions(xc_interface *xch, > domid_t domain_id, > return do_domctl(xch, ); > } > > +int xc_monitor_cpuid(xc_interface *xch, domid_t domain_id, bool enable) > +{ > +DECLARE_DOMCTL; > + > +domctl.cmd = XEN_DOMCTL_monitor_op; > +domctl.domain = domain_id; > +domctl.u.monitor_op.op = enable ? XEN_DOMCTL_MONITOR_OP_ENABLE > +: XEN_DOMCTL_MONITOR_OP_DISABLE; > +domctl.u.monitor_op.event = XEN_DOMCTL_MONITOR_EVENT_CPUID; > + > +return do_domctl(xch, ); > +} > + > /* > * Local variables: > * mode: C > diff --git a/tools/tests/xen-access/xen-access.c > b/tools/tests/xen-access/xen-access.c > index 02655d5..d525b82 100644 > --- a/tools/tests/xen-access/xen-access.c > +++ b/tools/tests/xen-access/xen-access.c > @@ -337,7 +337,7 @@ void usage(char* progname) > { > fprintf(stderr, "Usage: %s [-m] write|exec", progname); > #if defined(__i386__) || defined(__x86_64__) > -fprintf(stderr, "|breakpoint|altp2m_write|altp2m_exec|debug"); > +fprintf(stderr, > "|breakpoint|altp2m_write|altp2m_exec|debug|cpuid"); > #endif > fprintf(stderr, > "\n" > @@ -364,6 +364,7 @@ int main(int argc, char *argv[]) > int shutting_down = 0; > int altp2m = 0; > int debug = 0; > +int cpuid = 1; Should this be on by default? All the rest of the options are 0, and ... > uint16_t altp2m_view_id = 0; > > char* progname = argv[0]; > @@ -426,6 +427,10 @@ int main(int argc, char *argv[]) > { > debug = 1; > } > +else if ( !strcmp(argv[0], "cpuid") ) > +{ > +cpuid = 1; > +} > #endif ... you also set it to 1 here. Thanks, Razvan ___ Xen-devel mailing list Xen-devel@lists.xen.org https://lists.xen.org/xen-devel
[Xen-devel] [PATCH] vmx/monitor: CPUID events
This patch implements sending notification to a monitor subscriber when an x86/vmx guest executes the CPUID instruction. Signed-off-by: Tamas K Lengyel--- Cc: Ian Jackson Cc: Wei Liu Cc: Razvan Cojocaru Cc: Jan Beulich Cc: Andrew Cooper Cc: Jun Nakajima Cc: Kevin Tian --- tools/libxc/include/xenctrl.h | 1 + tools/libxc/xc_monitor.c| 13 + tools/tests/xen-access/xen-access.c | 33 - xen/arch/x86/hvm/monitor.c | 16 xen/arch/x86/hvm/vmx/vmx.c | 23 +++ xen/arch/x86/monitor.c | 13 + xen/include/asm-x86/domain.h| 1 + xen/include/asm-x86/hvm/monitor.h | 1 + xen/include/asm-x86/monitor.h | 3 ++- xen/include/public/domctl.h | 1 + xen/include/public/vm_event.h | 8 11 files changed, 107 insertions(+), 6 deletions(-) diff --git a/tools/libxc/include/xenctrl.h b/tools/libxc/include/xenctrl.h index 4a85b4a..e904bd5 100644 --- a/tools/libxc/include/xenctrl.h +++ b/tools/libxc/include/xenctrl.h @@ -2167,6 +2167,7 @@ int xc_monitor_guest_request(xc_interface *xch, domid_t domain_id, bool enable, bool sync); int xc_monitor_debug_exceptions(xc_interface *xch, domid_t domain_id, bool enable, bool sync); +int xc_monitor_cpuid(xc_interface *xch, domid_t domain_id, bool enable); /** * This function enables / disables emulation for each REP for a * REP-compatible instruction. diff --git a/tools/libxc/xc_monitor.c b/tools/libxc/xc_monitor.c index 264992c..4298813 100644 --- a/tools/libxc/xc_monitor.c +++ b/tools/libxc/xc_monitor.c @@ -172,6 +172,19 @@ int xc_monitor_debug_exceptions(xc_interface *xch, domid_t domain_id, return do_domctl(xch, ); } +int xc_monitor_cpuid(xc_interface *xch, domid_t domain_id, bool enable) +{ +DECLARE_DOMCTL; + +domctl.cmd = XEN_DOMCTL_monitor_op; +domctl.domain = domain_id; +domctl.u.monitor_op.op = enable ? XEN_DOMCTL_MONITOR_OP_ENABLE +: XEN_DOMCTL_MONITOR_OP_DISABLE; +domctl.u.monitor_op.event = XEN_DOMCTL_MONITOR_EVENT_CPUID; + +return do_domctl(xch, ); +} + /* * Local variables: * mode: C diff --git a/tools/tests/xen-access/xen-access.c b/tools/tests/xen-access/xen-access.c index 02655d5..d525b82 100644 --- a/tools/tests/xen-access/xen-access.c +++ b/tools/tests/xen-access/xen-access.c @@ -337,7 +337,7 @@ void usage(char* progname) { fprintf(stderr, "Usage: %s [-m] write|exec", progname); #if defined(__i386__) || defined(__x86_64__) -fprintf(stderr, "|breakpoint|altp2m_write|altp2m_exec|debug"); +fprintf(stderr, "|breakpoint|altp2m_write|altp2m_exec|debug|cpuid"); #endif fprintf(stderr, "\n" @@ -364,6 +364,7 @@ int main(int argc, char *argv[]) int shutting_down = 0; int altp2m = 0; int debug = 0; +int cpuid = 1; uint16_t altp2m_view_id = 0; char* progname = argv[0]; @@ -426,6 +427,10 @@ int main(int argc, char *argv[]) { debug = 1; } +else if ( !strcmp(argv[0], "cpuid") ) +{ +cpuid = 1; +} #endif else { @@ -548,6 +553,16 @@ int main(int argc, char *argv[]) } } +if ( cpuid ) +{ +rc = xc_monitor_cpuid(xch, domain_id, 1); +if ( rc < 0 ) +{ +ERROR("Error %d setting cpuid listener with vm_event\n", rc); +goto exit; +} +} + /* Wait for access */ for (;;) { @@ -560,6 +575,8 @@ int main(int argc, char *argv[]) rc = xc_monitor_software_breakpoint(xch, domain_id, 0); if ( debug ) rc = xc_monitor_debug_exceptions(xch, domain_id, 0, 0); +if ( cpuid ) +rc = xc_monitor_cpuid(xch, domain_id, 0); if ( altp2m ) { @@ -716,6 +733,20 @@ int main(int argc, char *argv[]) } break; +case VM_EVENT_REASON_CPUID: +printf("CPUID executed: rip=%016"PRIx64", vcpu %d. Insn length: %"PRIu32" " \ + "EAX: 0x%"PRIx64" EBX: 0x%"PRIx64" ECX: 0x%"PRIx64" EDX: 0x%"PRIx64"\n", + req.data.regs.x86.rip, + req.vcpu_id, + req.u.cpuid.insn_length, + req.data.regs.x86.rax, + req.data.regs.x86.rbx, + req.data.regs.x86.rcx, + req.data.regs.x86.rdx); +rsp.flags |= VM_EVENT_FLAG_SET_REGISTERS; +rsp.data = req.data; +rsp.data.regs.x86.rip += req.u.cpuid.insn_length; +