[ the math added to xen-tscmode.7 suggests that a domU should see a time
drift, which ntpd corrects. But the actual correction reported in
ntp.drift is entirely different than the one calculated in the
example. To me it is unclear why the example is wrong, more research
must be done. I'm sending this out just to get feedback about how
exactly the per-host knob must be implemented. ]
Add a knob to control when vTSC emulation will be activated for a domU
with tsc_mode=default. Without such option each TSC access from domU
will be emulated, which causes a significant perfomance drop for
workloads that make use of rdtsc.
One option to avoid the TSC option is to run domUs with tsc_mode=native.
This has the drawback that migrating a domU from a "2.3GHz" class host
to a "2.4GHz" class host will change the rate at wich the TSC counter
increases, the domU may not be prepared for that.
With the new knob the host admin can decide how a domU should behave
when it is migrated across systems of the same class. Since there is
always some jitter when Xen calibrates the cpu_khz value, all hosts of
the same class will most likely have slightly different values. As a
result vTSC emulation is unavoidable. Data collected during the incident
which triggered this change showed a jitter of up to 200 KHz across
systems of the same class.
Such knob can not be on a per-domU base because this would mean an
already running domU can not be handled anymore. Instead the newly added
knob is a global, optional per-host configuration file. Since it affects
only the restore path, the used configfile is /etc/xen/sr.conf.
Signed-off-by: Olaf Hering
--
v10:
- rebase to ae01a8e315
- remove changes for libxl and save/restore protocol, the feature has
to be per host instead of per guest
- add newline to tsc_set_info (Andrew)
- add pointer to xen-tscmode(7) in xl.cfg(5)/vtsc_tolerance_khz (Andrew)
- mention potential clock drift in the domU (Andrew)
- reword the newly added paragraph in xen-tscmode(7) (Andrew),
and also mention that it is about the measured/estimated TSC value
rather than the real value. The latter is simply unknown.
- use uint32 for internal representation of
xen_domctl_tsc_info.vtsc_tolerance_khz
and remove padding field
- add math for real TSC frequency to xen-tscmode
v9:
- extend commit msg, mention potential issues with xc_sr_rec_tsc_info._res1
v8:
- adjust also python stream checker for added tolerance member
v7:
- use uint16 in libxl_types.idl to match type used elsewhere in the patch
v6:
- mention default value in xl.cfg
- tsc_set_info: remove usage of __func__, use %d for domid
- tsc_set_info: use ABS to calculate khz_diff
v5:
- reduce functionality to allow setting of the tolerance value
only at initial domU startup
v4:
- add missing copyback in XEN_DOMCTL_set_vtsc_tolerance_khz
v3:
- rename vtsc_khz_tolerance to vtsc_tolerance_khz
- separate domctls to adjust values
- more docs
- update libxl.h
- update python tests
- flask check bound to tsc permissions
- not runtime tested due to dlsym() build errors in staging
---
docs/man/xen-tscmode.pod.7| 61 +
docs/man/xl.cfg.pod.5.in | 12 +
tools/libxc/include/xenctrl.h | 2 +
tools/libxc/xc_domain.c | 4 ++
tools/libxc/xc_sr_common.h| 2 +
tools/libxc/xc_sr_common_x86.c| 7 ++-
tools/libxc/xc_sr_restore.c | 93 +++
tools/libxl/libxl_x86.c | 2 +-
tools/python/xen/lowlevel/xc/xc.c | 2 +-
xen/arch/x86/domain.c | 2 +-
xen/arch/x86/domctl.c | 2 +
xen/arch/x86/time.c | 31 +++--
xen/include/asm-x86/domain.h | 1 +
xen/include/asm-x86/time.h| 6 ++-
xen/include/public/domctl.h | 2 +-
15 files changed, 218 insertions(+), 11 deletions(-)
diff --git a/docs/man/xen-tscmode.pod.7 b/docs/man/xen-tscmode.pod.7
index 819c61dd05..61cb69a381 100644
--- a/docs/man/xen-tscmode.pod.7
+++ b/docs/man/xen-tscmode.pod.7
@@ -103,6 +103,9 @@ whether or not the VM has been saved/restored/migrated
=back
+If the tsc_mode is set to "default" the decision to emulate TSC can be
+tweaked further with the "vtsc_tolerance_khz" option in /etc/xen/sr.conf.
+
To understand this in more detail, the rest of this document must
be read.
@@ -215,6 +218,64 @@ is emulated. Note that, though emulated, the "apparent"
TSC frequency
will be the TSC frequency of the initial physical machine, even after
migration.
+Since the calibration of the TSC frequency isn't 100% accurate, the
+value measured by Xen will vary across reboots. This means also several
+otherwise identical systems can have a slightly different _measured_ TSC
+frequency. As a result TSC access will be emulated if a domU is migrated
+from one host to another, identical host. To avoid the performance
+impact of TSC emulation a certain tolerance of the measured host TSC
+frequency can be
