On Sun, Apr 25, 2021 at 12:19:11AM -0500, Wei Huang wrote: > > > On 4/23/21 4:27 PM, Eduardo Habkost wrote: > > On Fri, Apr 23, 2021 at 12:32:00AM -0500, Wei Huang wrote: > > > There was a customer request for const_tsc support on AMD guests. Right > > > now > > > this feature is turned off by default for QEMU x86 CPU types (in > > > CPUID_Fn80000007_EDX[8]). However we are seeing a discrepancy in guest VM > > > behavior between Intel and AMD. > > > > > > In Linux kernel, Intel x86 code enables X86_FEATURE_CONSTANT_TSC based on > > > vCPU's family & model. So it ignores CPUID_Fn80000007_EDX[8] and guest VMs > > > have const_tsc enabled. On AMD, however, the kernel checks > > > CPUID_Fn80000007_EDX[8]. So const_tsc is disabled on AMD by default. > > > > Oh. This seems to defeat the purpose of the invtsc migration > > blocker we have. > > > > Do we know when this behavior was introduced in Linux? > > This code has existed in the kernel for a long time: > > commit 2b16a2353814a513cdb5c5c739b76a19d7ea39ce > Author: Andi Kleen <a...@linux.intel.com> > Date: Wed Jan 30 13:32:40 2008 +0100 > > x86: move X86_FEATURE_CONSTANT_TSC into early cpu feature detection > > There was another related commit which might explain the reasoning of > turning on CONSTANT_TSC based on CPU family on Intel: > > commit 40fb17152c50a69dc304dd632131c2f41281ce44 > Author: Venki Pallipadi <venkatesh.pallip...@intel.com> > Date: Mon Nov 17 16:11:37 2008 -0800 > > x86: support always running TSC on Intel CPUs > > According to the commit above, there are two kernel features: CONSTANT_TSC > and NONSTOP_TSC: > > * CONSTANT_TSC: TSC runs at constant rate > * NONSTOP_TSC: TSC not stop in deep C-states > > If CPUID_Fn80000007_EDX[8] == 1, both CONSTANT_TSC and NONSTOP_TSC are > turned on. This applies to all x86 vendors. For Intel CPU with certain CPU > families (i.e. c->x86 == 0x6 && c->x86_model >= 0x0e), it will turn on > CONSTANT_TSC (but no NONSTOP_TSC) with CPUID_Fn80000007_EDX[8]=0. > > I believe the migration blocker was created for the CONSTANT_TSC case: if > vCPU claims to have a constant TSC rate, we have to make sure src/dest are > matched with each other (having the same CPU frequency or have tsc_ratio). > NONSTOP_TSC doesn't matter in this scope. > > > > I am thinking turning on invtsc for EPYC CPU types (see example below). > > > Most > > > AMD server CPUs have supported invariant TSC for a long time. So this > > > change > > > is compatible with the hardware behavior. The only problem is live > > > migration > > > support, which will be blocked because of invtsc.
It should be blocked, if performed to a host with a different frequency or without TscRateMsr, if one desires the "constant TSC rate" meaning to be maintained. > > > However this problem > > > should be considered very minor because most server CPUs support > > > TscRateMsr > > > (see CPUID_Fn8000000A_EDX[4]), allowing VMs to migrate among CPUs with > > > different TSC rates. This live migration restriction can be lifted as long > > > as the destination supports TscRateMsr or has the same frequency as the > > > source (QEMU/libvirt do it). > > > > > > [BTW I believe this migration limitation might be unnecessary because it > > > is > > > apparently OK for Intel guests to ignore invtsc while claiming const_tsc. > > > Have anyone reported issues?] Not as far as i know. Fact is that libvirt will set the TSC_KHZ (from the value of KVM_GET_TSC_KHZ ioctl). That could be done inside QEMU itself, maybe by specifying -cpu AAA,cpu-freq=auto ? https://www.spinics.net/linux/fedora/libvir/msg141570.html