Some feature requests for guest consoles

[Andy Smith]

Hi,

Mike H  made a feature request in:

https://lists.xenproject.org/archives/html/xen-users/2022-03/msg9.html

for the Xen guest console as connected to with "xl console" to
correctly support the terminal size rather than always being 80x20.

Additionally I wondered about some other features:

- Ability to remap the "magic sysrq" key combination which is
  ctrl-o, and possibly disable it while leaving "xl sysrq" and
  /proc/sysrq-trigger in the guest generally working.

  Reason: guest administrators are often inexperienced with the
  details of Xen. ctrl-o is a bad choice because it's actually the
  "save buffer" shortcut in the popular editor nano. On more than
  one occasion I have had guest administrators be editing a file
  with nano on their console, they go to save it with ctrl-o which
  appears to do nothing (because Xen is waiting for the sysrq
  command that follows), so they do ctrl-o again which is taken as
  being command 'o' - immediate power off! I have had an emergency
  support ticket about this because "my guest randomly crashed while
  I was editing a file".

  I would therefore like to remap "magic sysrq" to something more
  obscure, or failing that disable it in guests as we/they will use
  "xl sysrq" instead.

A couple of other things which may already be possible but I don't
know how, so if anyone's done it maybe they could give hints:

- Systematically and automatically log all guest console activity to
  individual files without interfering with the ability to use "xl
  console"

- Somehow plumb the xenconsole pty to a web terminal like xterm.js

Thanks!
Andy

Re: qemu-xen is unavailable

[Andy Smith]

Hello,

On Wed, Jan 05, 2022 at 04:27:56PM +, Anthony PERARD wrote:
> The bug here is that libxl shouldn't print this message for PVH guest
> because it's confusing.

It also does it for PV guests, again if no qemu is installed (or
needed). I squash it by adding:

device_model_version="qemu-xen"
device_model_override="/bin/true"

do the domU config files.

Cheers,
Andy

Filesystem corruption on restore without "xen-blkfront: introduce blkfront_gather_backend_features()"

[Andy Smith]

Hi,

[This conversation started on the xen-security-issues-discuss list
as I mistakenly thought it was to do with then-embargoed XSA
patches]

I did "xl save" on 17 domUs that were running under dom0 kernel
4.19.0-16-amd64 (4.19.181-1), hypervisor 4.14.2. I then rebooted
dom0 into kernel 5.10.0-0.bpo.8-amd64 (5.10.46-4~bpo10+1). On
restore 3 of the domUs were unresponsive and their consoles were
scrolling with:

backed has not unmapped grant: 1073
backed has not unmapped grant: 881
backed has not unmapped grant: 1474

(note typo)

After a destroy and boot there was filesystem corruption in the
domUs extensive enough to not be recoverable.

Andrew Cooper pointed me towards:


https://lore.kernel.org/lkml/1437449441-2964-1-git-send-email-bob@oracle.com/

The affected domUs were running obsolete kernels that did not have
that fix, which appears to have reached upstream in v4.2-rc7. Two of
them were running a 3.16 kernel from Debian 8 (jessie), one a 2.6.32
kernel from CentOS 6.

Out of the 17 there were quite a few other obsolete guest kernels
that apparently were not affected - just luck?

As I have these plus more users still running these obsolete kernels
I have some questions about this patch.

- Is the problem here in my case that the dom0 kernel 4.19 had not
  used feature-persistent whereas the dom0 kernel 5.10 does, and
  guests with obsolete kernels don't recognise this when they
  restore?

- The obsolete guests that apparently managed to restore okay
  presumably still have the problem and so can never safely be
  restored again until they have booted into a kernel with this
  patch?

- Am I right in thinking that guest kernels with this patch should
  be safe to keep doing save/restore even if they have already been
  restored across that change of dom0 4.19->5.10?

- Can guests with obsolete kernels that have done a shutdown and
  boot under 5.10 now be safely saved and restored again, as long as
  no new backend features are added?

Basically I now have a population of domUs running that I think can't
be safely saved/restored and I now need to identify them.

(The guests administrators have already been ignoring advice to
upgrade for a long time, so I can't just upgrade the problematic
ones. They will probably accept loss of save/restore rather than
upgrade.)

Thanks,
Andy

Re: 5.10.40 dom0 kernel - nvme: Invalid SGL for payload:131072 nents:13

[Andy Smith]

Hello,

On Tue, Jul 20, 2021 at 10:32:39PM +, Andy Smith wrote:
> I have a Debian 10 (buster/stable) dom0 running hypervisor 4.14.2.
> For almost 2 years it's been using the packaged Debian stable kernel
> which is 4.19.x.
> 
> Last night I upgraded the kernel to the buster-backports package
> which is based on 5.10.40 and about 4 hours later got this:
> 
> Jul 20 02:17:54 lamb kernel: [21061.388607] sg[0] 
> phys_addr:0x0015eb803000 offset:0 length:4096 
> dma_address:0x00209e7b7000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.389775] sg[1] 
> phys_addr:0x0015eb7bc000 offset:0 length:4096 
> dma_address:0x00209e7b8000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.390874] sg[2] 
> phys_addr:0x0015eb809000 offset:0 length:4096 
> dma_address:0x00209e7b9000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.391974] sg[3] 
> phys_addr:0x0015eb766000 offset:0 length:4096 
> dma_address:0x00209e7ba000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.393042] sg[4] 
> phys_addr:0x0015eb7a3000 offset:0 length:4096 
> dma_address:0x00209e7bb000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.394086] sg[5] 
> phys_addr:0x0015eb7c6000 offset:0 length:4096 
> dma_address:0x00209e7bc000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.395078] sg[6] 
> phys_addr:0x0015eb7c2000 offset:0 length:4096 
> dma_address:0x00209e7bd000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.396042] sg[7] 
> phys_addr:0x0015eb7a9000 offset:0 length:4096 
> dma_address:0x00209e7be000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.397004] sg[8] 
> phys_addr:0x0015eb775000 offset:0 length:4096 
> dma_address:0x00209e7bf000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.397971] sg[9] 
> phys_addr:0x0015eb7c7000 offset:0 length:4096 
> dma_address:0x0020ff52 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.398889] sg[10] 
> phys_addr:0x0015eb7cb000 offset:0 length:4096 
> dma_address:0x0020ff521000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.399814] sg[11] 
> phys_addr:0x0015eb7e3000 offset:0 length:61952 
> dma_address:0x0020ff522000 dma_length:61952
> Jul 20 02:17:54 lamb kernel: [21061.400754] sg[12] 
> phys_addr:0x0015eb7f2200 offset:512 length:24064 
> dma_address:0x0020ff531200 dma_length:24064
> Jul 20 02:17:54 lamb kernel: [21061.401781] [ cut here 
> ]
> Jul 20 02:17:54 lamb kernel: [21061.402738] Invalid SGL for payload:131072 
> nents:13
> Jul 20 02:17:54 lamb kernel: [21061.403724] WARNING: CPU: 1 PID: 12669 at 
> drivers/nvme/host/pci.c:716 nvme_map_data+0x7e0/0x820 [nvme]
> Jul 20 02:17:54 lamb kernel: [21061.404728] Modules linked in: binfmt_misc 
> ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpmss nf_log_ipv6 
> nf_log_ipv4 nf_log_common xt_LOG xt_limit nfnetlink_log nfnetlink xt_NFLOG 
> xt_multiport xt_tcpudp ip6table_filter ip6_tables iptable_filter bonding 
> btrfs blake2b_generic dm_snapshot dm_bufio intel_rapl_msr intel_rapl_common 
> skx_edac nfit libnvdimm intel_powerclamp crc32_pclmul ghash_clmulni_intel 
> ipmi_ssif aesni_intel libaes crypto_simd cryptd glue_helper snd_hda_intel 
> snd_intel_dspcfg mei_wdt soundwire_intel soundwire_generic_allocation nvme 
> wdat_wdt snd_soc_core ast snd_compress watchdog drm_vram_helper 
> drm_ttm_helper soundwire_cadence pcspkr nvme_core ttm snd_hda_codec 
> drm_kms_helper snd_hda_core i2c_i801 snd_hwdep i2c_smbus cec soundwire_bus 
> snd_pcm drm snd_timer snd soundcore igb ptp pps_core i2c_algo_bit joydev 
> mei_me sg mei intel_lpss_pci intel_lpss idma64 acpi_ipmi ipmi_si ipmi_devintf 
> ioatdma dca wmi ipmi_msghandler button dm_mod xenfs xen_acpi_processor
> Jul 20 02:17:54 lamb kernel: [21061.404831]  xen_privcmd xen_pciback 
> xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn ip_tables x_tables 
> autofs4 ext4 crc16 mbcache jbd2 raid456 libcrc32c crc32c_generic 
> async_raid6_recov async_memcpy async_pq async_xor xor async_tx evdev 
> hid_generic usbhid hid raid6_pq raid0 multipath linear raid10 raid1 md_mod 
> sd_mod t10_pi crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common 
> xhci_pci ahci libahci crc32c_intel xhci_hcd libata usbcore scsi_mod usb_common
> Jul 20 02:17:54 lamb kernel: [21061.417998] CPU: 1 PID: 12669 Comm: 62.xvda-0 
> Not tainted 5.10.0-0.bpo.7-amd64 #1 Debian 5.10.40-1~bpo10+1
> Jul 20 02:17:54 lamb kernel: [21061.418459] Hardware name: Supermicro Super 
> Server/X11SRM-VF, BIOS 1.2a 02/18/2019
> Jul 20 02:17:54 lamb kernel: [21061.418922] RIP: 
> e030:nvme_map_data+0x7e0/0x820 [nvme]
> Jul 20 02:17:54 lamb kernel: [21061.419354] Code: d0 7b c0 48 c7 c7 40 d6 7b 
> c0 e8 5b 44 

Re: 5.10.40 dom0 kernel - nvme: Invalid SGL for payload:131072 nents:13

[Andy Smith]

On Fri, Jul 23, 2021 at 08:10:28PM +, Andy Smith wrote:
> Hmm, I have the sector offset in the MD device so maybe I can
> convert that into a logical volume to know if a particular guest is
> provoking it…

So for anyone who ever wants to do that sort of thing:

# Find out offset that LVM puts LVs from start of its physical
# device
$ sudo pvs --noheadings -o pe_start --units s /dev/md4
2048S
# Find out the sector size of each LVM extent (PE)
$ sudo pvdisplay --units s /dev/md4 | grep 'PE Size'
  PE Size   8192 Se
# Report PE number for each sector
$ for sect in 912000815 916064223 1934755601 914360207 1936852857; do 
lv_sect=$((sect-2048)); pe=$((sect / 8192)); printf "%s: sector %s PE %s\n" 
$sect $lv_sect $pe; done
912000815: sector 911998767 PE 111328
916064223: sector 916062175 PE 111824
1934755601: sector 1934753553 PE 236176
914360207: sector 914358159 PE 111616
1936852857: sector 1936850809 PE 236432

Looking the PE numbers up in the output of "pvdisplay --maps --units
s /dev/md4" I can see it's three hits for one guest and two for
another.

I will see if I can move the 3 time affected guest to test hardware.

Cheers,
Andy

Re: 5.10.40 dom0 kernel - nvme: Invalid SGL for payload:131072 nents:13

[Andy Smith]

Hi Jan,

On Wed, Jul 21, 2021 at 04:49:26PM +0200, Jan Beulich wrote:
> On 21.07.2021 16:19, Andy Smith wrote:
> > I understand that below 4GiB memory use of swiotlb is disabled so
> > all the time previously this was not used, and now is. Perhaps the
> > bug is in there?
> > 
> > I was told that the only simple way on a Xen dom0 to disable use of
> > swiotlb would be to set the memory below 4GiB again, so I might try
> > that.
> 
> I have no idea where you take this 4GiB aspect from. What the kernel
> considers "below 4GiB" in its view of the world may be at a much
> higher address in system address space. And the mere amount of
> memory doesn't matter here at all.

Ah, I was taking that from:


https://elixir.bootlin.com/linux/v5.10.40/source/arch/x86/kernel/pci-swiotlb.c#L41

…which I had found while looking around how to disable use of
swiotlb. But I think I'm probably confused - should I only be
looking at arch/x86/xen/pci-swiotlb-xen.c in the case of a PV domain
like dom0?

I have not been able to reproduce the problem by giving a test
system with identical hardware more RAM and getting fio in a guest
to do random reads with a blocksize between 4kiB and 4MiB.

Perhaps it is highly workload dependent then. In some ways it's a
pity that I do not get call traces for the later occurrences as then
I could see if it's always the same 62.xvda-0 process (and thus same
guest) triggering it.

It's happened three more times since my previous email, but these
have been up to 46 hours apart. These were all reads, so MD just
satisfied the read from the other device without kicking the nvme
device out.

Hmm, I have the sector offset in the MD device so maybe I can
convert that into a logical volume to know if a particular guest is
provoking it…

If anyone happens to have any suggestions as to what kind of IO might
provoke this at all so I could maybe get fio to reproduce it, please
do let me know.

Thanks,
Andy

Re: 5.10.40 dom0 kernel - nvme: Invalid SGL for payload:131072 nents:13

[Andy Smith]

Hi Jan,

On Wed, Jul 21, 2021 at 10:10:13AM +0200, Jan Beulich wrote:
> Since xen-blkback only talks in terms of bio-s, I don't think it is
> the party responsible for honoring such driver restrictions. Instead
> I'd expect the block layer's bio merging to be where this needs to be
> observed. Perhaps it simply doesn't expect to be passed requests in
> multiples of 11 sectors together with the capping at 64k (as said -
> iirc) and driver restrictions on where splits may occur? And as to
> earlier Linux versions working - perhaps the merging logic was less
> aggressive back then?

I later realised that there was another change in my setup besides
the kernel version going from 4.19 to 5.10: I gave the dom0 8GiB of
memory instead of the 3GiB it had before.

I understand that below 4GiB memory use of swiotlb is disabled so
all the time previously this was not used, and now is. Perhaps the
bug is in there?

I was told that the only simple way on a Xen dom0 to disable use of
swiotlb would be to set the memory below 4GiB again, so I might try
that.

I was also pointed to this patch as a maybe fix for my issue:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit?id=5f89468e2f060031cd89fd4287298e0eaf246bf6

Thanks,
Andy

5.10.40 dom0 kernel - nvme: Invalid SGL for payload:131072 nents:13

[Andy Smith]

Hi,

I have a Debian 10 (buster/stable) dom0 running hypervisor 4.14.2.
For almost 2 years it's been using the packaged Debian stable kernel
which is 4.19.x.

Last night I upgraded the kernel to the buster-backports package
which is based on 5.10.40 and about 4 hours later got this:

Jul 20 02:17:54 lamb kernel: [21061.388607] sg[0] phys_addr:0x0015eb803000 
offset:0 length:4096 dma_address:0x00209e7b7000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.389775] sg[1] phys_addr:0x0015eb7bc000 
offset:0 length:4096 dma_address:0x00209e7b8000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.390874] sg[2] phys_addr:0x0015eb809000 
offset:0 length:4096 dma_address:0x00209e7b9000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.391974] sg[3] phys_addr:0x0015eb766000 
offset:0 length:4096 dma_address:0x00209e7ba000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.393042] sg[4] phys_addr:0x0015eb7a3000 
offset:0 length:4096 dma_address:0x00209e7bb000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.394086] sg[5] phys_addr:0x0015eb7c6000 
offset:0 length:4096 dma_address:0x00209e7bc000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.395078] sg[6] phys_addr:0x0015eb7c2000 
offset:0 length:4096 dma_address:0x00209e7bd000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.396042] sg[7] phys_addr:0x0015eb7a9000 
offset:0 length:4096 dma_address:0x00209e7be000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.397004] sg[8] phys_addr:0x0015eb775000 
offset:0 length:4096 dma_address:0x00209e7bf000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.397971] sg[9] phys_addr:0x0015eb7c7000 
offset:0 length:4096 dma_address:0x0020ff52 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.398889] sg[10] phys_addr:0x0015eb7cb000 
offset:0 length:4096 dma_address:0x0020ff521000 dma_length:4096
Jul 20 02:17:54 lamb kernel: [21061.399814] sg[11] phys_addr:0x0015eb7e3000 
offset:0 length:61952 dma_address:0x0020ff522000 dma_length:61952
Jul 20 02:17:54 lamb kernel: [21061.400754] sg[12] phys_addr:0x0015eb7f2200 
offset:512 length:24064 dma_address:0x0020ff531200 dma_length:24064
Jul 20 02:17:54 lamb kernel: [21061.401781] [ cut here ]
Jul 20 02:17:54 lamb kernel: [21061.402738] Invalid SGL for payload:131072 
nents:13
Jul 20 02:17:54 lamb kernel: [21061.403724] WARNING: CPU: 1 PID: 12669 at 
drivers/nvme/host/pci.c:716 nvme_map_data+0x7e0/0x820 [nvme]
Jul 20 02:17:54 lamb kernel: [21061.404728] Modules linked in: binfmt_misc 
ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpmss nf_log_ipv6 
nf_log_ipv4 nf_log_common xt_LOG xt_limit nfnetlink_log nfnetlink xt_NFLOG 
xt_multiport xt_tcpudp ip6table_filter ip6_tables iptable_filter bonding btrfs 
blake2b_generic dm_snapshot dm_bufio intel_rapl_msr intel_rapl_common skx_edac 
nfit libnvdimm intel_powerclamp crc32_pclmul ghash_clmulni_intel ipmi_ssif 
aesni_intel libaes crypto_simd cryptd glue_helper snd_hda_intel 
snd_intel_dspcfg mei_wdt soundwire_intel soundwire_generic_allocation nvme 
wdat_wdt snd_soc_core ast snd_compress watchdog drm_vram_helper drm_ttm_helper 
soundwire_cadence pcspkr nvme_core ttm snd_hda_codec drm_kms_helper 
snd_hda_core i2c_i801 snd_hwdep i2c_smbus cec soundwire_bus snd_pcm drm 
snd_timer snd soundcore igb ptp pps_core i2c_algo_bit joydev mei_me sg mei 
intel_lpss_pci intel_lpss idma64 acpi_ipmi ipmi_si ipmi_devintf ioatdma dca wmi 
ipmi_msghandler button dm_mod xenfs xen_acpi_processor
Jul 20 02:17:54 lamb kernel: [21061.404831]  xen_privcmd xen_pciback 
xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn ip_tables x_tables 
autofs4 ext4 crc16 mbcache jbd2 raid456 libcrc32c crc32c_generic 
async_raid6_recov async_memcpy async_pq async_xor xor async_tx evdev 
hid_generic usbhid hid raid6_pq raid0 multipath linear raid10 raid1 md_mod 
sd_mod t10_pi crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common 
xhci_pci ahci libahci crc32c_intel xhci_hcd libata usbcore scsi_mod usb_common
Jul 20 02:17:54 lamb kernel: [21061.417998] CPU: 1 PID: 12669 Comm: 62.xvda-0 
Not tainted 5.10.0-0.bpo.7-amd64 #1 Debian 5.10.40-1~bpo10+1
Jul 20 02:17:54 lamb kernel: [21061.418459] Hardware name: Supermicro Super 
Server/X11SRM-VF, BIOS 1.2a 02/18/2019
Jul 20 02:17:54 lamb kernel: [21061.418922] RIP: e030:nvme_map_data+0x7e0/0x820 
[nvme]
Jul 20 02:17:54 lamb kernel: [21061.419354] Code: d0 7b c0 48 c7 c7 40 d6 7b c0 
e8 5b 44 c9 c0 8b 93 4c 01 00 00 f6 43 1e 04 75 36 8b 73 28 48 c7 c7 20 9c 7b 
c0 e8 8b 71 09 c1 <0f> 0b 41 bd 0a 00 00 00 e9 f7 fe ff ff 48 8d bd 68 02 00 00 
48 89
Jul 20 02:17:54 lamb kernel: [21061.420271] RSP: e02b:c90044797930 EFLAGS: 
00010286
Jul 20 02:17:54 lamb kernel: [21061.420727] RAX:  RBX: 
888157db4200 RCX: 0027
Jul 20 02:17:54 lamb kernel: [21061.421186] RDX: 0027 RSI: 
888292858a00 RDI: 888292858a08
Jul 20 02:17:54 

Re: dom0 suddenly blocking on all access to md device

[Andy Smith]

Hi Rob,

On Sat, Jun 12, 2021 at 05:47:49PM -0500, Rob Townley wrote:
> mdadm.conf has email reporting capabilities to alert to failing drives.
> Test that you receive emails.

I do receive those emails, when such things occur, but the drives
are not failing.

Devices are not kicked out of MD arrays, all IO just stalls
completely. Also these incidents coincide with an upgrade of OS and
hypervisor and are happening on 5 different servers so far, so it
would be highly unlikely that so many devices suddenly went bad.

> Use mdadm to run tests on the raid.

Weekly scrubs take place using /usr/share/mdadm/checkarray

> smartctl -a /dev/

Yep, SMART health checks and self-testing are enabled.

I've now put two test servers on linux-image-amd64/buster-backports
and any time any of the production servers experiences the issue I
will boot it into that kernel next time.

Cheers,
Andy

Re: dom0 suddenly blocking on all access to md device

[Andy Smith]

130
Jun 12 12:04:40 clockwork kernel: [216427.281136]  ? 
md_super_write.part.63+0x90/0x120 [md_mod]
Jun 12 12:04:40 clockwork kernel: [216427.281788] 
md_update_sb.part.65+0x3a8/0x8e0 [md_mod]
Jun 12 12:04:40 clockwork kernel: [216427.282480]  ? md_rdev_init+0xb0/0xb0 
[md_mod]
Jun 12 12:04:40 clockwork kernel: [216427.283106] md_check_recovery+0x272/0x530 
[md_mod]
Jun 12 12:04:40 clockwork kernel: [216427.283738]  raid1d+0x5c/0xf10 [raid1]
Jun 12 12:04:40 clockwork kernel: [216427.284345]  ? __schedule+0x2a7/0x840
Jun 12 12:04:40 clockwork kernel: [216427.284939]  ? md_rdev_init+0xb0/0xb0 
[md_mod]
Jun 12 12:04:40 clockwork kernel: [216427.285522]  ? schedule+0x28/0x80
Jun 12 12:04:40 clockwork kernel: [216427.286121]  ? 
schedule_timeout+0x26d/0x3b0
Jun 12 12:04:40 clockwork kernel: [216427.286702]  ? __schedule+0x2a7/0x840
Jun 12 12:04:40 clockwork kernel: [216427.287279]  ? md_rdev_init+0xb0/0xb0 
[md_mod]
Jun 12 12:04:40 clockwork kernel: [216427.287871]  ? md_thread+0x94/0x150 
[md_mod]
Jun 12 12:04:40 clockwork kernel: [216427.288458]  ? process_checks+0x4a0/0x4a0 
[raid1]
Jun 12 12:04:40 clockwork kernel: [216427.289062] md_thread+0x94/0x150 [md_mod]
Jun 12 12:04:40 clockwork kernel: [216427.289663]  ? finish_wait+0x80/0x80
Jun 12 12:04:40 clockwork kernel: [216427.290288] kthread+0x112/0x130
Jun 12 12:04:40 clockwork kernel: [216427.290858]  ? kthread_bind+0x30/0x30
Jun 12 12:04:40 clockwork kernel: [216427.291433] ret_from_fork+0x35/0x40

What I HAVEN'T yet tried is a much newer kernel. That will probably
be what I try next having exhausted all ideas about upgrading or
configuring Xen.

Should I take a kernel from buster-backports which would currently
be:


https://packages.debian.org/buster-backports/linux-image-5.10.0-0.bpo.5-amd64

or should I build a kernel package from a mainline release?

Thanks,
Andy

On Fri, Feb 26, 2021 at 10:39:27PM +, Andy Smith wrote:
> Hi,
> 
> I suspect this might be an issue in the dom0 kernel (Debian buster,
> kernel 4.19.0-13-amd64), but just lately I've been sporadically
> having issues where dom0 blocks or severely slows down on all access
> to the particular md device that hosts all domU block devices.
> 
> Setup in dom0: an md RAID10 that is used as an LVM PV for an LVM volume
> group, where all domU block devices are LVM logical volumes in that
> group. So the relevant part of a domU config file might look like:
> 
> disk = [ "phy:/dev/myvg/domu_debtest1_xvda,xvda,w",
>  "phy:/dev/myvg/domu_debtest1_xvdb,xvdb,w" ]
> 
> The guests are mostly PV, a sprinkling of PVH, no HVM.
> 
> There's 5 of these servers but 3 of them have only recently been
> upgraded to Xen 4.12.14 (on Debian buster) from Xen 4.10 (on Debian
> jessie). The fact that all of them have been pretty stable in the
> past, on differing hardware, makes me discount a hardware issue. The
> fact that two of them have been buster / 4.12.x for a long time
> without issue but are also now starting to see this does make me
> think that it's a recent dom0 kernel issue.
> 
> When the problem occurs, inside every domU I see things like this:
> 
> Feb 26 20:02:34 backup4 kernel: [2530464.736085] INFO: task 
> btrfs-transacti:333 blocked for more than 120 seconds.
> Feb 26 20:02:34 backup4 kernel: [2530464.736107]   Not tainted 
> 4.9.0-14-amd64 #1 Debian 4.9.246-2
> Feb 26 20:02:34 backup4 kernel: [2530464.736117] "echo 0 > 
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> Feb 26 20:02:34 backup4 kernel: [2530464.736131] btrfs-transacti D0   333 
>  2 0x
> Feb 26 20:02:34 backup4 kernel: [2530464.736146]  0246 
> 8800f4e0c400  8800f8a7f100
> Feb 26 20:02:34 backup4 kernel: [2530464.736168]  8800fad18a00 
> 8800fa7dd000 c90040b2f670 8161a979
> Feb 26 20:02:34 backup4 kernel: [2530464.736188]  8800fa6d0200 
>  8800fad18a00 0010
> Feb 26 20:02:34 backup4 kernel: [2530464.736209] Call Trace:
> Feb 26 20:02:34 backup4 kernel: [2530464.736223]  [] ? 
> __schedule+0x239/0x6f0
> Feb 26 20:02:34 backup4 kernel: [2530464.736236]  [] ? 
> schedule+0x32/0x80
> Feb 26 20:02:34 backup4 kernel: [2530464.736248]  [] ? 
> schedule_timeout+0x1dd/0x380
> Feb 26 20:02:34 backup4 kernel: [2530464.736263]  [] ? 
> xen_clocksource_get_cycles+0x11/0x20
> Feb 26 20:02:34 backup4 kernel: [2530464.736275]  [] ? 
> io_schedule_timeout+0x9d/0x100
> Feb 26 20:02:34 backup4 kernel: [2530464.736289]  [] ? 
> __sbitmap_queue_get+0x24/0x90
> Feb 26 20:02:34 backup4 kernel: [2530464.736302]  [] ? 
> bt_get.isra.6+0x160/0x220
> Feb 26 20:02:34 backup4 kernel: [2530464.736338]  [] ? 
> __btrfs_map_block+0x6c8/0x11d0 [btrfs]
> Feb 26 20:02:34 backup4 kernel: [2530464.736353]  [] ? 
> prepare_to_wai

Re: dom0 suddenly blocking on all access to md device

[Andy Smith]

Hello,

On Fri, Feb 26, 2021 at 10:39:27PM +, Andy Smith wrote:
> just lately I've been sporadically having issues where dom0 blocks
> or severely slows down on all access to the particular md device
> that hosts all domU block devices.

This just happened again on the same server as the previous
occurrence on 26th February.

Again I was able to get things going again and avoid having to
destroy all guests and hard reset the host by doing a destroy on a
guest that was seen to be trying to do the most block IO (through
xentop).

This means that this particular host is still on Debian buster
kernel 4.19.0-13-amd64 and credit2 scheduler. As it's been only a
few days this might present itself again on this host quite soon. Is
there any useful information I can provide when it does?

Unfortunately there were no logs of interest neither in the dom0
syslog, guest syslog nor dom0 serial console. After the heavy IO
guest was destroyed dom0 syslog did give:

Mar  2 00:14:08 clockwork kernel: [6732447.973298] xen-blkback: Scheduled work 
from previous purge is still busy, cannot purge list

…but I assume that is just a result of IO springing back to life.

Thanks,
Andy

Re: dom0 suddenly blocking on all access to md device

[Andy Smith]

Oops, I didn't finish this sentence before sending:

On Fri, Feb 26, 2021 at 10:39:27PM +, Andy Smith wrote:
> Also, it's always the md device that the guest block devices are
> on that is stalled - IO to other devices in dom0
…seems fine.

Thanks,
Andy

dom0 suddenly blocking on all access to md device

[Andy Smith]

Hi,

I suspect this might be an issue in the dom0 kernel (Debian buster,
kernel 4.19.0-13-amd64), but just lately I've been sporadically
having issues where dom0 blocks or severely slows down on all access
to the particular md device that hosts all domU block devices.

Setup in dom0: an md RAID10 that is used as an LVM PV for an LVM volume
group, where all domU block devices are LVM logical volumes in that
group. So the relevant part of a domU config file might look like:

disk = [ "phy:/dev/myvg/domu_debtest1_xvda,xvda,w",
 "phy:/dev/myvg/domu_debtest1_xvdb,xvdb,w" ]

The guests are mostly PV, a sprinkling of PVH, no HVM.

There's 5 of these servers but 3 of them have only recently been
upgraded to Xen 4.12.14 (on Debian buster) from Xen 4.10 (on Debian
jessie). The fact that all of them have been pretty stable in the
past, on differing hardware, makes me discount a hardware issue. The
fact that two of them have been buster / 4.12.x for a long time
without issue but are also now starting to see this does make me
think that it's a recent dom0 kernel issue.

When the problem occurs, inside every domU I see things like this:

Feb 26 20:02:34 backup4 kernel: [2530464.736085] INFO: task btrfs-transacti:333 
blocked for more than 120 seconds.
Feb 26 20:02:34 backup4 kernel: [2530464.736107]   Not tainted 
4.9.0-14-amd64 #1 Debian 4.9.246-2
Feb 26 20:02:34 backup4 kernel: [2530464.736117] "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 26 20:02:34 backup4 kernel: [2530464.736131] btrfs-transacti D0   333   
   2 0x
Feb 26 20:02:34 backup4 kernel: [2530464.736146]  0246 
8800f4e0c400  8800f8a7f100
Feb 26 20:02:34 backup4 kernel: [2530464.736168]  8800fad18a00 
8800fa7dd000 c90040b2f670 8161a979
Feb 26 20:02:34 backup4 kernel: [2530464.736188]  8800fa6d0200 
 8800fad18a00 0010
Feb 26 20:02:34 backup4 kernel: [2530464.736209] Call Trace:
Feb 26 20:02:34 backup4 kernel: [2530464.736223]  [] ? 
__schedule+0x239/0x6f0
Feb 26 20:02:34 backup4 kernel: [2530464.736236]  [] ? 
schedule+0x32/0x80
Feb 26 20:02:34 backup4 kernel: [2530464.736248]  [] ? 
schedule_timeout+0x1dd/0x380
Feb 26 20:02:34 backup4 kernel: [2530464.736263]  [] ? 
xen_clocksource_get_cycles+0x11/0x20
Feb 26 20:02:34 backup4 kernel: [2530464.736275]  [] ? 
io_schedule_timeout+0x9d/0x100
Feb 26 20:02:34 backup4 kernel: [2530464.736289]  [] ? 
__sbitmap_queue_get+0x24/0x90
Feb 26 20:02:34 backup4 kernel: [2530464.736302]  [] ? 
bt_get.isra.6+0x160/0x220
Feb 26 20:02:34 backup4 kernel: [2530464.736338]  [] ? 
__btrfs_map_block+0x6c8/0x11d0 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736353]  [] ? 
prepare_to_wait_event+0xf0/0xf0
Feb 26 20:02:34 backup4 kernel: [2530464.736364]  [] ? 
blk_mq_get_tag+0x23/0x90
Feb 26 20:02:34 backup4 kernel: [2530464.736377]  [] ? 
__blk_mq_alloc_request+0x1a/0x220
Feb 26 20:02:34 backup4 kernel: [2530464.736390]  [] ? 
blk_mq_map_request+0xd9/0x170
Feb 26 20:02:34 backup4 kernel: [2530464.736402]  [] ? 
blk_mq_make_request+0xbb/0x580
Feb 26 20:02:34 backup4 kernel: [2530464.736429]  [] ? 
__btrfs_map_block+0x6c8/0x11d0 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736444]  [] ? 
generic_make_request+0x115/0x2d0
Feb 26 20:02:34 backup4 kernel: [2530464.736456]  [] ? 
submit_bio+0x76/0x140
Feb 26 20:02:34 backup4 kernel: [2530464.736481]  [] ? 
btrfs_map_bio+0x19a/0x340 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736505]  [] ? 
btree_submit_bio_hook+0xf5/0x110 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736535]  [] ? 
submit_one_bio+0x68/0x90 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736561]  [] ? 
read_extent_buffer_pages+0x1cd/0x300 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736587]  [] ? 
free_root_pointers+0x60/0x60 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736609]  [] ? 
btree_read_extent_buffer_pages+0x8c/0x100 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736635]  [] ? 
read_tree_block+0x34/0x50 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736655]  [] ? 
read_block_for_search.isra.36+0x133/0x320 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736678]  [] ? 
unlock_up+0xd4/0x180 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736700]  [] ? 
btrfs_search_slot+0x3ad/0xa00 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736723]  [] ? 
btrfs_insert_empty_items+0x67/0xc0 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736748]  [] ? 
__btrfs_run_delayed_refs+0xfc4/0x13a0 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736763]  [] ? 
xen_mc_flush+0xdd/0x1d0
Feb 26 20:02:34 backup4 kernel: [2530464.736785]  [] ? 
btrfs_run_delayed_refs+0x9d/0x2b0 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736811]  [] ? 
btrfs_commit_transaction+0x57/0xa10 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736837]  [] ? 
start_transaction+0x96/0x480 [btrfs]
Feb 26 20:02:34 backup4 kernel: [2530464.736861]  [] ? 
transaction_kthread+0x1dc/0x200 

Re: zstd compressed kernels

[Andy Smith]

On Tue, Nov 17, 2020 at 08:48:25PM +, Andrew Cooper wrote:
> For domU's, tools/libs/guest/xg_dom_bzimageloader.c and
> xc_dom_probe_bzimage_kernel()
> 
> (Wow this plumbing is ugly and in need of some rationalisation...)

Though not part of Xen, the PV part of grub could also do with some
love as it is also missing lz4 kernel support. This is particularly
painful since Ubuntu switched to lz4 kernels from the 19.10 release.

I understand from Jürgen though that grub's PVH support just uses
the normal i386 loader so if grub supports a bzImage on bare metal
it should do so with PVH, so that's an option. Certainly that works
for lz4 anyway.

Cheers,
Andy

Re: [Xen-devel] bug: unable to LZ4 decompress ub1910 installer kernel when launching domU

[Andy Smith]

Hello,

On Sun, Dec 01, 2019 at 06:47:14PM +0100, Jeremi Piotrowski wrote:
> On Thu, Oct 24, 2019 at 10:12:19AM +0200, Jan Beulich wrote:
> > Would you please increase verbosity (xl -vvv create ...) such that we
> > can see what exactly the decompression code doesn't like about this

[…]

> I stumbled across the same issue

In case it is useful, I was recently chatting to Pry Mar on IRC
about this issue. It also affects the Ubuntu 20.04 kernels (both
installer and OS packaged) which is no surprise since it seems they
switched to LZ4 compression from 19.10.

Pry Mar was able to make it boot under Xen PV by manually
uncompressing the vmlinuz first.

I have been meaning to take a recent Debian kernel or mainline
kernel and enable the LZ4 compression options to see if it is
reproducible outside of Ubuntu, but haven't found time yet.

Thanks,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] livepatch-build: What does getting no output from "readelf -wi xen-syms" usually mean?

[Andy Smith]

Hi,

I've been looking into live patching for the first time.

Starting with a 4.12.1 build:

$ cd ~/dev
$ ls -l
total 8
drwxr-xr-x 3 andy andy 4096 Oct 25 16:11 xen
drwxr-xr-x 6 andy andy 4096 Dec  2 01:16 livepatch-build-tools

(there is already a 4.12.1 hypervisor built in /xen and is what's
running on this host with build_id
b18af774b56b0c98cfa6940a725ba2ba26066929)

$ cp -a xen xen-lptest
$ cd livepatch-build-tools
$ ./livepatch-build -j 1 -s /home/andy/dev/xen-lptest/xen-4.12.1 -c 
/home/andy/dev/xen-lptest/xen-4.12.1/xen/.config -p ./lptest.patch -o lptest -d 
--depends b18af774b56b0c98cfa6940a725ba2ba26066929
Building LivePatch patch: lptest

Xen directory: /home/andy/dev/xen-lptest/xen-4.12.1
Patch file: /home/andy/dev/livepatch-build-tools/lptest.patch
.config file: /home/andy/dev/xen-lptest/xen-4.12.1/xen/.config
Output directory: /home/andy/dev/livepatch-build-tools/xsa310


Perform full initial build with 1 CPU(s)...
Reading special section data
ERROR: can't find special struct size.

So it seems it completed the initial build without error but looking
at the livepatch-build script it runs readelf like this:

$ readelf -wi lptest/xen-syms
$

For me this produces no output. I've probably done something simple
wrong. Does that indicate some simple mistake in my process?

The patch was just a trivial addition of some logging as a test, but
I don't think it got as far as applying that.

$ readelf --version
GNU readelf (GNU Binutils for Debian) 2.31.1
Copyright (C) 2018 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) any later version.
This program has absolutely no warranty.
$ gcc --version
gcc (Debian 8.3.0-6) 8.3.0
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Thanks,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] "CPU N still not dead..." messages during microcode update stage of boot when smt=0

[Andy Smith]

Hi,

On Mon, Jul 22, 2019 at 01:06:03PM +0100, Andrew Cooper wrote:
> On 22/07/2019 10:16, Jan Beulich wrote:
> > On 21.07.2019 22:06, Andy Smith wrote:
> >> (XEN) Adding cpu 1 to runqueue 0
> >> (XEN) CPU 1 still not dead...
> >> (XEN) CPU 1 still not dead...
> >> (XEN) CPU 1 still not dead...
> >> (XEN) CPU 1 still not dead...

[…]

> Does reverting back to credit1 make the issue go away?

Yes, I don't see this with sched=credit, smt=0 and SMT enabled in
the BIOS:

(XEN) microcode: CPU2 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) microcode: CPU4 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) microcode: CPU6 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) microcode: CPU8 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) microcode: CPU10 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) microcode: CPU12 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) microcode: CPU14 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) Brought up 8 CPUs
(XEN) Parked 8 CPUs

Thanks,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] "CPU N still not dead..." messages during microcode update stage of boot when smt=0

[Andy Smith]

Hi,

My first time using smt=0 on hypervisor command line so not sure how
many versions and different pieces of hardware this happens with,
but I noticed this during the microcode update stage of boot:

(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Adding cpu 1 to runqueue 0
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) CPU 1 still not dead...
(XEN) Removing cpu 1 from runqueue 0
(XEN) microcode: CPU2 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) Adding cpu 2 to runqueue 0
(XEN) Adding cpu 3 to runqueue 0
(XEN) Removing cpu 3 from runqueue 0
(XEN) microcode: CPU4 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) Adding cpu 4 to runqueue 0
(XEN) Adding cpu 5 to runqueue 0
(XEN) Removing cpu 5 from runqueue 0
(XEN) microcode: CPU6 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) Adding cpu 6 to runqueue 0
(XEN) Adding cpu 7 to runqueue 0
(XEN) Removing cpu 7 from runqueue 0
(XEN) microcode: CPU8 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) Adding cpu 8 to runqueue 0
(XEN) Adding cpu 9 to runqueue 0
(XEN) Removing cpu 9 from runqueue 0
(XEN) microcode: CPU10 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) Adding cpu 10 to runqueue 0
(XEN) Adding cpu 11 to runqueue 0
(XEN) Removing cpu 11 from runqueue 0
(XEN) microcode: CPU12 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) Adding cpu 12 to runqueue 0
(XEN) Adding cpu 13 to runqueue 0
(XEN) Removing cpu 13 from runqueue 0
(XEN) microcode: CPU14 updated from revision 0x257 to 0x25e, date = 
2019-04-02 
(XEN) Adding cpu 14 to runqueue 0
(XEN) Adding cpu 15 to runqueue 0
(XEN) Removing cpu 15 from runqueue 0
(XEN) Brought up 8 CPUs
(XEN) Parked 8 CPUs

It doesn't happen with smt=1 and it also doesn't happen when SMT is
disabled in the BIOS.

Boot does continue normally after this point.

Is this expected? 4.12.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] 4.10.1 Xen crash and reboot

[Andy Smith]

Hi,

On Tue, Jan 01, 2019 at 07:46:57PM +, Andy Smith wrote:
> The test host is slightly different hardware to the others: Xeon
> E5-1680v4 on there as opposed to Xeon D-1540 previously.
> 
> Test host is now running with pcid=0 to see if that helps. The
> longest this guest has been able to run so far without crashing the
> host is 14 days.

Just to note, it has so far been 28 days and this host using pcid=0
on the command line has not crashed again yet.

I still have no way to reproduce the problem on demand but if anyone
wants me me to do any further debugging with pcid=1 I can do that.

Thanks,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] 4.10.1 Xen crash and reboot

[Andy Smith]

Hello,

On Fri, Jan 04, 2019 at 03:16:32AM -0700, Jan Beulich wrote:
> >>> On 01.01.19 at 20:46,  wrote:
> > I did move the suspect guest to a test host that does not have
> > pcid=0 and 10 days later it crashed too:
> 
> Thanks for trying this. It is now pretty clear that we need a means
> to repro (and debug).

Also interesting to me is that the guest that seems to trigger this
is Debian stretch 64-bit with an older kernel that does not have
L1TF mitigations. I have asked the owner of it not to upgrade their
kernel yet because I think when the guest kernel behaves correctly
it will stop crashing the host and we'll learn nothing.

Sadly I cannot reproduce this on demand.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] 4.10.1 Xen crash and reboot

[Andy Smith]

Hello,

On Fri, Dec 21, 2018 at 06:55:38PM +, Andy Smith wrote:
> Is it worth me moving this guest to a test host without pcid=0 to
> see if it crashes it, meanwhile keeping production hosts with
> pcid=0? And then putting pcid=0 on the test host to see if it
> survives longer?

I did move the suspect guest to a test host that does not have
pcid=0 and 10 days later it crashed too:

(XEN) [ Xen-4.10.3-pre  x86_64  debug=n   Not tainted ]
(XEN) CPU:15
(XEN) RIP:e008:[] guest_4.o#shadow_set_l1e+0x75/0x6a0
(XEN) RFLAGS: 00010246   CONTEXT: hypervisor (d7v0)
(XEN) rax: 82e07b2e69c0   rbx: 803d9734e027   rcx: 
(XEN) rdx: 82e0   rsi: 81c4003dfa70   rdi: 
(XEN) rbp: 03d9734e   rsp: 83400e2afbd8   r8:  03d93187
(XEN) r9:     r10: 8300789f2000   r11: 
(XEN) r12: 803d9734e027   r13: 833f5be74000   r14: 03d9734e
(XEN) r15: 81c4003dfa70   cr0: 80050033   cr4: 00372660
(XEN) cr3: 003f56c31000   cr2: 81c4003dfa70
(XEN) fsb: 7f9de67fc700   gsb: 88007f20   gss: 
(XEN) ds:    es:    fs:    gs:    ss:    cs: e008
(XEN) Xen code around  (guest_4.o#shadow_set_l1e+0x75/0x6a0):
(XEN)  0f 20 0f 85 23 01 00 00 <4d> 8b 37 4c 39 f3 0f 84 97 01 00 00 49 89 da 89
(XEN) Xen stack trace from rsp=83400e2afbd8:
(XEN)003d9734e000 03d93187  833f0002
(XEN)8300789f2000 833f5be74000 81c4003dfa70 83400e2afef8
(XEN)03d93187 03d9734e 8300789f2000 82d08033f6f2
(XEN)833deb418e08 88007bf4e4d8 833f5be74600 03d9734e
(XEN)03d9734e 03d9734e 83400e2afd70 83400e2afd20
(XEN)00088007bf4e 0078 82d0805802c0 00028033c294
(XEN)0880 0008 0ef8 82d0805802c0
(XEN)03d93187 88007bf4e4d8 0a70 014e
(XEN)81c0e2001ef8 01ff82d0 803d9734e027 82d0
(XEN)833f0001 0001789f2000 83400e2a 83400e2afd20
(XEN)006f 88007bf4e4d8 003e11814067 003e11706067
(XEN)003d9341f067 8010003d9734e067 03e1310f 03e11814
(XEN)03e11706 03d9341f 0005 82d0803265b4
(XEN)82e07b25e140 833f5be74000 82e07ead8620 00010001
(XEN)0dff834003e1310f 000d0010  
(XEN)   82d0802845d3
(XEN)000d 82d08032a359 833f5be74000 0001000d
(XEN)82d080354913 82d080354907 82d080354913 82d080354907
(XEN)82d080354913 82d080354907 82d080354913 82d080354907
(XEN)82d080354913 82d080354907 82d080354913 82d080354907
(XEN) Xen call trace:
(XEN)[] guest_4.o#shadow_set_l1e+0x75/0x6a0
(XEN)[] guest_4.o#sh_page_fault__guest_4+0x8f2/0x2060
(XEN)[] shadow_alloc+0x1d4/0x380
(XEN)[] get_page+0x13/0xe0
(XEN)[] sh_resync_all+0xb9/0x2b0
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] do_page_fault+0x1a2/0x4e0
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] x86_64/entry.S#handle_exception_saved+0x68/0x94
(XEN) 
(XEN) Pagetable walk from 81c4003dfa70:
(XEN)  L4[0x103] = 803f56c31063 
(XEN)  L3[0x110] =  
(XEN) 
(XEN) 
(XEN) Panic on CPU 15:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=]
(XEN) Faulting linear address: 81c4003dfa70
(XEN) 
(XEN) 
(XEN) Reboot in five seconds...

The test host is slightly different hardware to the others: Xeon
E5-1680v4 on there as opposed to Xeon D-1540 previously.

Test host is now running with pcid=0 to see if that helps. The
longest this guest has been able to run so far without crashing the
host is 14 days.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] 4.10.1 Xen crash and reboot

[Andy Smith]

Hello,

And again today:

(XEN) [ Xen-4.10.3-pre  x86_64  debug=n   Not tainted ]
(XEN) CPU:4
(XEN) RIP:e008:[] 
guest_4.o#sh_page_fault__guest_4+0x70b/0x2060
(XEN) RFLAGS: 00010203   CONTEXT: hypervisor (d61v1)
(XEN) rax: 00c422641dd0   rbx: 832005c49000   rcx: 81c0e060
(XEN) rdx:    rsi: 832005c49000   rdi: 00c422641dd0
(XEN) rbp: 81c0e0601880   rsp: 83207e607c38   r8:  0310
(XEN) r9:     r10:    r11: 
(XEN) r12: 83207e607ef8   r13: 00f9cea7   r14: 
(XEN) r15: 830079592000   cr0: 80050033   cr4: 00372660
(XEN) cr3: 001ffab1a001   cr2: 81c0e0601880
(XEN) fsb: 7f89c67fc700   gsb:    gss: 88007f30
(XEN) ds:    es:    fs:    gs:    ss:    cs: e008
(XEN) Xen code around  
(guest_4.o#sh_page_fault__guest_4+0x70b/0x2060):
(XEN)  49 c1 e8 1e 4a 8d 2c c1 <48> 8b 4d 00 f6 c1 01 0f 84 f8 06 00 00 48 c1 e1
(XEN) Xen stack trace from rsp=83207e607c38:
(XEN)830f68748208 00c422641dd0 832005c49600 00f9cea7
(XEN)832005c49660 832005c49000 83207e607d70 83207e607d20
(XEN)0c422641 0090 82d0805802c0 000205c49000
(XEN)0008 0880 0898 82d0805802c0
(XEN)01fd58a1 01ffab1a 0208 0041
(XEN)800f9cea7825 01ff82d0 000d 82d0
(XEN)832005c49000 0001000d 83207e607fff 83207e607d20
(XEN)00a1 00c422641dd0 000f86569067 000f86544067
(XEN)000f68748067 800f9cea7925 00f87171 00f86569
(XEN)00f86544 00f68748 0005 
(XEN)82e03ff56340 832005c49000 00057ff0 
(XEN)83207e607e18 830079592000 832005c49000 83207e607ef8
(XEN)00c422641dd0 82d08034e4b0  82d080349e20
(XEN) 83207e607fff 830079592000 82d08034e5ae
(XEN)82d080354913 82d080354907 82d080354913 82d080354907
(XEN)82d080354913 82d080354907 82d080354913 82d080354907
(XEN)82d080354913 82d080354907 82d080354913 82d080354907
(XEN)82d080354913 83207e607ef8 830079592000 00c422641dd0
(XEN)832005c49000 0004  82d0802a1842
(XEN)82d080354913 82d080354907 82d080354913 82d080354907
(XEN) Xen call trace:
(XEN)[] guest_4.o#sh_page_fault__guest_4+0x70b/0x2060
(XEN)[] do_iret+0/0x1a0
(XEN)[] toggle_guest_pt+0x30/0x160
(XEN)[] do_iret+0xfe/0x1a0
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] do_page_fault+0x1a2/0x4e0
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] x86_64/entry.S#handle_exception_saved+0x68/0x94
(XEN)
(XEN) Pagetable walk from 81c0e0601880:
(XEN)  L4[0x103] = 801ffab1a063 
(XEN)  L3[0x103] = 801ffab1a063 
(XEN)  L2[0x103] = 801ffab1a063 
(XEN)  L1[0x001] =  
(XEN)
(XEN) 
(XEN) Panic on CPU 4:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=]
(XEN) Faulting linear address: 81c0e0601880
(XEN) 
(XEN)
(XEN) Reboot in five seconds...
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

Host has now rebooted into a hypervisor with pcid=0 command line.

I note that:

(XEN) RFLAGS: 00010203   CONTEXT: hypervisor (d61v1)

and the previous incident (below):

(XEN) RFLAGS: 00010246   CONTEXT: hypervisor (d31v1)

These are the same guest.

Is it worth me moving this guest to a test host without pcid=0 to
see if it crashes it, meanwhile keeping production hosts with
pcid=0? And then putting pcid=0 on the test host to see if it
survives longer?

This will take quite a long time to gain confidence of, since the
incidents are about 2 weeks apart each time.

Thanks,
Andy

On Mon, Dec 10, 2018 at 03:58:41PM +, Andy Smith wrote:
> Hi,
> 
> Up front information:
> 
> Today one of my Xen hosts crashed with this logging on the serial:
> 
> (XEN) [ 

Re: [Xen-devel] 4.10.1 Xen crash and reboot

[Andy Smith]

Hi Jan,

On Mon, Dec 10, 2018 at 09:29:34AM -0700, Jan Beulich wrote:
> >>> On 10.12.18 at 16:58,  wrote:
> > Are there any other hypervisor command line options that would be
> > beneficial to set for next time?
> 
> Well, just like for your report from a couple of weeks ago - if this is
> on PCID/INVPCID capable hardware, have you tried disabling use
> of PCID?

Aside from a quick test at Andrew's suggestion I have not, because I
thought there were negative repercussions of this, and up until this
point it seemed like problems were restricted to guests and could be
avoided by guest kernel upgrade.

The previous issue with the memory corruption in guests was avoided
by booting with pcid=0.

Does setting pcid=0 leave me increasingly vulnerable to Meltdown
and/or negatively impact performance?

Thanks,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] 4.10.1 Xen crash and reboot

[Andy Smith]

Hi,

Up front information:

Today one of my Xen hosts crashed with this logging on the serial:

(XEN) [ Xen-4.10.1  x86_64  debug=n   Not tainted ]
(XEN) CPU:15
(XEN) RIP:e008:[] guest_4.o#shadow_set_l1e+0x75/0x6a0
(XEN) RFLAGS: 00010246   CONTEXT: hypervisor (d31v1)
(XEN) rax: 82e01ecfae80   rbx: 000f67d74025   rcx: 
(XEN) rdx: 82e0   rsi: 81bfd79f12d8   rdi: 
(XEN) rbp: 00f67d74   rsp: 83202628fbd8   r8:  010175c6
(XEN) r9:     r10: 830079592000   r11: 
(XEN) r12: 000f67d74025   r13: 832020549000   r14: 00f67d74
(XEN) r15: 81bfd79f12d8   cr0: 80050033   cr4: 00372660
(XEN) cr3: 001fd5b8d001   cr2: 81bfd79f12d8
(XEN) fsb: 7faf3e71f700   gsb:    gss: 88007f30
(XEN) ds:    es:    fs:    gs:    ss:    cs: e008
(XEN) Xen code around  (guest_4.o#shadow_set_l1e+0x75/0x6a0):
(XEN)  0f 20 0f 85 23 01 00 00 <4d> 8b 37 4c 39 f3 0f 84 97 01 00 00 49 89 da 89
(XEN) Xen stack trace from rsp=83202628fbd8:
(XEN)000f67d74000 010175c6  8322
(XEN)830079592000 832020549000 81bfd79f12d8 83202628fef8
(XEN)010175c6 00f67d74 830079592000 82d08033fc82
(XEN)800fad0dc125 7faf3e25bba0 832020549600 00f67d74
(XEN)00f67d74 00f67d74 83202628fd70 83202628fd20
(XEN)0007faf3e25b 00c0 82d0805802c0 000220549000
(XEN)07f8 05e0 0f88 82d0805802c0
(XEN)010175c6 7faf3e25bba0 02d8 005b
(XEN)81c0dfebcf88 01ff82d0 000f67d74025 82d0
(XEN)832020549000 0001000d 83202628 83202628fd20
(XEN)00e9 7faf3e25bba0 000f472df067 000f49296067
(XEN)000f499f1067 000f67d74125 00f498cf 00f472df
(XEN)00f49296 00f499f1 0015 
(XEN)82e03fab71a0 830079593000 82d0803557eb 82d08020bf4a
(XEN) 830079592000 832020549000 83202628fef8
(XEN)0002 82d08034e9b0 00633400 82d08034a330
(XEN)830079592000 83202628 830079592000 82d08034eaae
(XEN)82d080355913 82d080355907 82d080355913 82d080355907
(XEN)82d080355913 82d080355907 82d080355913 82d080355907
(XEN)82d080355913 82d080355907 82d080355913 82d080355907
(XEN) Xen call trace:
(XEN)[] guest_4.o#shadow_set_l1e+0x75/0x6a0
(XEN)[] guest_4.o#sh_page_fault__guest_4+0x8f2/0x2060
(XEN)[] common_interrupt+0x9b/0x120
(XEN)[] evtchn_check_pollers+0x1a/0xb0
(XEN)[] do_iret+0/0x1a0
(XEN)[] toggle_guest_pt+0x30/0x160
(XEN)[] do_iret+0xfe/0x1a0
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] do_page_fault+0x1a2/0x4e0
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] handle_exception+0x8f/0xf9
(XEN)[] handle_exception+0x9b/0xf9
(XEN)[] x86_64/entry.S#handle_exception_saved+0x68/0x94
(XEN) 
(XEN) Pagetable walk from 81bfd79f12d8:
(XEN)  L4[0x103] = 801fd5b8d063 
(XEN)  L3[0x0ff] =  
(XEN) 
(XEN) Reboot in five seconds...
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

The same host also crashed about 2 weeks ago but I had nothing in
place to record the serial console so I have no logs. There has also
been one other host crash on a different host but again no
information collected.

Longer background:

Around the weekend of 18 November I deployed a hypervisor built from
staging-4.10 plus the outstanding XSA patches including XSA-273
which I had up until then held off on.

As described in:

https://lists.xenproject.org/archives/html/xen-devel/2018-11/msg02811.html

within a few days I began noticing sporadic memory corruption issues
in some guests, we established there was a bug in the L1TF fixes,
and I was able to avoid the problem in affected guests by making
sure to upgrade their guest kernels so they have Linux's L1TF fixes.

During first reboot into that hypervisor one of my hosts crashed and
rebooted, but it went by too fast for me to get any information and
there wasn't enough scrollback on the serial 

Re: [Xen-devel] Sporadic PV guest malloc.c assertion failures and segfaults unless pv-l1tf=false is set

[Andy Smith]

Hi Andrew,

On Sun, Nov 25, 2018 at 02:48:48PM +, Andrew Cooper wrote:
> Which are your two types of Intel server?

7 of them have Xeon D-1540, 2 of them have Xeon E5-1680v4. I've
seen this issue on guests running on both kinds, and my reproducer
guest was moved from a production D-1540 server to a test E5-1680v4
and still suffered.

My only available test host at the moment is E5-1680v4.

> You say that you only see this with 64bit Debian kernels?

Yes, but this seems quite subtle. I've got one Debian stretch guest
where php-fpm crashes every time, and another Debian stretch guest
(unknown kernel) where a particular perl script has an assertion
failure in malloc.c every time. Apart from that across several
hundred other guests it's only been observed a handful of times in a
week and these times were all on 64-bit Debian jessie and stretch.
So with the limited data this could still be coincidence.

I have one guest administrator with a 64-bit Gentoo guest saying
they might have seen it once because gcc crashed during a
compilation, but I am still waiting for clarification on that one.

> Could you experiment with disabling PCID (`pcid=0` on the xen command
> line) and seeing if that affects the reproducibility.

I am unable to reproduce the problem with pcid=0. staging-4.10,
64-bit Debian PV guest with the kernel revision just before L1TF
fixes (linux-image-4.9.0-7-amd64 4.9.110-3+deb9u2). Xen dmesg does
say shadow paging is in effect.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Sporadic PV guest malloc.c assertion failures and segfaults unless pv-l1tf=false is set

[Andy Smith]

Hello,

On Sun, Nov 25, 2018 at 06:18:49AM +, Andy Smith wrote:
> In the text for XSA-273 it says:
> 
> "Shadowing comes with a workload-dependent performance hit to
> the guest.  Once the guest kernel software updates have been
> applied, a well behaved guest will not write vulnerable PTEs,
> and will therefore avoid the performance penalty (or crash)
> entirely."
> 
> Does anyone have a reference to what is needed in the Linux kernel
> for that?

Perhaps stupidly, I have only just now thought to check whether the
one guest I have an easy reproducer on (predictable failure of
php-fpm) was actually running an up to date kernel. It was not.

It is Debian stretch and was running kernel package
linux-image-4.9.0-7-amd64 version 4.9.110-3+deb9u2. The guest's
administrator obviously had not done any upgrades since install time
because updated kernel linux-image-4.9.0-8-amd64 version 4.9.130-2
was available.

After installing and booting with that, the guest no longer causes
"L1TF-vulnerable L1e 6a6ff960 - Shadowing" to be emitted in
the hypervisor dmesg, and the problems I described disappear.

I assume this is because of:

https://metadata.ftp-master.debian.org/changelogs//main/l/linux/linux_4.9.130-2_changelog

linux (4.9.110-3+deb9u3) stretch-security; urgency=high

  [ Salvatore Bonaccorso ]
* Add L1 Terminal Fault fixes (CVE-2018-3620, CVE-2018-3646)

So, I can tell affected guest administrators to upgrade their
kernels to include Linux's L1TF protections and their problems
should go away.

Do you care that without those fixes there appear to be memory
corruption issues? If so, I can keep this reproducer guest around
and debug it further.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Sporadic PV guest malloc.c assertion failures and segfaults unless pv-l1tf=false is set

[Andy Smith]

Hi,

Last weekend I deployed a hypervisor built from 4.10.1 release
plus the most recent XSAs (which were under embargo at that time).
Previously to this I had only gone as far as XSA-267, having taken a
decision to wait before applying later XSAs. So, this most recent
deployment included the fixes for XSA-273 for the first time.

Over the course of this past week, some guests started to experience
sporadic assertion failures in libc/malloc.c or strange segmentation
violations. In most cases it is not easily reproducible, but I got a
report from one guest administrator that their php-fpm process is
reliably segfaulting immediately. For example:

[19-Nov-2018 06:39:56] WARNING: [pool www] child 3682 exited on signal 11 
(SIGSEGV) after 18.601413 seconds from start
[19-Nov-2018 06:39:56] NOTICE: [pool www] child 3683 started
[19-Nov-2018 06:40:16] WARNING: [pool www] child 3683 exited on signal 11 
(SIGSEGV) after 20.364357 seconds from start
[19-Nov-2018 06:40:16] NOTICE: [pool www] child 3684 started
[19-Nov-2018 06:43:43] WARNING: [pool www] child 3426 exited on signal 11 
(SIGSEGV) after 1327.885798 seconds from start
[19-Nov-2018 06:43:43] NOTICE: [pool www] child 3739 started
[19-Nov-2018 06:43:59] WARNING: [pool www] child 3739 exited on signal 11 
(SIGSEGV) after 15.922980 seconds from start

The failures that mention malloc.c are happening in multiple
different binaries, including grep, perl and shells. They look like
this:

grep: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char *) 
&((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd 
&& old_size == 0) || ((unsigned long) (old_size) >= (unsigned 
long)__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 
*(sizeof(size_t))) - 1)) & ~((2 *(sizeof(size_t))) - 1))) && ((old_top)->size & 
0x1) && ((unsigned long) old_end & pagemask) == 0)' failed.

I have not been able to reproduce these problems when I boot the
hypervisor with pv-l1tf=false. The php-fpm one was previously
reproducible 100% of the time. The other cases are very hard to
trigger but with pv-l1tf=false I am not able to at all.

I have since checked out staging-4.10 and am experiencing the same
thing, so I'm fairly confident it is not something I've introduced
when applying XSA patches.

My workload is several hundred PV guests across 9 servers with two
different types of Intel CPU. The guests are of many different Linux
distributions, probably a 70/30 split between 32- and 64-bit. I have
so far only encountered this with 64-bit guests running Debian
jessie and stretch, less than 10 guests are affected (so far
reported), and all of them trigger the "d1 L1TF-vulnerable L1e
6a6ff960 - Shadowing" warning in dmesg (though there are
hundreds of others which trigger it yet seem unaffected). There is
an unconfirmed report from 64-bit Gentoo.

In the text for XSA-273 it says:

"Shadowing comes with a workload-dependent performance hit to
the guest.  Once the guest kernel software updates have been
applied, a well behaved guest will not write vulnerable PTEs,
and will therefore avoid the performance penalty (or crash)
entirely."

Does anyone have a reference to what is needed in the Linux kernel
for that? Perhaps I can see what the status of that is within kernel
upstream / Debian and then get past the problem by getting an
updated guest kernel onto affected guests.

Also:

"This behaviour is active by default for guests on affected
hardware (controlled by `pv-l1tf=`), but is disabled by default
for dom0. Dom0's exemption is because of instabilities when
being shadowed, which are under investigation"

I have not had these issues in any of my 9 dom0s which are all
64-bit Debian jessie. Since these L1TF fixes are not active for
dom0, that makes sense. Are the observed dom0 instabilities similar
to what I am seeing in some guests?

Any suggestions for further debugging? I have attached an "xl
dmesg".

Cheers,
Andy
(XEN) parameter "placeholder" unknown!
 __  ___  __  ___   _   
 \ \/ /___ _ __   | || |  / |/ _ \ |___ /_ __  _ __ ___ 
  \  // _ \ '_ \  | || |_ | | | | |  |_ \ __| '_ \| '__/ _ \
  /  \  __/ | | | |__   _|| | |_| | ___) |__| |_) | | |  __/
 /_/\_\___|_| |_||_|(_)_|\___(_)/   | .__/|_|  \___|
|_| 
(XEN) Xen version 4.10.3-pre (a...@bitfolk.com) (gcc (Debian 4.9.2-10+deb8u1) 
4.9.2) debug=n  Sun Nov 25 05:48:03 UTC 2018
(XEN) Latest ChangeSet: Tue Nov 20 15:45:04 2018 +0100 git:b6e203b
(XEN) Bootloader: GRUB 2.02~beta2-22+deb8u1
(XEN) Command line: placeholder dom0_mem=2048M,max:4096M dom0_max_vcpus=2 
com1=115200,8n1,0x2f8,10 console=com1,vga ucode=scan serial_tx_buffer=256k 
loglvl=info
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 1 seconds
(XEN)  EDID info not retrieved because no 

Re: [Xen-devel] Problems booting 32-bit PV; just me or more widespread?

[Andy Smith]

Hi Boris,

On Thu, Aug 30, 2018 at 05:59:38PM -0400, Boris Ostrovsky wrote:
> On 08/29/2018 08:51 PM, Andy Smith wrote:
> > I cannot get any of the Ubuntu packaged 32-bit mainline kernels
> > after v4.13.16 that are found at
> > http://kernel.ubuntu.com/~kernel-ppa/mainline/ to boot in 32-bit PV
> > mode. All of them from v4.14.0rc1 onwards crash on xl create either
> > saying "error: no XEN note found." 
> 
> Don't know what this error is, perhaps kernel was not compiled with
> CONFIG_XEN.

So, your patch gets the packaged linux-image-generic working. As
mentioned above, in the course of looking into this problem I found
that many of the other 32-bit Ubuntu kernels crashed as well.

You're probably right that whatever I was testing above was not
compiled with CONFIG_XEN.

This one should work though as it's their netboot installer:


http://archive.ubuntu.com/ubuntu/dists/bionic/main/installer-i386/current/images/netboot/xen/

When I do xl create with this one (direct kernel boot), I get this
crash in the xl dmesg:

(XEN) domain_crash_sync called from entry.S: fault at 82d0803510d8 
x86_64/compat/entry.S#compat_create_bounce_frame+0xd9/0xed
(XEN) Domain 73 (vcpu#0) crashed on cpu#4:
(XEN) [ Xen-4.10.1  x86_64  debug=n   Not tainted ]
(XEN) CPU:4
(XEN) RIP:e019:[<c1038ef9>]
(XEN) RFLAGS: 0292   EM: 1   CONTEXT: pv guest (d73v0)
(XEN) rax: c1cd3540   rbx: c18a01e0   rcx: 
(XEN) rdx:    rsi: c1bfdeec   rdi: c1bfdf34
(XEN) rbp: c1bfdf10   rsp: c1bfdecc   r8:  
(XEN) r9:     r10:    r11: 
(XEN) r12:    r13:    r14: 
(XEN) r15:    cr0: 80050033   cr4: 003526e0
(XEN) cr3: 003f56d54000   cr2: 0014
(XEN) fsb:    gsb:    gss: 
(XEN) ds: e021   es: e021   fs: e021   gs: e021   ss: e021   cs: e019
(XEN) Guest stack trace from esp=c1bfdecc:
(XEN)    c1038ef9 0001e019 00010092 c1cd3540   
(XEN)       c1bfdf24 c1cd3540 c1cd3540 c1bfdf30
(XEN)   c1bfdf34 c1bfdf50 c1039a77 c1bfdf3c c1bfdf38 c1bfdf40 202e 1000
(XEN)     8008     c248a000
(XEN)   c1eda000 c1bfdffc c1cf5222     
(XEN)          
(XEN)          
(XEN)          
(XEN)          
(XEN)     c248a000 c1eda000    
(XEN)          
(XEN)          
(XEN)          
(XEN)          
(XEN)          
(XEN)          
(XEN)          
(XEN)          
(XEN)          
(XEN)          

Any ideas about that one?

Thanks,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Problems booting 32-bit PV; just me or more widespread?

[Andy Smith]

Hi,

I'm sorry this is a long email, but I wanted to explain everything
that I have tried, because it seems like quite a few different
versions of 32-bit upstream Linux kernel no longer boot as PV guest
and I'm surprised I am the first to encounter this. Probably I
have done something wrong.

I cannot get any of the Ubuntu packaged 32-bit mainline kernels
after v4.13.16 that are found at
http://kernel.ubuntu.com/~kernel-ppa/mainline/ to boot in 32-bit PV
mode. All of them from v4.14.0rc1 onwards crash on xl create either
saying "error: no XEN note found." or else immediately producing a
kernel panic like:

.
.
.
[ 0.114370] clocksource: jiffies: mask: 0x max_cycles: 0x, 
max_idle_ns: 764504178510 ns
[ 0.114382] futex hash table entries: 256 (order: 2, 16384 bytes)
[ 0.114423] pinctrl core: initialized pinctrl subsystem
[ 0.134326] RTC time: 165:165:165, date: 165/165/65
[ 0.134442] NET: Registered protocol family 16
[ 0.134457] xen:grant_table: Grant tables using version 1 layout
[ 0.134502] Grant table initialized
[ 0.134544] audit: initializing netlink subsys (disabled)
[ 0.134611] audit: type=2000 audit(1535307799.132:1): state=initialized 
audit_enabled=0 res=1
[ 0.134678] EISA bus registered
[ 0.136019] PCI: setting up Xen PCI frontend stub
[ 0.136073] BUG: unable to handle kernel paging request at edc21fd9
[ 0.136084] IP: eisa_bus_probe+0x19/0x36
[ 0.136089] *pdpt = 01ee6027 *pde = 29cc6067 *pte = 

[ 0.136100] Oops:  [#1] SMP
[ 0.136105] Modules linked in:
[ 0.136111] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0-33-generic 
#36-Ubuntu
[ 0.136120] EIP: eisa_bus_probe+0x19/0x36
[ 0.136125] EFLAGS: 00010246 CPU: 0
[ 0.136130] EAX: edc21fd9 EBX:  ECX: 01e0d000 EDX: 0200
[ 0.136138] ESI: c1d0d452 EDI: c1dd34a4 EBP: e9c89f24 ESP: e9c89f24
[ 0.136145] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: e021
[ 0.136154] CR0: 80050033 CR2: edc21fd9 CR3: 01e1 CR4: 00042660
[ 0.136166] Call Trace:
[ 0.136173] do_one_initcall+0x49/0x174
[ 0.136179] ? parse_args+0x143/0x390
[ 0.136187] ? set_debug_rodata+0x14/0x14
[ 0.136193] kernel_init_freeable+0x149/0x1c5
[ 0.136201] ? rest_init+0xa0/0xa0
[ 0.136207] kernel_init+0xd/0xf0
[ 0.136213] ret_from_fork+0x2e/0x38
[ 0.14] Code: ff b8 df 43 ae c1 e8 35 1b 88 ff e8 20 12 88 ff c9 c3 3e 8d 
74 26 00 55 b9 04 00 00 00 31 d2 b8 d9 ff 0f 00 89 e5 e8 35 8d 35 ff <8b> 10 81 
fa 45 49 53 41 75 0a c7 05 a0 76 ed c1 01 00 00 00 e8
[ 0.14] EIP: eisa_bus_probe+0x19/0x36 SS:ESP: e021:e9c89f24
[ 0.14] CR2: edc21fd9
[ 0.14] ---[ end trace 8c00b3cb7d4f06ba ]---
[ 0.140013] Kernel panic - not syncing: Attempted to kill init! 
exitcode=0x0009

(that one was from the currently-packaged linux-image-generic in
Ubuntu 18.04 LTS).

amd64 kernels work no problem.

On the host I'm running Xen 4.10.1 with up to date XSAs.

The guests are booted by either 32-bit or 64-bit pvgrub2 but to take
that out of the picture I copied the kernels and initramfs files to
the host and tried direct kernel boot. In that case I get:

$ sudo xl create -c /var/tmp/debtest1.conf
Parsing config from /var/tmp/debtest1.conf
xc: error: panic: xc_dom_core.c:702: xc_dom_find_loader: no loader found: 
Invalid kernel
libxl: error: libxl_dom.c:718:libxl__build_dom: xc_dom_parse_image failed: No 
such file or directory
libxl: error: libxl_create.c:1264:domcreate_rebuild_done: Domain 43:cannot 
(re-)build domain: -3
libxl: error: libxl_domain.c:1000:libxl__destroy_domid: Domain 43:Non-existant 
domain
libxl: error: libxl_domain.c:959:domain_destroy_callback: Domain 43:Unable to 
destroy guest
libxl: error: libxl_domain.c:886:domain_destroy_cb: Domain 43:Destruction of 
domain failed

The "No such file or directory" is strange as the file does exist at
the correct path. I can only think that the previous "Invalid
kernel" causes that.

I then began to wonder what the situation was like in Debian.

32-bit Debian stable works fine, but that's a 4.9.x kernel. That
works both from pvgrub2 and from direct kernel boot. A kernel based
on 4.9.17 is available in stretch-backports so I tried that
(linux-image-4.17.0-0.bpo.3-686). That behaves as above (either "No
XEN note" from pvgrub2 or "Invalid kernel" from xl create).

I then used the procedure described in the Debian Kernel Handbook
(https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-kernel-org-package)
to build a kernel from
https://git.kernel.org/torvalds/t/linux-4.19-rc1.tar.gz. That also
fails to boot in the same way.

So, my investigations so far suggest that everything from v4.14.0
onwards will not boot as a 32-bit PV guest. Is this a known issue,
or did I do something wrong? As I say, if it were an upstream kernel
issue then I am surprised I am the first to encounter it, but
perhaps the use of more modern kernel versions and 32-bit PV is
quite low.

If this is not a known issue then perhaps I can git bisect to find
what broke this, or perform any other 

Re: [Xen-devel] [Xen-users] Future of 32-bit PV support

[Andy Smith]

Hi Juergen,

As this was also addressed to -user I'm going to assume that you do
want user response as well.

On Thu, Aug 16, 2018 at 08:17:13AM +0200, Juergen Gross wrote:
> We'd like to evaluate whether anyone would see problems with:
> 
> - deprecating 32-bit PV guest support in Xen, meaning that we'd
>   eventually switch to support 32-bit PV guests only via PV-shim from
>   Xen 4.12 or 4.13

Although amd64 has been the default for us for many years, at the
moment we still have 64% of our customers running 32-bit PV. If
there remains a way for us to boot them through PV-shim and then
pvgrub2 with no functional changes and no work inside the guest then
that's fine, we'll adapt.

> - dropping 32-bit PV support from upstream Linux kernel, resulting in
>   current 32-bit PV guests no longer being able to upgrade to the newest
>   kernel version any longer

I doubt there is any technical reason why they can't switch to
64-bit, it's just that in the majority of cases that involves a
complete reinstall and the users just haven't bothered to.

If they are forced to switch because an impending kernel update will
leave them with a kernel that doesn't boot, they are going to be
upset that they are forced to reinstall their guest, or switch to a
64-bit kernel with their existing 32-bit userland.

It will of course help if they have plenty of warning that they need
to make the switch. But unless we're talking 2+ years of warning I'm
sure there will be some who will be unhappy.

I was hoping to transition to PVH guests as soon as possible, but
last time I looked into it there was a problem booting the stable
Linux kernel under PVH, and also no support in grub2.

Will it remain possible to boot a 32-bit Linux guest in PVH mode?

If so, could the final removal of 32-bit PV in the Linux kernel be
held off until there is:

1) a kernel shipping in Debian stable, Ubuntu LTS and CentOS that
   boots under PVH, and;

2) support in grub2 so I can build a grub image that boots under
   PVH?

If grub PVH support is not going to happen, what is the roadmap for
user-specified guest kernels under PVH?

> - is there any Linux distribution still shipping 32-bit PV-capable
>   systems?

Debian stable 32-bit kernels still boot under PV, as do Ubuntu 18.04
LTS ones. Ubuntu LTS releases are supposed to be supported (by
Canonical) for 5 years, and while of course Xen does not fall under
the category of software that they support, there will be people
sticking with 18.04 LTS as long as they can.

I'm not saying that people running 32-bit PV Ubuntu 18.04 are right
to expect that to continue being supported until 2023. I'm just
saying that human nature dictates that those sorts of expectations
will exist.

It will help a lot if there is an easy way for us to switch them
from 32-bit PV to PVH, while still letting them install their own
kernels.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] XPTI patches for 4.10 don't apply

[Andy Smith]

Hi Jan,

On Fri, Jan 19, 2018 at 02:52:50AM -0700, Jan Beulich wrote:
> >>> On 19.01.18 at 08:21,  wrote:
> > Maybe this is a silly question but which tag are the 4.10 XPTI
> > commits from https://xenbits.xen.org/xsa/xsa254/README.pti against?
> > They don't apply to RELEASE-4.10.0.
> 
> As always with security fixes, they are against the tip of the
> respective staging tree at the point they were being applied.

Ah, I thought it might be something like that.

I'm used to patches in XSAs being against a specific release. Will
there be patches published against RELEASE-4.10.0 or is the
expectation for people trying this to be using the tip of
4.10-staging?

Thanks,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] XPTI patches for 4.10 don't apply

[Andy Smith]

Hi,

Maybe this is a silly question but which tag are the 4.10 XPTI
commits from https://xenbits.xen.org/xsa/xsa254/README.pti against?
They don't apply to RELEASE-4.10.0.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 1/2] x86: Meltdown band-aid against malicious 64-bit PV guests

[Andy Smith]

Hi Jan,

On Tue, Jan 16, 2018 at 08:21:52AM -0700, Jan Beulich wrote:
> This is a very simplistic change limiting the amount of memory a running
> 64-bit PV guest has mapped (and hence available for attacking): Only the
> mappings of stack, IDT, and TSS are being cloned from the direct map
> into per-CPU page tables.

Can this be used with Comet/Vixen to further protect PV guests? i.e.
if the shim hypervisor has these changes then will it also limit
what a process in the PV guest can see in that shim hypervisor,
which therefore protects its own guest kernel a bit too?

Thanks,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Clarification regarding Meltdown and 64-bit PV guests

[Andy Smith]

Hi Hans,

On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
> By injecting a copy of a hypervisor between the outer level hypervisor
> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
> but it can attack the intermediate hypervisor which results in reading
> it's own memory from the fake intermediate "host memory".

So are you saying that, considering only SP3/Variant 3/Meltdown, it
works out like this:

== 64-bit PV mode guest ==

- Can't use SP3/Variant 3/Meltdown directly on its own kernel.

- Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
  hypervisor so effectively everything including other kernels and
  its own kernel.

- Can't be mitigated by KPTI in the guest.

== PV-in-Comet and PV-in-Vixen ==

- Can't use SP3/Variant 3/Meltdown directly on its own kernel

- Can't use SP3/Variant 3/Meltdown on the real hypervisor.

- Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
  still gain access to data from itself.

- Can't be mitigated by KPTI in the guest.

== HVM and PVHv2 ==

- Can use SP3/Variant 3/Meltdown directly on its own kernel.

- Can't use SP3/Variant 3/Meltdown on the hypervisor.

- Can be mitigated by KPTI in the guest (becomes not a Xen issue).

?

If so, then I can see how the FAQ, README.Comet and README.Vixen
can all be correct in this regard, but do note that this is
extremely confusing and a lot of people are only reading the
comments that say that Xen PV can't make use of SP3/Variant
3/Meltdown.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Clarification regarding Meltdown and 64-bit PV guests

[Andy Smith]

Hi,

In
:

"On Intel processors, only 64-bit PV mode guests can attack Xen
using Variant 3. Guests running in 32-bit PV mode, HVM mode, and
PVH mode (both v1 and v2) cannot attack the hypervisor using
Variant 3. However, in 32-bit PV mode, HVM mode, and PVH mode
(both v1 and v2), guest userspaces can attack guest kernels
using Variant 3; so updating guest kernels is advisable.

Interestingly, guest kernels running in 64-bit PV mode are not
vulnerable to attack using Variant 3, because 64-bit PV guests
already run in a KPTI-like mode."

However, in multiple other places, including
 and
:

"Note that both of these shim-based approaches prevent attacks on
the host, but leave the guest vulnerable to Meltdown attacks by
its own unprivileged processes; this is true even if the guest
OS has KPTI or similar Meltdown mitigation."

These seem to contradict each other.

The FAQ seems to suggest that:

- 32-bit PV guest userland processes can use Variant 3 against their
  own kernels and that the KPTI patch would protect against that.

- Without Comet/Vixen, 64-bit PV guests can't use Variant 3 on
  themselves but can use it on the hypervisor, and KPTI patches in
  the guest do not prevent that.

- Running PV guests inside Comet or Vixen prevents them making use
  of Variant 3, they still cannot use Variant 3 against their own
  kernels, and KPTI patches in the guest are not necessary.

The Comet and Vixen READMEs seem to suggest that:

- Use of Comet/Vixen prevents PV guests from using Variant 3 against
  the hypervisor (and thus other guests as well).

- The guest itself remains able to use Variant 3 on its own kernel
  and KPTI patches inside the guest cannot prevent this.

Which is correct, or have I misunderstood and they are somehow both
correct?

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Vixen - does no migration imply no save/restore?

[Andy Smith]

Hi,

I understand that Vixen does not support migration at this stage.
Does that also mean that save/restore is also not expected to work
for PV guests running with Vixen?

I tried it and it doesn't work, whereas it does when the guest is
started normal PV. I thought I better check expectations first.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Trying out vixen: qemu processes left behind

[Andy Smith]

Hi Anthony,

On Thu, Jan 11, 2018 at 03:47:25PM -0800, Anthony Liguori wrote:
> On Thu, Jan 11, 2018 at 3:00 PM, Andy Smith <a...@strugglers.net> wrote:
> > $ sudo xl list
> > NameID   Mem VCPUs  State   
> > Time(s)
> > Domain-0 0  2048 2 r-  
> > 104114.0
> > (null)  17 1 2 --ps-d  
> > 14.2
> > $ ps awux | grep qemu
> > root  3310  1.2  1.2 430648 24676 ?SLl  22:48   0:06 
> > /usr/local/lib/xen/bin/qemu-system-i386 -xen-domid 17 -chardev 
> > socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-17,server,nowait 
> > -no-shutdown -mon chardev=libxl-cmd,mode=control -chardev 
> > socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-17,server,nowait 
> > -mon chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config -name 
> > debtest1 -display none -serial pty -boot order=c -smp 2,maxcpus=2 -netdev 
> > type=tap,id=net0,ifname=vif17.0-emu,script=no,downscript=no -machine xenfv 
> > -cdrom /var/lib/xen/pvshim-sidecars/debtest1.iso -m 2552
> >
> > If I kill the qemu process then the domain does away.
> >
> > Is this expected? It doesn't seem workable if so.
> 
> Usually this means there is a dangling backend that wasn't properly
> terminated.  It would be useful to xenstore-ls the domain and see
> which backend is not in a disconnected state.

After halting that:

$ sudo xl list
NameID   Mem VCPUs  State   Time(s)
Domain-0 0  2048 2 r-  104246.6
(null)  19 1 2 --ps-d  86.6
$ sudo xenstore-ls
tool = ""
 xenstored = ""
local = ""
 domain = ""
  0 = ""
   control = ""
feature-poweroff = "1"
feature-reboot = "1"
feature-suspend = "1"
   domid = "0"
   name = "Domain-0"
   device-model = ""
vm = ""
libxl = ""
$ sudo xenstore-ls /local/domain/19
xenstore-ls: xs_directory (/local/domain/19): No such file or directory

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Trying out vixen: vif-route issue

[Andy Smith]

Hi,

On Thu, Jan 11, 2018 at 10:26:36PM +, Andy Smith wrote:
> Parsing config from /etc/xen/debtest1-with-shim.conf
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: 
> /etc/xen/scripts/vif-route add [31567] exited with error status 1
> libxl: error: libxl_device.c:1225:device_hotplug_child_death_cb: script: 
> /etc/xen/scripts/vif-route failed; error detected.
> libxl: error: libxl_create.c:1461:domcreate_attach_devices: unable to add nic 
> devices
> libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: 
> /etc/xen/scripts/vif-route remove [31751] exited with error status 1
> libxl: error: libxl_device.c:1225:device_hotplug_child_death_cb: script: 
> /etc/xen/scripts/vif-route failed; error detected.

I seem to have got it working. The vif-route thing was this:

https://lists.xen.org/archives/html/xen-users/2015-08/msg5.html

i.e. /etc/xen/scripts/vif-route in HVM mode is *still* being called
with both "online" and "add", and fails with the latter, leading to
"unable to add nic devices". I used Martti's suggested workaround
and things seem to work now.

In dom0 I have an extra v-debtest-emu interface that is shutdown.
What is that for? Do I need it? If not, can it be gotten rid of
somehow? It can't be doing too much if it's shutdown with no
addresses on it.

There is something odd going on with my pvgrub2 kernel which was
compiled with a ram disk. The first thing that happens is I get:

error: file `/ramdisk' not found.

Press any key to continue...

Then I can either press a key or else wait about 5 seconds, either
way pvgrub2 continues to boot, presents the menu I put in its ram
disk and the options correctly chain to whatever they are meant to
(only tested grub2 inside guest so far). The /ramdisk message
doesn't appear in PV mode so I don't know what that is about yet.

qemu process still hangs around after guest is shut down.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Trying out vixen: qemu processes left behind

[Andy Smith]

Hi,

I'm giving Vixen a try by following the instructions in
https://xenbits.xen.org/xsa/xsa254/README.vixen

Debian jessie, xen 4.8.1 packages from jessie-backports with XSAs
applied.

I finally got a guest booted although its networking doesn't work.
Every time I've started a guest and had it crash it's left behind a
domain called "(null)" and a matching qemu process so I assume
that's the device model. I thought that was just because the guest
was failing to start.

Now that I have one which boots, even when I shut it down cleanly
there's still a domain left behind:

$ sudo xl list
NameID   Mem VCPUs  State   Time(s)
Domain-0 0  2048 2 r-  104114.0
(null)  17 1 2 --ps-d  14.2
$ ps awux | grep qemu
root  3310  1.2  1.2 430648 24676 ?SLl  22:48   0:06 
/usr/local/lib/xen/bin/qemu-system-i386 -xen-domid 17 -chardev 
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-17,server,nowait -no-shutdown 
-mon chardev=libxl-cmd,mode=control -chardev 
socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-17,server,nowait -mon 
chardev=libxenstat-cmd,mode=control -nodefaults -no-user-config -name debtest1 
-display none -serial pty -boot order=c -smp 2,maxcpus=2 -netdev 
type=tap,id=net0,ifname=vif17.0-emu,script=no,downscript=no -machine xenfv 
-cdrom /var/lib/xen/pvshim-sidecars/debtest1.iso -m 2552

If I kill the qemu process then the domain does away.

Is this expected? It doesn't seem workable if so.

Cheers,
Andy

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [Xen-users] Trying out vixen: failure to start device model

[Andy Smith]

[Cc'ing xen-devel as this bit seems like a bug in pvshim]

On Thu, Jan 11, 2018 at 09:59:24PM +, Andy Smith wrote:
> libxl: debug: libxl_dm.c:2094:libxl__spawn_local_dm: Spawning device-model 
> /var/lib/xen/pvshim-sidecars/debtest1.dm with arguments:
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> /var/lib/xen/pvshim-sidecars/debtest1.dm
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -xen-domid
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   9
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-9,server,nowait
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -no-shutdown
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> chardev=libxl-cmd,mode=control
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -chardev
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-9,server,nowait
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -mon
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> chardev=libxenstat-cmd,mode=control
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -nodefaults
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -no-user-config
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -name
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   debtest1
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -vnc
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   127.0.0.1:0,to=99
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -display
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   none
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -kernel
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> /opt/grub/lib/grub-x86_64-xen.bin
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -serial
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   pty
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -device
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   cirrus-vga,vgamem_mb=8
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -boot
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   order=c
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -smp
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   2,maxcpus=2
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -device
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> rtl8139,id=nic0,netdev=net0,mac=00:16:5e:00:02:39
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -netdev
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> type=tap,id=net0,ifname=vif9.0-emu,script=no,downscript=no
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -machine
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   xenfv
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -cdrom
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> /var/lib/xen/pvshim-sidecars/debtest1.iso
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -m
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   2552
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -drive
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> file=/dev/ssdvg/domu_debtest1_xvda,if=ide,index=0,media=disk,format=raw,cache=writeback
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   -drive
> libxl: debug: libxl_dm.c:2096:libxl__spawn_local_dm:   
> file=/dev/ssdvg/domu_debtest1_xvdb,if=ide,index=1,media=disk,format=raw,cache=writeback
> libxl: debug: libxl_dm.c:2098:libxl__spawn_local_dm: Spawning device-model 
> /var/lib/xen/pvshim-sidecars/debtest1.dm with additional environment:
> libxl: debug: libxl_dm.c:2100:libxl__spawn_local_dm:   
> XEN_QEMU_CONSOLE_LIMIT=1048576
> libxl: debug: libxl_event.c:636:libxl__ev_xswatch_register: watch w=0x1b46a58 
> wpath=/local/domain/0/device-model/9/state token=2/2: register slotnum=2
> libxl: debug: libxl_event.c:573:watchfd_callback: watch w=0x1b46a58 
> wpath=/local/domain/0/device-model/9/state token=2/2: event 
> epath=/local/domain/0/device-model/9/state
> libxl: debug: libxl_exec.c:398:spawn_watch_event: domain 9 device model: 
> spawn watch p=(null)
> libxl: debug: libxl_event.c:673:libxl__ev_xswatch_deregister: watch 
> w=0x1b46a58 wpath=/local/domain/0/device-model/9/state token=2/2: deregister 
> slotnum=2
> libxl: error: libxl_dm.c:2189:device_model_spawn_outcome: domain 9 device 
> model: spawn failed (rc=-3)
> libxl: error: libxl_create.c:1504:domcreate_devmodel_started: device model 
> did not start: -3

I looked in the generated /var/lib/xen/pvshim-si