Public bug reported:
Hugepages + split-core + NUMA + pinned CPUs make a simple apt-get
install for multiple packages to hang the CPU. Killing the process and
re-running the command would work (the packages are installed).
Installing single packages also works fine, so it only happens for
multiple packages.
Reproduce
=========
- reboot host
- ppc64_cpu --smt=on
- ppc64_cpu --sub-cores-per-core=4
-- ppc64_cpu --smt=off
- for i in 0 1 16 17; do echo 1300 >
/sys/devices/system/node/node$i/hugepages/hugepages-16384kB/nr_hugepages; done
- for s in libvirt-bin nova-compute nova-api; do service $s restart; done
- created a vm with the following config:
...
<vcpu placement='static'>8</vcpu>
<cputune>
<shares>8192</shares>
<vcpupin vcpu='0' cpuset='56'/>
<vcpupin vcpu='1' cpuset='58'/>
<vcpupin vcpu='2' cpuset='60'/>
<vcpupin vcpu='3' cpuset='62'/>
<vcpupin vcpu='4' cpuset='72'/>
<vcpupin vcpu='5' cpuset='74'/>
<vcpupin vcpu='6' cpuset='76'/>
<vcpupin vcpu='7' cpuset='78'/>
<emulatorpin cpuset='56,58,60,62,72,74,76,78'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='16'/>
<memnode cellid='0' mode='strict' nodeset='16'/>
</numatune>
...
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='1' cores='4' threads='2'/>
<numa>
<cell id='0' cpus='0-7' memory='10485760' unit='KiB' memAccess='shared'/>
</numa>
</cpu>
...
- apt-get install <pkg1> <pkg2> <pkg3>...
- cpu hangs, 100% busy
apt-get
=======
sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy=
apt-get --option Dpkg::Options::=--force-confold --assume-yes install
apache2 apache2-dev bc bridge-utils bsdmainutils curl g++ gcc gettext
git graphviz iputils-ping libapache2-mod-proxy-uwsgi libffi-dev libjpeg-
dev libmysqlclient-dev libpq-dev libssl-dev libsystemd-dev libxml2-dev
libxslt1-dev libyaml-dev lsof openssh-server openssl pkg-config psmisc
python2.7 python-dev python-gdbm screen tar tcpdump unzip uuid-runtime
wget wget zlib1g-dev lvm2 open-iscsi qemu-utils thin-provisioning-tools
lvm2 open-iscsi qemu-utils thin-provisioning-tools dstat libkrb5-dev
libldap2-dev libsasl2-dev memcached python-mysqldb sqlite3 conntrack
curl dnsmasq-base dnsmasq-utils ebtables gawk genisoimage iptables
iputils-arping kpartx libjs-jquery-tablesorter libmysqlclient-dev parted
pm-utils python-mysqldb socat sqlite3 sudo vlan cryptsetup genisoimage
gir1.2-libosinfo-1.0 netcat-openbsd open-iscsi qemu-utils sg3-utils
sysfsutils ipset acl dnsmasq-base ebtables haproxy iptables iputils-
arping iputils-ping libmysqlclient-dev postgresql-server-dev-all python-
mysqldb sqlite3 sudo vlan conntrack conntrackd keepalived curl
liberasurecode-dev make memcached sqlite3 xfsprogs
top
===
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
13231 root 20 0 40192 33792 6848 R 100.0 0.3 1:02.68 apt-get
strace
======
... <truncated>
= 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=7883, si_uid=0,
si_status=0, si_utime=0, si_stime=0} ---
close(16) = 0
waitpid(7883, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 7883
stat("/var/lib/dpkg/status", {st_mode=S_IFREG|0644, st_size=818492, ...}) = 0
stat("/var/lib/dpkg/status", {st_mode=S_IFREG|0644, st_size=818492, ...}) = 0
open("/var/lib/dpkg/status", O_RDONLY) = 16
fcntl(16, F_SETFD, FD_CLOEXEC) = 0
read(16, "Package: python-apt-common\nStatus: install ok installed\nPriority:
optional\nSection: python\nInstalled-Size: 244\nMaintainer: Ubuntu Developers
<ubuntu-devel-disc...@lists.ubuntu.com>\nArchitecture: all\nSource:
python-apt\nVersion: 1.1.0~beta1build1\nReplaces: python-apt (<<
0.7.98+nmu1)\nDepends: python | python3\nBreaks: python-apt (<<
0.7.98+nmu1)\nEnhances: python-apt, python3-apt\nDescription: Python interface
to libapt-pkg (locales)\n The apt_pkg Python interface will provide full access
to the internal\n libapt-pkg structures allowing Python programs to easily
perform a\n variety of functions.\n .\n This package contains
locales.\nOriginal-Maintainer: APT Development Team
<de...@lists.debian.org>\n\nPackage: node-lockfile\nStatus: install ok
installed\nPriority: extra\nSection: web\nInstalled-Size: 43\nMaintainer:
Ubuntu Developers <ubuntu-devel-disc...@lists.ubuntu.com>\nArchitecture:
all\nVersion: 0.4.1-1\nDepends: nodejs\nDescription: Asynchronous file lock mod
ule for Node.js\n This module provides asynchronous and synchronous concurrent
file\n locking. It supports timeouts, expirations, and retrying upon failure.\n
.\n Node.js is an event-based server-side javascript
engine.\nOriginal-Maintainer: Debian Javascript Maintainers
<pkg-javascript-de...@lists.alioth.debian.org>\nHomepage:
https://github.com/isaacs/lockfile\n\nPackage: libgtk2.0-bin\nStatus: install
ok installed\nPriority: optional\nSection: misc\nInstalled-Size:
80\nMaintainer: Ubuntu Desktop Team
<ubuntu-desk...@lists.ubuntu.com>\nArchitecture: ppc64el\nMulti-Arch:
foreign\nSource: gtk+2.0\nVersion: 2.24.30-1ubuntu1\nDepends: libgtk2.0-0 (=
2.24.30-1ubuntu1), libgtk2.0-common\nDescription: programs for the GTK+
graphical user interface library\n GTK+ is a multi-platform toolkit for
creating graphical user\n interfaces. Offering a complete set of widgets, GTK+
is suitable\n for projects ranging from small one-off tools to complete
application\n suites.\n .\n This package conta
ins the utilities which are used by the libraries\n and other
packages.\nHomepage: http://www.gtk.org/\nOriginal-Maintain";..., 32771) = 32771
fstat(16, {st_mode=S_IFREG|0644, st_size=818492, ...}) = 0
fstat(16, {st_mode=S_IFREG|0644, st_size=818492, ...}) = 0
libvirt xml
===========
<domain type='kvm' id='13'>
<name>instance-0002fd88</name>
<uuid>36c7dc00-b230-4cb2-a28d-43f81ba5fa80</uuid>
<metadata>
<nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0";>
<nova:package version="13.1.2"/>
<nova:name>devstack-xenial-smt-ppc64-smt-151554</nova:name>
<nova:creationTime>2017-06-14 16:41:33</nova:creationTime>
<nova:flavor name="numa-hugepages">
<nova:memory>10240</nova:memory>
<nova:disk>80</nova:disk>
<nova:swap>10240</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>8</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user uuid="d17cb3d4398f4c8c84f96d78e3ff50e8">pkvmci</nova:user>
<nova:project
uuid="7b62ae2fea4545a2b6b68c9024aafbf1">pkvmci</nova:project>
</nova:owner>
<nova:root type="image" uuid="6b6f077d-1860-4864-838d-84ea1085afee"/>
</nova:instance>
</metadata>
<memory unit='KiB'>10485760</memory>
<currentMemory unit='KiB'>10485760</currentMemory>
<memoryBacking>
<hugepages>
<page size='16384' unit='KiB' nodeset='0'/>
</hugepages>
</memoryBacking>
<vcpu placement='static'>8</vcpu>
<cputune>
<shares>8192</shares>
<vcpupin vcpu='0' cpuset='56'/>
<vcpupin vcpu='1' cpuset='58'/>
<vcpupin vcpu='2' cpuset='60'/>
<vcpupin vcpu='3' cpuset='62'/>
<vcpupin vcpu='4' cpuset='72'/>
<vcpupin vcpu='5' cpuset='74'/>
<vcpupin vcpu='6' cpuset='76'/>
<vcpupin vcpu='7' cpuset='78'/>
<emulatorpin cpuset='56,58,60,62,72,74,76,78'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='16'/>
<memnode cellid='0' mode='strict' nodeset='16'/>
</numatune>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='ppc64le' machine='pseries-2.6'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='1' cores='4' threads='2'/>
<numa>
<cell id='0' cpus='0-7' memory='10485760' unit='KiB' memAccess='shared'/>
</numa>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='writeback'/>
<source
file='/raid-data/instances/36c7dc00-b230-4cb2-a28d-43f81ba5fa80/disk'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source
file='/raid-data/instances/_base/8b0287808ddb3011d589c92ff4e9474a4efa4700'/>
<backingStore/>
</backingStore>
<target dev='vda' bus='virtio'/>
<alias name='virtio-disk0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='writeback'/>
<source
file='/raid-data/instances/36c7dc00-b230-4cb2-a28d-43f81ba5fa80/disk.swap'/>
<backingStore type='file' index='1'>
<format type='raw'/>
<source file='/raid-data/instances/_base/swap_10240'/>
<backingStore/>
</backingStore>
<target dev='vdb' bus='virtio'/>
<alias name='virtio-disk1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw' cache='writeback'/>
<source
file='/raid-data/instances/36c7dc00-b230-4cb2-a28d-43f81ba5fa80/disk.config'/>
<backingStore/>
<target dev='sdz' bus='scsi'/>
<readonly/>
<alias name='scsi3-0-0-4'/>
<address type='drive' controller='3' bus='0' target='0' unit='4'/>
</disk>
<controller type='usb' index='0'>
<alias name='usb'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</controller>
<controller type='pci' index='0' model='pci-root'>
<alias name='pci.0'/>
</controller>
<controller type='scsi' index='0'>
<alias name='scsi0'/>
<address type='spapr-vio' reg='0x2000'/>
</controller>
<controller type='scsi' index='1'>
<alias name='scsi1'/>
<address type='spapr-vio' reg='0x3000'/>
</controller>
<controller type='scsi' index='2'>
<alias name='scsi2'/>
<address type='spapr-vio' reg='0x4000'/>
</controller>
<controller type='scsi' index='3'>
<alias name='scsi3'/>
<address type='spapr-vio' reg='0x5000'/>
</controller>
<interface type='bridge'>
<mac address='fa:16:3e:10:d5:b4'/>
<source bridge='brq47fb4423-56'/>
<target dev='tapaf9bc01f-18'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
</interface>
<serial type='file'>
<source
path='/raid-data/instances/36c7dc00-b230-4cb2-a28d-43f81ba5fa80/console.log'/>
<target port='0'/>
<alias name='serial0'/>
<address type='spapr-vio' reg='0x30000000'/>
</serial>
<serial type='pty'>
<source path='/dev/pts/6'/>
<target port='1'/>
<alias name='serial1'/>
<address type='spapr-vio' reg='0x30001000'/>
</serial>
<console type='file'>
<source
path='/raid-data/instances/36c7dc00-b230-4cb2-a28d-43f81ba5fa80/console.log'/>
<target type='serial' port='0'/>
<alias name='serial0'/>
<address type='spapr-vio' reg='0x30000000'/>
</console>
<input type='tablet' bus='usb'>
<alias name='input0'/>
</input>
<input type='keyboard' bus='usb'>
<alias name='input1'/>
</input>
<input type='mouse' bus='usb'>
<alias name='input2'/>
</input>
<graphics type='vnc' port='5905' autoport='yes' listen='0.0.0.0'
keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='vga' vram='16384' heads='1'/>
<alias name='video0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</video>
<memballoon model='virtio'>
<stats period='10'/>
<alias name='balloon0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</memballoon>
<panic model='pseries'/>
</devices>
</domain>
More apt-get install examples
=============================
sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy=
apt-get --option Dpkg::Options::=--force-confold --assume-yes install
libvirt-bin libvirt-dev
Processing triggers for libc-bin (2.23-0ubuntu7) ...
Processing triggers for systemd (229-4ubuntu17) ...
^C
$ sudo DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy=
apt-get --option Dpkg::Options::=--force-confold --assume-yes install fakeroot
make openvswitch-switch
Reading package lists... 0%
^C
rfolco@devstack-xenial-smt-ppc64-smt-151554:~$ sudo
DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get
--option Dpkg::Options::=--force-confold --assume-yes install fakeroot make
openvswitch-switch
Reading package lists... 0%
c^C
rfolco@devstack-xenial-smt-ppc64-smt-151554:~$ ^C
rfolco@devstack-xenial-smt-ppc64-smt-151554:~$ ^C
rfolco@devstack-xenial-smt-ppc64-smt-151554:~$ sudo
DEBIAN_FRONTEND=noninteractive http_proxy= https_proxy= no_proxy= apt-get
--option Dpkg::Options::=--force-confold --assume-yes install fakeroot make
openvswitch-switch
Reading package lists... 0%
** Affects: kvm (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1698846
Title:
[ppc64el] Hugepages w/ split-core freezes cpu
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/kvm/+bug/1698846/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
