Public bug reported:
(Sorry I'm not sure exactly what package to report this against - kernel
perhaps? libvirt is what I was using to replicate the problem)
Host platform: ubuntu 14.04 amd64, Mac Mini, 16GB RAM.
Short version: create an LXC domain with memtune > swap_hard_limit set
in the XML:
<domain type='lxc'>
<name>gold-lxc-20140717</name>
<uuid>b2a02d49-bb1e-4aec-81d1-58910892780e</uuid>
<memory unit='KiB'>327680</memory>
<currentMemory unit='KiB'>327680</currentMemory>
<memtune>
<swap_hard_limit unit='KiB'>131072</swap_hard_limit>
</memtune>
...
(full version at end of this report)
Now try to start it:
$ virsh -c lxc: start gold-lxc-20140717error: Failed to start domain
gold-lxc-20140717
error: internal error: guest failed to start: Unable to write to
'/sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.memsw.limit_in_bytes':
No such file or directory
The reason this matters is because otherwise the LXC memory limit
applies only to real RAM used. If the guest exceeds this it can still
use as much swap space as it likes, and is therefore effectively
unlimited (and can happily DoS the swap disk).
Long version:
I created an ubuntu 14.04 i386 VM image using python-vmbuilder,
loopback-mounted it with qemu-nbd, and rsync'd it to create a root
filesystem for an LXC guest.
Then defined a guest using libvirt XML and started it using "virsh -c
lxc: start <domain>" (as per XML at end but without the <memtune>
section). It starts successfully, networking is fine, I can get a
console etc.
Now, the libvirt XML description says the guest's memory limit is 320MB:
<memory unit='KiB'>327680</memory>
<currentMemory unit='KiB'>327680</currentMemory>
and indeed the cgroups setting has been set:
$ cat
/sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.limit_in_bytes
335544320
However inside the guest I can happily allocate as much memory as I
like, up to just under 4GB, which is the limit for a 32-bit guest.
Here's the test program I ran in the guest (usemem.c):
#include <stdlib.h>
#include <unistd.h>
#include <stdio.h>
int main(void)
{
char *p;
int i,j;
int ok=0, fail=0;
for (i=0; i<4096; i++) {
p = malloc(1024*1024);
if (p) {
ok++;
for (j=0; j<1024*1024; j++)
p[j] = rand();
}
else
fail++;
}
fprintf(stderr, "Done: %d ok, %d fail\n", ok, fail);
sleep(600);
return fail ? 1 : 0;
}
Result from running:
Done: 4076 ok, 20 fail
View from the host:
nsrc@kit1:~/workshop-kit$ ps auxwww | grep usemem | grep -v grep
nsrc 10506 96.1 1.3 4192152 224776 ? S+ 14:41 0:55 ./usemem
$ cat
/sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.limit_in_bytes
335544320
$ cat
/sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.max_usage_in_bytes
335544320
$ cat
/sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/memory.usage_in_bytes
292331520
You can see there's definitely 4GB in use by this process, and yet the
cgroup thinks less than 280MB is in use, which is below the 320MB limit.
However if you look at swap usage in the host while the memory suck
program is running:
$ free
total used free shared buffers cached
Mem: 16338300 3066952 13271348 1684 135512 1489268
-/+ buffers/cache: 1442172 14896128
Swap: 16678908 3971248 12707660
and after it has terminated:
$ free
total used free shared buffers cached
Mem: 16338300 2774440 13563860 1684 135544 1489188
-/+ buffers/cache: 1149708 15188592
Swap: 16678908 5484 16673424
i.e. the LXC guest used nearly 4GB of swap, and then gave it up when it
terminated.
Additional info:
cgroup view from inside the guest:
$ cat /proc/self/cgroup
11:name=systemd:/
10:hugetlb:/
9:perf_event:/machine/gold-lxc-20140717.libvirt-lxc
8:blkio:/machine/gold-lxc-20140717.libvirt-lxc
7:freezer:/machine/gold-lxc-20140717.libvirt-lxc
6:devices:/machine/gold-lxc-20140717.libvirt-lxc
5:memory:/machine/gold-lxc-20140717.libvirt-lxc
4:cpuacct:/machine/gold-lxc-20140717.libvirt-lxc
3:cpu:/machine/gold-lxc-20140717.libvirt-lxc
2:cpuset:/machine/gold-lxc-20140717.libvirt-lxc
cgroup settings visible in the host:
$ ls /sys/fs/cgroup/memory/machine/gold-lxc-20140717.libvirt-lxc/
cgroup.clone_children memory.limit_in_bytes
cgroup.event_control memory.max_usage_in_bytes
cgroup.procs memory.move_charge_at_immigrate
memory.failcnt memory.numa_stat
memory.force_empty memory.oom_control
memory.kmem.failcnt memory.pressure_level
memory.kmem.limit_in_bytes memory.soft_limit_in_bytes
memory.kmem.max_usage_in_bytes memory.stat
memory.kmem.slabinfo memory.swappiness
memory.kmem.tcp.failcnt memory.usage_in_bytes
memory.kmem.tcp.limit_in_bytes memory.use_hierarchy
memory.kmem.tcp.max_usage_in_bytes notify_on_release
memory.kmem.tcp.usage_in_bytes tasks
memory.kmem.usage_in_bytes
And here's the full XML as promised:
<domain type='lxc'>
<name>gold-lxc-20140717</name>
<uuid>b2a02d49-bb1e-4aec-81d1-58910892780e</uuid>
<memory unit='KiB'>327680</memory>
<currentMemory unit='KiB'>327680</currentMemory>
<memtune>
<swap_hard_limit unit='KiB'>131072</swap_hard_limit>
</memtune>
<vcpu placement='static'>1</vcpu>
<resource>
<partition>/machine</partition>
</resource>
<os>
<type arch='x86_64'>exe</type>
<init>/sbin/init</init>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/lib/libvirt/libvirt_lxc</emulator>
<filesystem type='mount' accessmode='passthrough'>
<source dir='/data1/lxc/gold-20140717/rootfs'/>
<target dir='/'/>
</filesystem>
<interface type='bridge'>
<mac address='52:54:5d:00:0a:88'/>
<source bridge='br-lan'/>
</interface>
<console type='pty'>
<target type='lxc' port='0'/>
</console>
</devices>
<seclabel type='none'/>
</domain>
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: libvirt-bin 1.2.2-0ubuntu13.1.1
ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
Uname: Linux 3.13.0-32-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
Date: Fri Jul 25 14:29:54 2014
InstallationDate: Installed on 2014-07-16 (8 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64
(20140416.2)
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Permission
denied: '/etc/libvirt/qemu.conf']
** Affects: libvirt (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug trusty
--
You received this bug notification because you are a member of Ubuntu
Server Team, which is subscribed to libvirt in Ubuntu.
https://bugs.launchpad.net/bugs/1348688
Title:
LXC does not limit swap usage (memory.memsw.limit_in_bytes missing)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1348688/+subscriptions
--
Ubuntu-server-bugs mailing list
Ubuntu-server-bugs@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
