Sorry, forgot to mention that the host is running Ubuntu 15.10, the lxc version is 1.1.5-0ubuntu0.15.10.3, and the container is running Ubuntu 15.10.
On Thu, Jan 7, 2016 at 12:01 PM, Shimin Guo <shi...@databricks.com> wrote: > Saw the following in syslog from a machine in our CI cluster. It's causing > all kinds of problems for our application. I can't reproduce it reliably, > but our test suite, which brings up and tears down containers many times > during its course, triggers this problem pretty much every time. > Unfortunately since we use EC2, the machine is long gone, and I don't have > much beyond the contents of /var/log/. Can someone derive some hint from > this log as to what is going on? > > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: *** buffer overflow > detected ***: /sbin/cgmanager terminated > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: ======= Backtrace: > ========= > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /lib/x86_64-linux-gnu/libc.so.6(+0x78c4e)[0x7f29189b1c4e] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f2918a51e8c] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /lib/x86_64-linux-gnu/libc.so.6(+0x116e80)[0x7f2918a4fe80] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /lib/x86_64-linux-gnu/libc.so.6(+0x118dd7)[0x7f2918a51dd7] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /lib/x86_64-linux-gnu/libnih.so.1(nih_io_select_fds+0xd0)[0x7f2919162970] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /lib/x86_64-linux-gnu/libnih.so.1(nih_main_loop+0x109)[0x7f2919166f79] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /sbin/cgmanager(+0x5145)[0x562e317f4145] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f2918959a40] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: > /sbin/cgmanager(+0x53d9)[0x562e317f43d9] > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: ======= Memory map: > ======== > > Jan 7 06:59:32 ip-10-97-235-172 cgmanager[801]: 562e317ef000-562e31811000 > r-xp 00000000 ca:01 12884 /sbin/cgmanager > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 562e31a11000-562e31a13000 > r--p 00022000 ca:01 12884 /sbin/cgmanager > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 562e31a13000-562e31a14000 > rw-p 00024000 ca:01 12884 /sbin/cgmanager > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 562e339e7000-562e34426000 > rw-p 00000000 00:00 0 [heap] > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29171a5000-7f29171bb000 > r-xp 00000000 ca:01 395872 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29171bb000-7f29173ba000 > ---p 00016000 ca:01 395872 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29173ba000-7f29173bb000 > r--p 00015000 ca:01 395872 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29173bb000-7f29173bc000 > rw-p 00016000 ca:01 395872 > /lib/x86_64-linux-gnu/libgcc_s.so.1 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29173bc000-7f29173c8000 > r-xp 00000000 ca:01 403839 /lib/x86_64-linux-gnu/ > libnss_files-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29173c8000-7f29175c7000 > ---p 0000c000 ca:01 403839 /lib/x86_64-linux-gnu/ > libnss_files-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29175c7000-7f29175c8000 > r--p 0000b000 ca:01 403839 /lib/x86_64-linux-gnu/ > libnss_files-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29175c8000-7f29175c9000 > rw-p 0000c000 ca:01 403839 /lib/x86_64-linux-gnu/ > libnss_files-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29175c9000-7f29175d4000 > r-xp 00000000 ca:01 403823 /lib/x86_64-linux-gnu/ > libnss_nis-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29175d4000-7f29177d3000 > ---p 0000b000 ca:01 403823 /lib/x86_64-linux-gnu/ > libnss_nis-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29177d3000-7f29177d4000 > r--p 0000a000 ca:01 403823 /lib/x86_64-linux-gnu/ > libnss_nis-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29177d4000-7f29177d5000 > rw-p 0000b000 ca:01 403823 /lib/x86_64-linux-gnu/ > libnss_nis-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29177d5000-7f29177ec000 > r-xp 00000000 ca:01 403845 /lib/x86_64-linux-gnu/ > libnsl-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29177ec000-7f29179eb000 > ---p 00017000 ca:01 403845 /lib/x86_64-linux-gnu/ > libnsl-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29179eb000-7f29179ec000 > r--p 00016000 ca:01 403845 /lib/x86_64-linux-gnu/ > libnsl-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29179ec000-7f29179ed000 > rw-p 00017000 ca:01 403845 /lib/x86_64-linux-gnu/ > libnsl-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29179ed000-7f29179ef000 > rw-p 00000000 00:00 0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29179ef000-7f29179f7000 > r-xp 00000000 ca:01 403841 /lib/x86_64-linux-gnu/ > libnss_compat-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29179f7000-7f2917bf6000 > ---p 00008000 ca:01 403841 /lib/x86_64-linux-gnu/ > libnss_compat-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2917bf6000-7f2917bf7000 > r--p 00007000 ca:01 403841 /lib/x86_64-linux-gnu/ > libnss_compat-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2917bf7000-7f2917bf8000 > rw-p 00008000 ca:01 403841 /lib/x86_64-linux-gnu/ > libnss_compat-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2917bf8000-7f2917c0a000 > r-xp 00000000 ca:01 394903 > /lib/x86_64-linux-gnu/libgpg-error.so.0.15.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2917c0a000-7f2917e09000 > ---p 00012000 ca:01 394903 > /lib/x86_64-linux-gnu/libgpg-error.so.0.15.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2917e09000-7f2917e0a000 > r--p 00011000 ca:01 394903 > /lib/x86_64-linux-gnu/libgpg-error.so.0.15.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2917e0a000-7f2917e0b000 > rw-p 00012000 ca:01 394903 > /lib/x86_64-linux-gnu/libgpg-error.so.0.15.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2917e0b000-7f2917e0e000 > r-xp 00000000 ca:01 403830 /lib/x86_64-linux-gnu/ > libdl-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2917e0e000-7f291800d000 > ---p 00003000 ca:01 403830 /lib/x86_64-linux-gnu/ > libdl-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f291800d000-7f291800e000 > r--p 00002000 ca:01 403830 /lib/x86_64-linux-gnu/ > libdl-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f291800e000-7f291800f000 > rw-p 00003000 ca:01 403830 /lib/x86_64-linux-gnu/ > libdl-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f291800f000-7f29180e7000 > r-xp 00000000 ca:01 394902 > /lib/x86_64-linux-gnu/libgcrypt.so.20.0.3 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29180e7000-7f29182e7000 > ---p 000d8000 ca:01 394902 > /lib/x86_64-linux-gnu/libgcrypt.so.20.0.3 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29182e7000-7f29182e8000 > r--p 000d8000 ca:01 394902 > /lib/x86_64-linux-gnu/libgcrypt.so.20.0.3 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29182e8000-7f29182f1000 > rw-p 000d9000 ca:01 394902 > /lib/x86_64-linux-gnu/libgcrypt.so.20.0.3 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29182f1000-7f2918312000 > r-xp 00000000 ca:01 396567 > /lib/x86_64-linux-gnu/liblzma.so.5.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918312000-7f2918511000 > ---p 00021000 ca:01 396567 > /lib/x86_64-linux-gnu/liblzma.so.5.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918511000-7f2918512000 > r--p 00020000 ca:01 396567 > /lib/x86_64-linux-gnu/liblzma.so.5.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918512000-7f2918513000 > rw-p 00021000 ca:01 396567 > /lib/x86_64-linux-gnu/liblzma.so.5.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918513000-7f291852b000 > r-xp 00000000 ca:01 403834 /lib/x86_64-linux-gnu/ > libpthread-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f291852b000-7f291872b000 > ---p 00018000 ca:01 403834 /lib/x86_64-linux-gnu/ > libpthread-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f291872b000-7f291872c000 > r--p 00018000 ca:01 403834 /lib/x86_64-linux-gnu/ > libpthread-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f291872c000-7f291872d000 > rw-p 00019000 ca:01 403834 /lib/x86_64-linux-gnu/ > libpthread-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f291872d000-7f2918731000 > rw-p 00000000 00:00 0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918731000-7f2918738000 > r-xp 00000000 ca:01 403846 /lib/x86_64-linux-gnu/ > librt-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918738000-7f2918937000 > ---p 00007000 ca:01 403846 /lib/x86_64-linux-gnu/ > librt-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918937000-7f2918938000 > r--p 00006000 ca:01 403846 /lib/x86_64-linux-gnu/ > librt-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918938000-7f2918939000 > rw-p 00007000 ca:01 403846 /lib/x86_64-linux-gnu/ > librt-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918939000-7f2918af9000 > r-xp 00000000 ca:01 403850 /lib/x86_64-linux-gnu/ > libc-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918af9000-7f2918cf9000 > ---p 001c0000 ca:01 403850 /lib/x86_64-linux-gnu/ > libc-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918cf9000-7f2918cfd000 > r--p 001c0000 ca:01 403850 /lib/x86_64-linux-gnu/ > libc-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918cfd000-7f2918cff000 > rw-p 001c4000 ca:01 403850 /lib/x86_64-linux-gnu/ > libc-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918cff000-7f2918d03000 > rw-p 00000000 00:00 0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918d03000-7f2918d4d000 > r-xp 00000000 ca:01 395303 > /lib/x86_64-linux-gnu/libdbus-1.so.3.14.3 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918d4d000-7f2918f4d000 > ---p 0004a000 ca:01 395303 > /lib/x86_64-linux-gnu/libdbus-1.so.3.14.3 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918f4d000-7f2918f4e000 > r--p 0004a000 ca:01 395303 > /lib/x86_64-linux-gnu/libdbus-1.so.3.14.3 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918f4e000-7f2918f4f000 > rw-p 0004b000 ca:01 395303 > /lib/x86_64-linux-gnu/libdbus-1.so.3.14.3 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918f4f000-7f2918f58000 > r-xp 00000000 ca:01 395230 > /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2918f58000-7f2919157000 > ---p 00009000 ca:01 395230 > /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919157000-7f2919158000 > r--p 00008000 ca:01 395230 > /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919158000-7f2919159000 > rw-p 00009000 ca:01 395230 > /lib/x86_64-linux-gnu/libnih-dbus.so.1.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919159000-7f2919171000 > r-xp 00000000 ca:01 396370 > /lib/x86_64-linux-gnu/libnih.so.1.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919171000-7f2919370000 > ---p 00018000 ca:01 396370 > /lib/x86_64-linux-gnu/libnih.so.1.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919370000-7f2919371000 > r--p 00017000 ca:01 396370 > /lib/x86_64-linux-gnu/libnih.so.1.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919371000-7f2919372000 > rw-p 00018000 ca:01 396370 > /lib/x86_64-linux-gnu/libnih.so.1.0.0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919372000-7f2919396000 > r-xp 00000000 ca:01 403833 /lib/x86_64-linux-gnu/ > ld-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f29194ff000-7f2919504000 > rw-p 00000000 00:00 0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919504000-7f2919581000 > r-xp 00000000 ca:01 394592 > /lib/x86_64-linux-gnu/libsystemd.so.0.10.2 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919581000-7f2919584000 > r--p 0007c000 ca:01 394592 > /lib/x86_64-linux-gnu/libsystemd.so.0.10.2 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919584000-7f2919585000 > rw-p 0007f000 ca:01 394592 > /lib/x86_64-linux-gnu/libsystemd.so.0.10.2 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919585000-7f2919587000 > rw-p 00000000 00:00 0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919592000-7f2919595000 > rw-p 00000000 00:00 0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919595000-7f2919596000 > r--p 00023000 ca:01 403833 /lib/x86_64-linux-gnu/ > ld-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919596000-7f2919597000 > rw-p 00024000 ca:01 403833 /lib/x86_64-linux-gnu/ > ld-2.21.so > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7f2919597000-7f2919598000 > rw-p 00000000 00:00 0 > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7ffeed99e000-7ffeed9bf000 > rw-p 00000000 00:00 0 [stack] > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7ffeed9ce000-7ffeed9d0000 > r--p 00000000 00:00 0 [vvar] > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: 7ffeed9d0000-7ffeed9d2000 > r-xp 00000000 00:00 0 [vdso] > > Jan 7 06:59:33 ip-10-97-235-172 cgmanager[801]: > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 > [vsyscall] > > Jan 7 06:59:34 ip-10-97-235-172 systemd[1]: cgmanager.service: Main > process exited, code=dumped, status=6/ABRT > > Jan 7 06:59:34 ip-10-97-235-172 systemd[1]: cgmanager.service: Unit > entered failed state. > Jan 7 06:59:34 ip-10-97-235-172 systemd[1]: cgmanager.service: Failed > with result 'core-dump'. >
_______________________________________________ lxc-users mailing list lxc-users@lists.linuxcontainers.org http://lists.linuxcontainers.org/listinfo/lxc-users
