Here I attach the 2 files with the output of those commands: one.txt is right after I copied several gigabytes of files to that container (using samba), two.txt is after I removed it.
It looks like the most relevant number that is changed is `total_cache` On 27 September 2017 at 02:24, Stéphane Graber <[email protected]> wrote: > Hi, > > This sounds like a lxcfs issue. > > Can you file a bug at https://github.com/lxc/lxcfs or find one which > matches your symptoms. > > We'll want at least: > > - /proc/meminfo from the container > - /sys/fs/cgroup/memory/lxc/CONTAINER/memory.usage_in_bytes from the host > - /sys/fs/cgroup/memory/lxc/CONTAINER/memory.stat from the host > > That should let us track down where the memory usage comes from and what > may be wrong with it. > > On Wed, Sep 20, 2017 at 01:18:01PM +1200, Ivan Kurnosov wrote: > > Hi, > > > > there is a server that currently runs ~100 containers. > > > > One of those containers is a subject of my interest. > > > > Brief details about the container: it runs ubuntu xenial, and it's a tiny > > file server (samba based) with near to no traffic at all. > > > > I have found that after you upload files to that server, the available > > memory size is decreased (while the "buff/cache" size stays at 0). And if > > you remove the just uploaded files - the memory consumption drops to the > > same value as it was before uploading. > > > > Here is a output of the top (sorted by resident memory size, processes > with > > more than 500kib rss): > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > > 48 root 20 0 52048 18880 14428 S 0.0 0.9 0:10.11 > > /lib/systemd/systemd-journald > > 18609 www-data 20 0 349208 15404 7516 S 0.0 0.7 0:13.72 > > /usr/sbin/smbd -D > > 7176 www-data 20 0 345500 10720 6720 S 6.7 0.5 0:06.91 > > /usr/sbin/smbd -D > > 25124 root 20 0 340104 9624 6744 S 0.0 0.5 0:00.12 > > /usr/sbin/smbd -D > > 37541 root 20 0 344828 8012 4520 S 0.0 0.4 0:02.36 > > /usr/sbin/smbd -D > > 15593 root 20 0 344352 6368 3444 S 0.0 0.3 0:00.39 > > /usr/sbin/smbd -D > > 2450 root 20 0 336636 4072 1520 S 0.0 0.2 0:06.09 > > /usr/sbin/smbd -D > > 25401 root 20 0 40560 3728 3112 R 0.3 0.2 0:00.49 top > > 2447 root 20 0 336636 3528 976 S 0.0 0.2 0:04.30 > > /usr/sbin/smbd -D > > 25287 root 20 0 19972 3044 2872 S 0.0 0.1 0:00.01 bash > > 2476 root 20 0 238728 2944 1336 S 0.0 0.1 0:28.52 > > /usr/sbin/nmbd -D > > 25271 ivan 20 0 21328 2784 2764 S 0.0 0.1 0:00.04 -bash > > 24250 root 20 0 858936 2616 0 S 0.0 0.1 0:01.98 > > /usr/sbin/collectd > > 2448 root 20 0 426848 2504 20 S 0.3 0.1 0:01.65 > > /usr/sbin/smbd -D > > 1 root 20 0 37884 2488 1676 S 0.0 0.1 0:17.13 > > /sbin/init > > 25285 root 20 0 51660 2404 2400 S 0.0 0.1 0:00.00 sudo > su > > 25270 ivan 20 0 95368 2172 1960 S 0.0 0.1 0:00.24 sshd: > > ivan@pts/0 > > 25286 root 20 0 51008 1908 1908 S 0.0 0.1 0:00.00 su > > 8041 zabbix 20 0 95520 1680 1512 S 0.0 0.1 0:02.10 > > /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec] > > 25240 root 20 0 95368 1620 1572 S 0.0 0.1 0:00.02 sshd: > > ivan [priv] > > 145 message+ 20 0 42892 1164 872 S 0.0 0.1 0:01.55 > > /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile > > --systemd-activation > > 6453 www-data 20 0 125348 1152 656 S 0.0 0.1 0:32.26 > nginx: > > worker process > > 20811 postfix 20 0 67640 1136 656 S 0.0 0.1 0:00.86 qmgr > -l > > -t unix -u > > 8038 zabbix 20 0 95520 1084 880 S 0.0 0.1 0:01.04 > > /usr/sbin/zabbix_agentd: listener #1 [waiting for connection] > > 8039 zabbix 20 0 95520 972 768 S 0.0 0.0 0:01.05 > > /usr/sbin/zabbix_agentd: listener #2 [waiting for connection] > > 142 root 20 0 27732 924 636 S 0.0 0.0 0:05.32 > > /usr/sbin/cron -f > > 8040 zabbix 20 0 95520 872 668 S 0.0 0.0 0:01.07 > > /usr/sbin/zabbix_agentd: listener #3 [waiting for connection] > > 8037 zabbix 20 0 93444 728 628 S 0.0 0.0 0:14.57 > > /usr/sbin/zabbix_agentd: collector [idle 1 sec] > > 6462 www-data 20 0 125348 500 0 S 0.0 0.0 0:32.14 > nginx: > > worker process > > > > > > > > As you can see - the cumulative RSS is could barely get to the 100mb. > > > > While this is what `free` returns: > > > > # free -m > > total used free shared buff/cache > > available > > Mem: 2048 1785 261 1750 0 > > 261 > > Swap: 512 14 497 > > > > > > So, it clearly states about 85% of ram is occupied. > > > > `slabtop` (due to cgroup limitations?) does not work: > > > > # slabtop > > fopen /proc/slabinfo: Permission denied > > > > > > But if I clear the system caches on the host > > > > echo 3 > /proc/sys/vm/drop_caches > > > > > > the container memory consumption drops to the expected <100mb. > > > > So the question, how to monitor the memory consumption from the container > > reliably? And why does `free` count caches as used memory inside > container? > > > > -- > > With best regards, Ivan Kurnosov > > > _______________________________________________ > > lxc-users mailing list > > [email protected] > > http://lists.linuxcontainers.org/listinfo/lxc-users > > > -- > Stéphane Graber > Ubuntu developer > http://www.ubuntu.com > > _______________________________________________ > lxc-users mailing list > [email protected] > http://lists.linuxcontainers.org/listinfo/lxc-users > -- With best regards, Ivan Kurnosov
container# cat /proc/meminfo MemTotal: 2097152 kB MemFree: 36 kB MemAvailable: 36 kB Buffers: 0 kB Cached: 12 kB SwapCached: 0 kB Active: 24 kB Inactive: 64 kB Active(anon): 16 kB Inactive(anon): 60 kB Active(file): 8 kB Inactive(file): 4 kB Unevictable: 0 kB Mlocked: 29112 kB SwapTotal: 524288 kB SwapFree: 513980 kB Dirty: 79564 kB Writeback: 0 kB AnonPages: 19891352 kB Mapped: 4096016 kB Shmem: 1521488 kB Slab: 0 kB SReclaimable: 0 kB SUnreclaim: 0 kB KernelStack: 111328 kB PageTables: 525296 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 76478396 kB Committed_AS: 48821700 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 14186496 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 863168 kB DirectMap2M: 52568064 kB DirectMap1G: 82837504 kB host# cat /sys/fs/cgroup/memory/lxc/CONTAINER/memory.usage_in_bytes 2147471360 host# cat /sys/fs/cgroup/memory/lxc/CONTAINER/memory.stat cache 4096 rss 77824 rss_huge 0 mapped_file 4096 dirty 0 writeback 0 swap 540672 pgpgin 6877 pgpgout 6857 pgfault 8312 pgmajfault 95 inactive_anon 61440 active_anon 16384 inactive_file 0 active_file 4096 unevictable 0 hierarchical_memory_limit 2147483648 hierarchical_memsw_limit 2684354560 total_cache 2107715584 total_rss 39424000 total_rss_huge 6291456 total_mapped_file 17145856 total_dirty 0 total_writeback 0 total_swap 10444800 total_pgpgin 1722046 total_pgpgout 1210617 total_pgfault 2085946 total_pgmajfault 1399 total_inactive_anon 29151232 total_active_anon 19386368 total_inactive_file 2026012672 total_active_file 72556544 total_unevictable 32768
container# cat /proc/meminfo MemTotal: 2097152 kB MemFree: 1922572 kB MemAvailable: 1922572 kB Buffers: 0 kB Cached: 4 kB SwapCached: 0 kB Active: 20 kB Inactive: 64 kB Active(anon): 16 kB Inactive(anon): 64 kB Active(file): 4 kB Inactive(file): 0 kB Unevictable: 0 kB Mlocked: 29112 kB SwapTotal: 524288 kB SwapFree: 513860 kB Dirty: 4412 kB Writeback: 0 kB AnonPages: 18971936 kB Mapped: 4089980 kB Shmem: 1521992 kB Slab: 0 kB SReclaimable: 0 kB SUnreclaim: 0 kB KernelStack: 111152 kB PageTables: 515284 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 76478396 kB Committed_AS: 47856592 kB VmallocTotal: 34359738367 kB VmallocUsed: 0 kB VmallocChunk: 0 kB HardwareCorrupted: 0 kB AnonHugePages: 13834240 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 0 HugePages_Free: 0 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 863168 kB DirectMap2M: 52568064 kB DirectMap1G: 82837504 kB host# cat /sys/fs/cgroup/memory/lxc/CONTAINER/memory.usage_in_bytes 178917376 host# cat /sys/fs/cgroup/memory/lxc/CONTAINER/memory.stat cache 4096 rss 81920 rss_huge 0 mapped_file 4096 dirty 0 writeback 0 swap 536576 pgpgin 6878 pgpgout 6857 pgfault 8312 pgmajfault 95 inactive_anon 65536 active_anon 16384 inactive_file 0 active_file 4096 unevictable 0 hierarchical_memory_limit 2147483648 hierarchical_memsw_limit 2684354560 total_cache 139726848 total_rss 39190528 total_rss_huge 6291456 total_mapped_file 18042880 total_dirty 0 total_writeback 0 total_swap 10674176 total_pgpgin 1723052 total_pgpgout 1692146 total_pgfault 2086960 total_pgmajfault 1433 total_inactive_anon 27209728 total_active_anon 21180416 total_inactive_file 56582144 total_active_file 73912320 total_unevictable 32768
_______________________________________________ lxc-users mailing list [email protected] http://lists.linuxcontainers.org/listinfo/lxc-users
