On 07/21/2020 06:55 PM, Daniel Pearson wrote:
Thanks for posting this, I have some doubts on your explanation along
with some information we received in tickets as well. For clarification
we primarily run web-server VM's.
We do not use swappages, this is set 0:0 within the containers. However,
over a long period of time (100~ days) regardless of the kernel we use,
we get very odd processes that drop into the node level swap regardless
of free memory.
Hi Daniel,
OK, and this is completely unrelated to vSwap.
You don't have vSwap configured on your nodes (don't use swappages for
Containers),
(you can check if vSwap is used on a node at all:
# grep Tswap /proc/meminfo
Tswap: 0 kB)
thus your question relates to a generic memory managing algorithm of a Hardware
Node.
And yes, it's completely fine if mm decides to move some anonymous memory into
swap
(physical!, no vSwap here), and use RAM for more caches instead.
So during these 100 days available memory has never gone to zero, nor
has it gone to the point where this many processes should swap. However,
we get 20GB worth of swap, the majority of which end up being very
active applications such as MySQL, httpd processes,
And here comes your real question - why active applications might get into the
swap?
(leaving apart the question if those applications are really _always_ active,
because if the are not active _for some time_, then no surprise)
One of very common reason is some process - does not matter on the host on in a
Container,
which generates _a lot of_ pagecache.
Most often it's some kind of homebrew backup processes - just because it's
their job to read
files from disk while performing a backup - and thus they generate a lot of
pagecache.
Once again - it's possible such processes are run on the host (so hoster
performs backups)
or in a Container (CT owner performs backups on his own).
And mm does not know if this pagecache is "useful" and will be used by someone
a bit later or not,
and by default considers it as useful. And if your disks are very fast, new
pagecache is generated
so fast that even those application you consider as "active" are not active
enough from algorithm's
point of view and thus pages from these processes might go into swap.
What can be done with such "backup" processes?
1) if those processes are run on the host:
a) rewrite backups not to generate pagecache.
For example:
# dd if=/vz/private/123/root.hdd/root.hds iflag=direct of=- | gzip
(note, i'm not talking now about the consistency, that you must backup
snapshots, not live image, etc.
It's a different theme. Use commercial Virtuozzo backups at last :). )
b) if you cannot rewrite backup software, run backup in a separate memory
cgroup
and limit pagecache for this cgroup.
This is a Virtuozzo specific memcg limit, mainstream does not have it =>
LXC/Proxmox too.
We've implemented this feature especially to fight with such processes
which generate a lot of pagecache by their nature.
memcg::memory.cache.limit_in_bytes
2) if those processes are run by a Container owner:
a) you can limit the io/iops for the CT (thus pagecache is generated slower
and active processes
are considered as active enough not to get into swap)
b) you can disable tcache for this Container memcg::memory.disable_cleancache
(raise your hand if you wish me to explain what tcache is)
c) you can limit the max amount of memory which can be used for pagecache
for this Container
memcg::memory.cache.limit_in_bytes
These are my guesses surely, but this is the most often reason.
<skipped>
"b) if there is free memory on the host, the Container's memory is saved
in a special swap cache in host's RAM and no real write to host's
physical swap occurs"
But how can this be shown or proven? I do not believe this bit functions
at all on the nodes we run.
Yes, i think vSwap does not work on your nodes if no swap is configured for
Containers.
How to check the number of tswap pages at the moment on the Hardware Node
overall
(tswap - is a backend for vSwap, so these are exactly pages which were going to
be put into physical swap from Containers,
but were put into "tswap" - a special cache in Hardware Node RAM)
# grep Tswap /proc/meminfo
If you wish to check if vSwap works at all (i mean - if it avoids physical swap
io if there is free RAM on the node),
* you can take an idle node for the experiment with, say 3X RAM,
* create a Container with, say, X RAM + X vSwap,
* run a memory eater process for 2X RAM in the Container,
* check on the node the io in block device with swap (say, in /proc/diskstats)
Hope that helps.
--
Best regards,
Konstantin Khorenko,
Virtuozzo Linux Kernel Team
_______________________________________________
Users mailing list
Users@openvz.org
https://lists.openvz.org/mailman/listinfo/users
