On Wed, May 22, 2019 at 05:16:38PM -0600, Jim Fehlig wrote:
> Hi All,
> 
> I recently received an internal bug report of VM "crashing" due to hitting
> thread limits. Seems there was an assert in pthread_create within the VM
> when hitting the limit enforced by pids controller on the host
> 
> Apr 28 07:45:46 lpcomp02007 kernel: cgroup: fork rejected by pids controller
> in /machine.slice/machine-qemu\x2d90028\x2dinstance\x2d0000634b.scope
> 
> The user has TasksMax set to infinity in machine.slice, but apparently that
> is not inherited by child scopes and appears to be hardcoded to 16384
> 
> https://github.com/systemd/systemd/blob/51aba17b88617515e037e8985d3a4ea871ac47fe/src/machine/machined-dbus.c#L1344
> 
> The TasksMax property can be set when creating the machine as is done in the
> attached proof of concept patch. Question is whether this should be a
> tunable? My initial thought when seeing the report was TasksMax could be
> calculated based on number of vcpus, iothreads, emulator threads, etc. But
> it appears that could be quite tricky. The following mail thread describes
> the basic scenario encountered by my user
> 
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-March/008174.html
> 
> As you can see, many rbd images attached to a VM can result in an awful lot
> of threads. 300 images could result in 720K threads! We could punt and set
> the limit to infinity, but it exists for a reason - fork bomb prevention. A
> potential compromise between a hardcoded value and per-VM tunable is a
> driver tunable in qemu.conf. If a per-VM tunable is preferred, suggestions
> on where to place it and what to call it would be much appreciated :-).

Yeah, RBD is problematic as you can't predict how many threads it will
use.

We currently have a "max_processes" stting in qemu.conf for the ulimit
base process limit. This applies to the user as a whole though, not the
cgroup.

On Fedora we don't seem to have any "tasks_max" cgroup setting or TasksMax
systemd setting, at least when running with cgroups v1, so we can't set that
unconditionally.

I'd be inclined to have a new qemu.conf setting "max_tasks". If this is
set to 0, then we should just set TasksMax to infinity, otherwise honour
the setting.


> >From 0583ee3b26b2ee43efe8d25226eceb8547400d97 Mon Sep 17 00:00:00 2001
> From: Jim Fehlig <jfeh...@suse.com>
> Date: Wed, 22 May 2019 17:12:14 -0600
> Subject: [PATCH] systemd: set TasksMax when calling CreateMachine
> 
> An example of how to set TasksMax when creating a scope for a machine.
> 
> Signed-off-by: Jim Fehlig <jfeh...@suse.com>
> ---
>  src/util/virsystemd.c | 10 ++++++----
>  1 file changed, 6 insertions(+), 4 deletions(-)
> 
> diff --git a/src/util/virsystemd.c b/src/util/virsystemd.c
> index 3f03e3bd63..6177447bdb 100644
> --- a/src/util/virsystemd.c
> +++ b/src/util/virsystemd.c
> @@ -341,10 +341,11 @@ int virSystemdCreateMachine(const char *name,
>                                (unsigned int)pidleader,
>                                NULLSTR_EMPTY(rootdir),
>                                nnicindexes, nicindexes,
> -                              3,
> +                              4,
>                                "Slice", "s", slicename,
>                                "After", "as", 1, "libvirtd.service",
> -                              "Before", "as", 1, 
> "virt-guest-shutdown.target") < 0)
> +                              "Before", "as", 1, 
> "virt-guest-shutdown.target",
> +                              "TasksMax", "t", UINT64_C(32768)) < 0)
>              goto cleanup;
>  
>          if (error.level == VIR_ERR_ERROR) {
> @@ -382,10 +383,11 @@ int virSystemdCreateMachine(const char *name,
>                                iscontainer ? "container" : "vm",
>                                (unsigned int)pidleader,
>                                NULLSTR_EMPTY(rootdir),
> -                              3,
> +                              4,
>                                "Slice", "s", slicename,
>                                "After", "as", 1, "libvirtd.service",
> -                              "Before", "as", 1, 
> "virt-guest-shutdown.target") < 0)
> +                              "Before", "as", 1, 
> "virt-guest-shutdown.target",
> +                              "TasksMax", "t", UINT64_C(32768)) < 0)
>              goto cleanup;
>      }
>  
> -- 
> 2.21.0
> 

> --
> libvir-list mailing list
> libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

Reply via email to