Package: gitlab Version: 8.13.11+dfsg1-8 Followup-For: Bug #852787 Dear Maintainer,
we had the same issue with sidekiq. Sidekiq is known to suffer from memory leaks, that's what probably triggers the OOM killer mentioned by Libor Klepáč. We activated the Sidekiq MemoryKiller as documented by GitLab: https://docs.gitlab.com/ce/administration/operations/sidekiq_memory_killer.html As result, the Sidekiq MemoryKiller shuts down the whole gitlab.service. As mentioned by Libor Klepáč the gitlab-sidekiq.service returns after OOM killer due to the "Restart=on-abnormal" directive, but every signal send to the sidekiq process will result in the shutdown of gitlab.service as the gitlab-sidekiq.service is referenced as a dependency in the "BindsTo" directive. In our scenario the Sidekiq MemoryKiller sends a SIGTERM after a 15min grace period or a SIGKILL after additional 30sec. Both signals have the same effect on the gitlab.service, which shuts down completely. A patch to fix the configuration of gitlab.service and gitlab-sidekiq.service is attached. We configured systemd to not tear down the whole gitlab.service when gitlab-sidekiq.service terminates by removing gitlab-sidekiq.service from the "BindsTo" directive and using a softer "Wants" directive for gitlab-sidekiq.service in the gitlab.service instead. The gitlab-sidekiq.service itself is configured with "Restart" directive to restart "always" instead of just "on-abnormal", so that it even restarts when it gets a clean exit code 0 by the SIGTERM of the Sidekiq MemoryKiller. As recommended by GitLab and used in the Omnibus package, we should use the Sidekiq MemoryKiller as a measure against the memory leaks (patch attached). Best -- Patrik Hagedorn System Developer - Data Center Automation ProfitBricks GmbH Greifswalder Str. 207 D - 10405 Berlin Email: patrik.haged...@profitbricks.com URL: https://www.profitbricks.de Sitz der Gesellschaft: Berlin Registergericht: Amtsgericht Charlottenburg, HRB 125506 B Geschäftsführer: Achim Weiss, Matthias Steinberg
>From 1f177089791f0d6bde137182f9aa857ef1a36d89 Mon Sep 17 00:00:00 2001 From: Patrik Hagedorn <patrik.haged...@profitbricks.com> Date: Thu, 28 Sep 2017 17:50:28 +0200 Subject: [PATCH 1/2] Prevent dying sidekiq to shutdown gitlab.service Sidekiq has memory leaks and when it gets terminated by OOM killer or GitLab's Sidekiq MemoryKiller [1] the whole gitlab.service shuts down with it, due to the "BindsTo" directive. Configure systemd to not tear down the whole gitlab.service when gitlab-sidekiq.service terminates by removing gitlab-sidekiq.service from the "BindsTo" directive and using a softer "Wants" directive for gitlab-sidekiq.service in the gitlab.service instead. The gitlab-sidekiq.service is configured with "Restart=always" directive, so that it even restarts when it gets a clean exit code 0 by the SIGTERM of the Sidekiq MemoryKiller. Bug-Debian: https://bugs.debian.org/852787 [1] https://docs.gitlab.com/ce/administration/operations/sidekiq_memory_killer.html --- debian/gitlab.gitlab-sidekiq.service | 2 +- debian/gitlab.service | 3 ++- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/debian/gitlab.gitlab-sidekiq.service b/debian/gitlab.gitlab-sidekiq.service index 2ac84d9..d1e87c0 100644 --- a/debian/gitlab.gitlab-sidekiq.service +++ b/debian/gitlab.gitlab-sidekiq.service @@ -15,7 +15,7 @@ SyslogIdentifier=gitlab-sidekiq ExecStart=/usr/bin/bundle exec sidekiq \ -C /etc/gitlab/sidekiq_queues.yml \ -e $RAILS_ENV -Restart=on-abnormal +Restart=always ## No "Type=notify" support; wait for service to initialise: ExecStartPost=-/bin/sh -c "for i in 4 4 4 4 4 4 4 4; do sleep $i; (ps -h -o command -p $MAINPID | grep -q -P \"sidekiq\s\d\.\d\.\d\") && exit 0; done" diff --git a/debian/gitlab.service b/debian/gitlab.service index 363826b..97b533b 100644 --- a/debian/gitlab.service +++ b/debian/gitlab.service @@ -1,6 +1,7 @@ [Unit] Description=GitLab Services -BindsTo=gitlab-unicorn.service gitlab-sidekiq.service gitlab-mailroom.service gitlab-workhorse.service +Wants=gitlab-sidekiq.service +BindsTo=gitlab-unicorn.service gitlab-mailroom.service gitlab-workhorse.service After=gitlab-unicorn.service gitlab-sidekiq.service gitlab-mailroom.service gitlab-workhorse.service [Service] -- 2.11.0
>From ea18d7aeac0b3760527dd34fd595f50308977442 Mon Sep 17 00:00:00 2001 From: Patrik Hagedorn <patrik.haged...@profitbricks.com> Date: Thu, 28 Sep 2017 18:46:12 +0200 Subject: [PATCH 2/2] Enable Sidekiq MemoryKiller GitLab recommends to enable Sidekiq MemoryKiller [1]. The GitLab Omnibus package uses 1 GB as default threshold [2]. Configure Sidekiq MemoryKiller to kill sidekiq when exceeding the threshold of 1 GB allocated memory. Bug-Debian: https://bugs.debian.org/852787 [1] https://docs.gitlab.com/ce/administration/operations/sidekiq_memory_killer.html [2] https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/attributes/default.rb --- debian/conf/gitlab-debian.conf.example | 1 + 1 file changed, 1 insertion(+) diff --git a/debian/conf/gitlab-debian.conf.example b/debian/conf/gitlab-debian.conf.example index 1ef2d43..e68271a 100644 --- a/debian/conf/gitlab-debian.conf.example +++ b/debian/conf/gitlab-debian.conf.example @@ -1,6 +1,7 @@ # Variables with all small letters are debian specific # Variables with all caps are passed to gitlab app RAILS_ENV=production +SIDEKIQ_MEMORY_KILLER_MAX_RSS=1000000 DB=postgres gitlab_app_root=/usr/share/gitlab gitlab_data_dir=/var/lib/gitlab -- 2.11.0