This is the second morning in a row I come to work to find our gitlab server unwell. The symptoms are:
1. Going to the login page gives me a 500 2. Status seems OK, but stopping fails: root@gitlab:/# gitlab-ctl status run: gitlab-git-http-server: (pid 286) 82345s; down: log: 1s, normally up, want up run: logrotate: (pid 30879) 3123s; down: log: 1s, normally up, want up run: nginx: (pid 308) 82343s; down: log: 1s, normally up, want up run: postgresql: (pid 217) 82349s; down: log: 0s, normally up, want up run: redis: (pid 209) 82351s; down: log: 1s, normally up, want up run: sidekiq: (pid 276) 82346s; down: log: 0s, normally up, want up run: unicorn: (pid 251) 82347s; down: log: 1s, normally up, want up root@gitlab:/# gitlab-ctl stop ok: down: gitlab-git-http-server: 0s, normally up ok: down: logrotate: 0s, normally up ok: down: nginx: 0s, normally up ok: down: postgresql: 0s, normally up timeout: run: redis: (pid 209) 82394s, want down, got TERM timeout: run: sidekiq: (pid 276) 82420s, want down, got TERM ok: down: unicorn: 0s, normally up 3. I see this repeating in the logs (when running `gitlab-ctl tail`): ==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/ci/api/v1/builds/register.json" for 127.0.0.1 at 2015-11-04 08:10:22 +0000 Errno::EAGAIN (Resource temporarily unavailable - connect(2) for /var/opt/gitlab/redis/redis.socket): config/initializers/rack_attack_git_basic_auth.rb:5:in `block in <top (required)>' This is a description of my setup: - GitLab is run in a Docker container, gitlab/gitlab-ce (124cd013a598). - The container's `/etc/gitlab`, `/var/log/gitlab` and `/var/opt/gitlab` are mapped onto folders on an NFS mounted area. (To ensure they get backed up regularly.) - I've configured GitLabCI and have one runner, it's running in a Docker container on the same host, gitlab/gitlab-runner (d3b921e60fc1). - I've configured LDAP authentication against our LDAP server. - I've configured mail sending via GMail (I've not managed to verify - that setup yet though). I'm suspecting the use of NFS for the config/log/data is unusual, but I've not seen anything mentioned about it. I've had this setup for a few weeks now and not seen any problems with it until yesterday. My current fix is to shut down gitlab, remove it, and then run a new instance. Any comments, suggestions, etc are welcome! /M -- Magnus Therning, magnus.thern...@cipherstone.com Cipherstone Technologies AB Theres Svenssons gata 10, 417 55 Gothenburg, Sweden Time is a great teacher, but unfortunately it kills all its pupils. -- Hector Louis Berlioz -- You received this message because you are subscribed to the Google Groups "GitLab" group. To unsubscribe from this group and stop receiving emails from it, send an email to gitlabhq+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/gitlabhq/20151104083838.GA4433%40sobel.cipherstone.com. For more options, visit https://groups.google.com/d/optout.