Last night a user on our Gitlab instance reported "I tried creating a local
branch based on a copy of a branch on the server and pushing that back to
the server tonight, which I've done many times before, and got GitLab: An
unexpected error occurred (redis-cli returned 1)
"
When I attempt to restart Gitlab I see the following in the Sidekiq Log:
````
2016-02-08T12:46:24.395Z 7332 TID-owzis9auo INFO: Booting Sidekiq 3.3.0
with redis options {:url=>"unix:/var/run/redis/redis.sock",
:namespace=>"resque:gitlab"}
Permission denied - connect(2) for /var/run/redis/redis.sock
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/connection/ruby.rb:180:in
`connect_nonblock'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/connection/ruby.rb:180:in
`connect'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/connection/ruby.rb:209:in
`connect'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:322:in
`establish_connection'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:94:in
`block in connect'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:279:in
`with_reconnect'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:93:in
`connect'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:350:in
`ensure_connected'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:207:in
`block in process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:292:in
`logging'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:206:in
`process'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:112:in
`call'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis.rb:789:in
`block in get'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis.rb:37:in
`block in synchronize'
/usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis.rb:37:in
`synchronize'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis.rb:788:in
`get'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in
`call_with_namespace'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in
`method_missing'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidetiq-0.6.3/lib/sidetiq/schedulable.rb:63:in
`block in get_schedulable_key'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:64:in
`block (2 levels) in with'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:63:in
`handle_interrupt'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:63:in
`block in with'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:60:in
`handle_interrupt'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:60:in
`with'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidetiq-0.6.3/lib/sidetiq/schedulable.rb:63:in
`get_schedulable_key'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidetiq-0.6.3/lib/sidetiq/schedulable.rb:46:in
`recurrence'
/home/git/gitlab/app/workers/stuck_ci_builds_worker.rb:7:in
`<class:StuckCiBuildsWorker>'
/home/git/gitlab/app/workers/stuck_ci_builds_worker.rb:1:in `<top
(required)>'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
`require'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
`block in require'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:232:in
`load_dependency'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
`require'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:348:in
`require_or_load'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:307:in
`depend_on'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:225:in
`require_dependency'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:468:in
`block (2 levels) in eager_load!'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:467:in
`each'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:467:in
`block in eager_load!'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:465:in
`each'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:465:in
`eager_load!'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:346:in
`eager_load!'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/application/finisher.rb:58:in
`each'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/application/finisher.rb:58:in
`block in <module:Finisher>'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/initializable.rb:30:in
`instance_exec'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/initializable.rb:30:in
`run'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/initializable.rb:55:in
`block in run_initializers'
/usr/local/lib/ruby/2.1.0/tsort.rb:226:in `block in tsort_each'
/usr/local/lib/ruby/2.1.0/tsort.rb:348:in `block (2 levels) in
each_strongly_connected_component'
/usr/local/lib/ruby/2.1.0/tsort.rb:427:in
`each_strongly_connected_component_from'
/usr/local/lib/ruby/2.1.0/tsort.rb:347:in `block in
each_strongly_connected_component'
/usr/local/lib/ruby/2.1.0/tsort.rb:345:in `each'
/usr/local/lib/ruby/2.1.0/tsort.rb:345:in `call'
/usr/local/lib/ruby/2.1.0/tsort.rb:345:in
`each_strongly_connected_component'
/usr/local/lib/ruby/2.1.0/tsort.rb:224:in `tsort_each'
/usr/local/lib/ruby/2.1.0/tsort.rb:205:in `tsort_each'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/initializable.rb:54:in
`run_initializers'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/application.rb:300:in
`initialize!'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/railtie.rb:194:in
`public_send'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/railtie.rb:194:in
`method_missing'
/home/git/gitlab/config/environment.rb:5:in `<top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
`require'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
`block in require'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:232:in
`load_dependency'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
`require'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/cli.rb:236:in
`boot_system'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/cli.rb:50:in
`run'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.0/bin/sidekiq:8:in
`<top (required)>'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/sidekiq:23:in `load'
/home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/sidekiq:23:in `<main>'
````
I've restart Redis with no errors:
````
[1378] 08 Feb 07:38:54.554 # Server started, Redis version 2.8.4
[1378] 08 Feb 07:38:54.554 # WARNING overcommit_memory is set to 0!
Background save may fail under low memory condition. To fix this issue add
'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the
command 'sysctl vm.overcommit_memory=1' for this to take effect.
[1378] 08 Feb 07:38:54.729 * DB loaded from disk: 0.175 seconds
[1378] 08 Feb 07:38:54.730 * The server is now ready to accept connections
at /var/run/redis/redis.sock
````
Our production.log shows:
````
Errno::EACCES (Permission denied - connect(2) for
/var/run/redis/redis.sock):
config/initializers/rack_attack_git_basic_auth.rb:5:in `block in <top
(required)>'
````
We have not made any configuration changes or updates to the server since
we last updated it to 8.2.3. Here is our environment check:
````
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.1.6p336
Gem Version: 2.2.3
Bundler Version:1.10.6
Rake Version: 10.4.2
Sidekiq Version:3.3.0
GitLab information
Version: 8.2.3
Revision: 19daba8
Directory: /home/git/gitlab
DB Adapter: postgresql
URL: https://gitlab.domain.com
HTTP Clone URL: https://gitlab.domain.com/some-group/some-project.git
SSH Clone URL: g...@gitlab.domain.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: xxxx
GitLab Shell
Version: 2.6.8
Repositories: /home/git/repositories/
Hooks: /home/git/gitlab-shell/hooks/
Git: /usr/local/bin/git
````
Our VM has automatic updates for security patches turned on, everything
else is manually updated. We have no done a manual update in around 1 week.
Any ideas? Our server is currently show 500 errors from the browser and
timing out from the cli when trying to clone.
--
You received this message because you are subscribed to the Google Groups
"GitLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to gitlabhq+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/gitlabhq/e8991d7e-95a0-483d-aba6-08a20e95596e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
