Seems there was one update last night which removed git from the redis
group. Adding it back fixed this issue.
On Monday, February 8, 2016 at 8:18:25 AM UTC-5, James Riordon wrote:
>
> Last night a user on our Gitlab instance reported "I tried creating a
> local branch based on a copy of a branch on the server and pushing that
> back to the server tonight, which I've done many times before, and got
> GitLab: An unexpected error occurred (redis-cli returned 1)
> "
>
> When I attempt to restart Gitlab I see the following in the Sidekiq Log:
>
> ````
> 2016-02-08T12:46:24.395Z 7332 TID-owzis9auo INFO: Booting Sidekiq 3.3.0
> with redis options {:url=>"unix:/var/run/redis/redis.sock",
> :namespace=>"resque:gitlab"}
> Permission denied - connect(2) for /var/run/redis/redis.sock
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/connection/ruby.rb:180:in
>
> `connect_nonblock'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/connection/ruby.rb:180:in
>
> `connect'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/connection/ruby.rb:209:in
>
> `connect'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:322:in
>
> `establish_connection'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:94:in
>
> `block in connect'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:279:in
>
> `with_reconnect'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:93:in
>
> `connect'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:350:in
>
> `ensure_connected'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:207:in
>
> `block in process'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:292:in
>
> `logging'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:206:in
>
> `process'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis/client.rb:112:in
>
> `call'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis.rb:789:in
>
> `block in get'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis.rb:37:in
> `block in synchronize'
> /usr/local/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis.rb:37:in
> `synchronize'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-3.2.1/lib/redis.rb:788:in
>
> `get'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:393:in
>
> `call_with_namespace'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/redis-namespace-1.5.2/lib/redis/namespace.rb:290:in
>
> `method_missing'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidetiq-0.6.3/lib/sidetiq/schedulable.rb:63:in
>
> `block in get_schedulable_key'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:64:in
>
> `block (2 levels) in with'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:63:in
>
> `handle_interrupt'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:63:in
>
> `block in with'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:60:in
>
> `handle_interrupt'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/connection_pool-2.2.0/lib/connection_pool.rb:60:in
>
> `with'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidetiq-0.6.3/lib/sidetiq/schedulable.rb:63:in
>
> `get_schedulable_key'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidetiq-0.6.3/lib/sidetiq/schedulable.rb:46:in
>
> `recurrence'
> /home/git/gitlab/app/workers/stuck_ci_builds_worker.rb:7:in
> `<class:StuckCiBuildsWorker>'
> /home/git/gitlab/app/workers/stuck_ci_builds_worker.rb:1:in `<top
> (required)>'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
>
> `require'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
>
> `block in require'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:232:in
>
> `load_dependency'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
>
> `require'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:348:in
>
> `require_or_load'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:307:in
>
> `depend_on'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:225:in
>
> `require_dependency'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:468:in
>
> `block (2 levels) in eager_load!'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:467:in
>
> `each'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:467:in
>
> `block in eager_load!'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:465:in
>
> `each'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:465:in
>
> `eager_load!'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/engine.rb:346:in
>
> `eager_load!'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/application/finisher.rb:58:in
>
> `each'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/application/finisher.rb:58:in
>
> `block in <module:Finisher>'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/initializable.rb:30:in
>
> `instance_exec'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/initializable.rb:30:in
>
> `run'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/initializable.rb:55:in
>
> `block in run_initializers'
> /usr/local/lib/ruby/2.1.0/tsort.rb:226:in `block in tsort_each'
> /usr/local/lib/ruby/2.1.0/tsort.rb:348:in `block (2 levels) in
> each_strongly_connected_component'
> /usr/local/lib/ruby/2.1.0/tsort.rb:427:in
> `each_strongly_connected_component_from'
> /usr/local/lib/ruby/2.1.0/tsort.rb:347:in `block in
> each_strongly_connected_component'
> /usr/local/lib/ruby/2.1.0/tsort.rb:345:in `each'
> /usr/local/lib/ruby/2.1.0/tsort.rb:345:in `call'
> /usr/local/lib/ruby/2.1.0/tsort.rb:345:in
> `each_strongly_connected_component'
> /usr/local/lib/ruby/2.1.0/tsort.rb:224:in `tsort_each'
> /usr/local/lib/ruby/2.1.0/tsort.rb:205:in `tsort_each'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/initializable.rb:54:in
>
> `run_initializers'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/application.rb:300:in
>
> `initialize!'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/railtie.rb:194:in
>
> `public_send'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/railties-4.1.12/lib/rails/railtie.rb:194:in
>
> `method_missing'
> /home/git/gitlab/config/environment.rb:5:in `<top (required)>'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
>
> `require'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
>
> `block in require'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:232:in
>
> `load_dependency'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/activesupport-4.1.12/lib/active_support/dependencies.rb:247:in
>
> `require'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/cli.rb:236:in
>
> `boot_system'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.0/lib/sidekiq/cli.rb:50:in
>
> `run'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/gems/sidekiq-3.3.0/bin/sidekiq:8:in
> `<top (required)>'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/sidekiq:23:in `load'
> /home/git/gitlab/vendor/bundle/ruby/2.1.0/bin/sidekiq:23:in `<main>'
> ````
>
> I've restart Redis with no errors:
>
> ````
> [1378] 08 Feb 07:38:54.554 # Server started, Redis version 2.8.4
> [1378] 08 Feb 07:38:54.554 # WARNING overcommit_memory is set to 0!
> Background save may fail under low memory condition. To fix this issue add
> 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the
> command 'sysctl vm.overcommit_memory=1' for this to take effect.
> [1378] 08 Feb 07:38:54.729 * DB loaded from disk: 0.175 seconds
> [1378] 08 Feb 07:38:54.730 * The server is now ready to accept connections
> at /var/run/redis/redis.sock
> ````
>
> Our production.log shows:
> ````
> Errno::EACCES (Permission denied - connect(2) for
> /var/run/redis/redis.sock):
> config/initializers/rack_attack_git_basic_auth.rb:5:in `block in <top
> (required)>'
> ````
>
>
>
> We have not made any configuration changes or updates to the server since
> we last updated it to 8.2.3. Here is our environment check:
>
> ````
> System information
> System: Ubuntu 14.04
> Current User: git
> Using RVM: no
> Ruby Version: 2.1.6p336
> Gem Version: 2.2.3
> Bundler Version:1.10.6
> Rake Version: 10.4.2
> Sidekiq Version:3.3.0
>
> GitLab information
> Version: 8.2.3
> Revision: 19daba8
> Directory: /home/git/gitlab
> DB Adapter: postgresql
> URL: https://gitlab.domain.com
> HTTP Clone URL: https://gitlab.domain.com/some-group/some-project.git
> SSH Clone URL: g...@gitlab.domain.com:some-group/some-project.git
> Using LDAP: no
> Using Omniauth: yes
> Omniauth Providers: xxxx
>
> GitLab Shell
> Version: 2.6.8
> Repositories: /home/git/repositories/
> Hooks: /home/git/gitlab-shell/hooks/
> Git: /usr/local/bin/git
> ````
>
> Our VM has automatic updates for security patches turned on, everything
> else is manually updated. We have no done a manual update in around 1 week.
>
> Any ideas? Our server is currently show 500 errors from the browser and
> timing out from the cli when trying to clone.
>
--
You received this message because you are subscribed to the Google Groups
"GitLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to gitlabhq+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/gitlabhq/22cac6d4-552e-49fc-9ab4-7f02bf0d7faa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
