We've run into the exact same problem. We spent a few good solid hours trying to fix the problem with no avail (but we do have a 'solution').
Server specs: Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-27-generic x86_64) Rails & Ruby: Rails 4.0.0.beta1 Ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-linux] Unicorn 4.6.2 - preload_app true Capistrano The first thing we tried was removing all of the gems (foreman, sidekiq, etc.) and that didn't work. We tried preload_app false, and that worked (but this is not the situation we want). We also tried searching for any gems that were capturing USR2 (or any signals in general). We tried monkey patching Kernel & Trap to see if we could pinpoint the problem, but that didn't seem to turn up much. We then jumped in unicorn's http_server.rb and threw in some logging. The only thing that seemed to show us was that USR2 was unable to get through to unicorn. Signals HUP, QUIT, etc. all worked fine (and made their way through as per norm). We then fired up strace with unicorn to see if we could 'see' the USR2 signal. It looks almost identical to Jeffery's strace. It looks like the USR2 signal is received by the process, but then dropped (or ignored). If we pass it a QUIT signal it works just fine and we can see that in the strace. So here's the chain of events that we think is occuring: 1. We start unicorn (a unicorn blessing) 2. before_fork is triggered (Signal trapping works here) 3. A unicorn is born (Signal trapping does not work here) 4. The after fork is triggered In our situation. Once a unicorn is born, it can no longer received a USR2 (for whatever reason). We never actually checked to see if the after fork received a USR2. After we managed to figure out the path of execution, we figured out a quick fix. We actually call reexec method directly on the unicorn server in the before_fork block in unicorn.rb. Here's our unicorn.rb https://gist.github.com/RyanonRails/5541902 ----- snip ----- before_fork do |server, worker| Signal.trap 'USR2' do puts 'Since a black hole is lurking and eating USR2s we will hit http_server#reexec ourselves' server.send(:reexec) end end ----- snip ----- This way we can actually force the USR2 directly on the server (since USR2 is available inside of the before_fork). This seems to be working well for us now. We're still frustrated that we weren't able to find the cause of the problem, but we least we can move forward (with minor disruption to the code/code base). Lastly, here are our current theories as to why this could be occuring: 1. When unicorn is building it's native extensions something weird is happening in regards to the USR2 signal 2. A gem is somehow gobbling up the signal (we were quite thorough, but we might've missed something) Hopefully this helps! Thanks, Ryan & Mike _______________________________________________ Unicorn mailing list - [email protected] http://rubyforge.org/mailman/listinfo/mongrel-unicorn Do not quote signatures (like this one) or top post when replying
