On Wed, Jan 7, 2015 at 3:53 PM, Lin Jen-Shin (godfat) <[email protected]> wrote:
> I just realized that all those disconnected connections almost
> sharing the same object_id. I think this didn't prove that the
> connection was shared amongst processes, but I guess this
> highly suggested the possibility, therefore I tried to turn off
> preload_app.
>
> Not too surprising but this still surprised me! The problem was
> gone. So somehow there's still a connection sneaked in Unicorn's
> master process and got copied over the workers.
>
> I don't understand how this got in because I did verify that
> in the after_fork hook provided in Unicorn, the size of the
> connection pool was really 0 amongst all the workers and
> in the master in before_fork.
>
> I am worried that using preload_app=false would get us some
> troubles by hitting the memory limit soon, but... anyway...
>
> Would keep you posted after I got more clues.
Alright, thank you so much for helping this, Jeremy. I've figured
this out and fixed it. In short, this was totally my own fault.
I should have realized this much earlier. Spent over 15+ hours.
Actually this was really caused by upgrading to Rails 4, because
they fixed a bug. Since we're running on Heroku, we need to use
ENV['DATABASE_URL']. In that case, ActiveRecord::Base.configurations
would be empty on Rails 3, and it would be properly setup on Rails 4.
Why did this matter? Because I was depending on this behaviour.
I wrote a comment long ago:
# XXX: We need to make sure we're connecting to the same database in test.
# This is a special case because Rails would switch from development
# to test when you run `rake test`, but then, Sequel didn't switch
# with Rails. However, if we're running Rails with ENV['DATABASE_URL'],
# ActiveRecord::Base.configurations would be `{}`
#
# We skip it at that case, assuming then Rails would not try to switch
# database.
That means, at the time we're running Rails 3, this hack was skipped on
Heroku. But this hack did get triggered at the time we upgraded to Rails 4.
The hack was not properly written, it was:
db = ActiveRecord::Base.configurations[Rails.env]['database']
Model.db = Sequel.connect(Model.db.opts.merge(:database => db))
The problem was, now we have *2* Sequel databases running.
I was not aware that this would cause connections get shared
amongst processes...
I guess this was what happening:
First Rails loads *some* of the models inheriting Model,
and those models get the 1st database. The above hack
was located in Rails initializers, which creates the 2nd
database, and then the rest models use the 2nd database.
The above assumption might not really be correct according to the log
I observed. I am not sure how Sequel picks the database object.
I assume once it's set, it never changes. In the before_fork hook in
Unicorn, we disconnect the 2nd database, but not the 1st one.
Occasionally, some query Model just picked the 1st database, which was
not disconnected in Unicorn master process, which got copied over the
workers, was problematic.
At that time, a disconnection happened.
I discovered this by looking at the object_id for Sequel::Database.
By checking Sequel::DATABASES, I realized that if I don't load Rails,
there's only one. But if I load Rails, it's 2!
I should probably just do this in before_fork:
Sequel::DATABASE.each(&:disconnect)
We don't want to share any connections anyway, right?
Though this might also hide the problematic hack I put
in the first place.
Now I changed the hack to:
if db != Model.db.opts[:database]
Model.db.disconnect
Model.db.opts[:database] = db
end
I guess this is a much better way to switch database.
This also breaks the compatibility for Rails 3, but that's fine.
I removed all the logs and connection_validator extension.
Everything works perfectly fine now.
Thanks!!
--
You received this message because you are subscribed to the Google Groups
"sequel-talk" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/sequel-talk.
For more options, visit https://groups.google.com/d/optout.
