[This is mostly an update on
http://stackoverflow.com/questions/11447042/ (thanks Fred!), but I'll
take my answer from wherever I can get it...]
Short form: does the postgresql db adaptor cache queries? I'm passing
lots of large raw SQL queries (160k bytes each), and my private virtual
memory is growing and growing and growing...
Long form:
I'm launching external Rails tasks to fetch data from the web and store
said data into a database. These tasks grow without apparent bound, and
eventually bring my machine to its knees when its 8GB of physical RAM
fills up and it starts to swap.
I haven't been able to track down the source of the bloat. At each
Delayed::Job#after callback, I do an explicit
ObjectSpace.each_object(ActiveRecord::Relation).each(&:reset)
GC.start
report_object_use
In report_object_use, I print out ObjectSpace.each_object(cls).count for
Object, String, Array, Hash, and a few other classes. I also print out
Symbol.all_symbols.count. NONE of those numbers seem to be growing
much.
BUT I'm beginning to wonder if the db connection is caching the queries.
In particular, I'm constructing my own SQL queries and executing them
via:
ActiveRecord::Base.connection.select_all(query)
And the queries can be really really big -- about 160K bytes per
transaction, and I'm doing about a transaction per second (on a good
day). Note that if the db connection IS caching the queries, it's
happening under the Ruby level, since
ObjectSpace.each_object(String).count isn't increasing.
So the questions:
* Does the DB adaptor somehow cache SQL queries passed to it? Or is
there a way to find out?
* Um, I guess that's all the questions.
- ff
P.S.: Rails 1.9.3, Ruby 3.2.1, Postgresql 9.1.2, pgserver 8.3.14
P.P.S.: I know the bloat isn't happening on the network query side since
I created a mock object to simulate fetching the data.
--
Posted via http://www.ruby-forum.com/.
--
You received this message because you are subscribed to the Google Groups "Ruby
on Rails: Talk" group.
To post to this group, send email to rubyonrails-talk@googlegroups.com.
To unsubscribe from this group, send email to
rubyonrails-talk+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/rubyonrails-talk?hl=en-US.
