On Fri, Apr 24, 2009 at 9:36 PM, <[email protected]> wrote: > My name's Nick, and I work for a company called Bytemark - we use > backgroundrb > in a range of internal projects (all our internal apps are Ruby on Rails, > so > it makes sense). > > Basically, and as I mentioned on IRC, I've been tasked with making > backgroundrb "better" over the next month or so, and I'd like to push as > much > upstream as I can while I'm at it (although some stuff I can guarantee you > won't want - I'll my making my own local branch use a login plugin with a > different syntax, for instance). > > First on the agenda is this backtrace: > > > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:147:in > `log': Mysql::Error: MySQL server has gone away: SELECT * FROM > `bdrb_job_queues` WHERE ( worker_name = 'dhshell_export_worker' AND > taken > = 0 AND scheduled_at <= '2009-04-24 10:41:44' ) LIMIT 1 FOR UPDATE > (ActiveRecord::StatementInvalid) > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:302:in > `execute' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/mysql_adapter.rb:537:in > `select' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:7:in > `select_all_without_query_cache' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/query_cache.rb:61:in > `select_all' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:586:in > `find_by_sql' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:1345:in > `find_every' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:1307:in > `find_initial' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/base.rb:538:in > `find' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_job_queue.rb:11:in > `find_next' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/connection_adapters/abstract/database_statements.rb:66:in > `transaction' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/rails/activerecord/lib/active_record/transactions.rb:79:in > `transaction' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/lib/backgroundrb/bdrb_job_queue.rb:8:in > `find_next' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:271:in > `check_for_enqueued_tasks' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:133:in > `worker_init' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_periodic_event.rb:23:in > `call' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_periodic_event.rb:23:in > `run' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in > `check_for_timer_events' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in > `each' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:301:in > `check_for_timer_events' > from > /home/bmrack/bmrack/releases/20090404011130/vendor/plugins/backgroundrb/server/lib/meta_worker.rb:296:in > `check_for_timer_events' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:140:in > `start_reactor' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:139:in > `loop' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_core.rb:139:in > `start_reactor' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/../lib/packet/packet_worker.rb:20:in > `start_worker' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:33:in > `load_worker' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:26:in > `initialize' > from > /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47:in > `new' > from /usr/lib/ruby/gems/1.8/gems/packet-0.1.14/bin/packet_worker_runner:47 > from /usr/bin/packet_worker_runner:19:in `load' > from /usr/bin/packet_worker_runner:19 > > (not latest git, but definitely git - we're just updating to packet 0.1.15, > but I doubt that'll affect this particular error) - this kills the server > from time to time for us (we have our MySQL server set to go away after > half > a day) - it's not the full story, since for that to happen, worker requests > must have stopped too (or something else), but fixing this is number 1 on > my > list. I've not delved much into the source yet (5pm on a Friday is *not* > the > time to start with that!); I was wondering if you guys have a preferred / > global strategy for dealing with errors. My approach to the above would be > to > catch the error, and respond by attempting to re-establish the connection. > If > that succeeds, I'd re-do the scheduled item (so the database going away > would > be transparent to the client); if not... hmm. Do we have a method of > reporting a failed task back to the worker?
There is already a fix for this in git master. See this commit: http://is.gd/ukbl > > > Another problem I was seeing that led me to stop using the backgroundrb > cache > object was that requests into a worker via the MiddleMan API would just > freeze up from time to time, leaving the Rails controller method hanging on > for it. That's another one for me to investigate (we weren't using > memcached, > mind, but it annoys me when the Hash - a far simpler system - is less > reliable). Really BackgrounDRbs inbuilt cache wasn't written to handle the caching scenario we are talking about it. Its just there so you can develop with cache without installing memcache and in production you are definitely expected to use memcache. One can perhaps improve this, but as memory size of ruby process goes its going to spend far more time just in GC. Ruby isn't a language to run with huge amount of memory. In other words, please install memcache and get rid of this headache. > > > I'm also tasked with redeploying the current backgroundrb setup - right > now, > we have one server per application using it (which is about 4,5 > backgroundrb > servers in total), with the backgroundrb table stored in the application's > database. I'm kind of moving towards a scheme where we have a pair of > backgroundrb servers (transparently load-balanced) used by all > applications. > Each backgroundrb server would have its own separate *SQLite* database. > Requests would go to one or another of the backgroundrb servers; if one of > the servers died, all the requests would go to the other server, and > vice-versa. If possible (I know this is a -devel list), I'd like to get > comments on whether that's a good setup or not, and suggestions for > improvement. If not, well, just consider it to be a bit of context ;) > > Whats wrong with keeping one table of jobs which is shared across all backgroundrb servers? Surely if one server crashes automatically other server will start picking the jobs, because for jobs added to persistent queue, there is nothing like "it goes to server X". For persistent queue, job goes to queue not to a particular bdrb server and hence if one server dies other should start processing it without problems. I do not see advantage of having localized sqlite based queues.
_______________________________________________ Backgroundrb-devel mailing list [email protected] http://rubyforge.org/mailman/listinfo/backgroundrb-devel
