I recently introuduced Resque to my application and I've started moving background tasks into queues. In day-to-day use, I use an :after_save hook to index articles in the background, but I also have a rake task:
=== desc "Indexing articles" task :index_articles => :environment do begin Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, a.main_text : a.raw_main_text), a.class.name, a.publish_date.to_i) } end end === When I ran this, I expected to see a couple of thousand articles appear almost instantly on the relevant queue on Resque, and be able to increase the number of workers to speed through them. Instead, I never see more than 5-10 at a time on the queue, and the rake task takes quite some time to run. I have 20,000 subscribers on a mailing list that currently takes 2-3 hours to send via SendGrid in a rake task. I was hoping that queuing these up on Resque would allow me to send them in minutes rather than hours with enough workers, but I'm guessing now that's not the case. Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, that could be sent off in minutes rather than hours? Or would Resque handle this and I just haven't configured it properly. Or is it something inherent in rake that is slowing batch jobs down in general. Although, it's not essential that the email goes out in minutes, those hours waiting for the emails to go out are incredibly stressful since the mailing list will sometimes break. Stephen PS:I noticed that SendGrid can now manage newsletter delivery for you, and I may explore that option. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.