Hi Stephen,
Using Resque and Sendgrid (running locally / not on heroku) with 4 resque
workers, we are able to completely mail out 30K emails within an hour or two.
It took practically no time to enqueue.
I may be remembering this incorrectly, as luckily we don't send out slews of
emails every night.
Article.all.each loads everything into memory. You may want to try find_each /
find_in_batches or a pure sql route. (There are ways to do this in rails 2 and
rails 3)
Good Luck,
Keenan
On Sunday, August 28, 2011 at 5:01 AM, Stephen Cremin wrote:
> I created another rake task that pushed jobs onto Resque, and indeed it did
> push thousands of jobs onto the queue and let me ramp through them by
> increasing the number of workers. Perhaps I was hit by the recent worker bug
> outlined on Heroku's status page. If nobody else has direct experience with
> Sendgrid on Resque vs Appoxy, I'll do my own experiment and report back what
> kind of throughput I can get.
>
> Stephen
>
>
> On 28 August 2011 08:50, Gabriel (mailto:ummo...@gmail.com)> wrote:
> > On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in
> > seconds so I think something else is going on here. Maybe it's
> > ActiveRecord taking a while to load a bunch of objects?
> >
> > On Aug 26, 6:10 pm, Stephen Cremin > (mailto:asianf...@gmail.com)> wrote:
> > > I recently introuduced Resque to my application and I've started moving
> > > background tasks into queues. In day-to-day use, I use an :after_save hook
> > > to index articles in the background, but I also have a rake task:
> > >
> > > ===
> > > desc "Indexing articles"
> > >
> > > task :index_articles => :environment do
> > > begin
> > > Article.all.each { |a| Resque.enqueue(IndexArticle, a.id (http://a.id),
> > > a.main_text :
> > > a.raw_main_text), a.class.name (http://a.class.name),
> > > a.publish_date.to_i) }
> > > end
> > > end
> > > ===
> > >
> > > When I ran this, I expected to see a couple of thousand articles appear
> > > almost instantly on the relevant queue on Resque, and be able to increase
> > > the number of workers to speed through them. Instead, I never see more
> > > than
> > > 5-10 at a time on the queue, and the rake task takes quite some time to
> > > run.
> > >
> > > I have 20,000 subscribers on a mailing list that currently takes 2-3 hours
> > > to send via SendGrid in a rake task. I was hoping that queuing these up on
> > > Resque would allow me to send them in minutes rather than hours with
> > > enough
> > > workers, but I'm guessing now that's not the case.
> > >
> > > Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue,
> > > that
> > > could be sent off in minutes rather than hours? Or would Resque handle
> > > this
> > > and I just haven't configured it properly. Or is it something inherent in
> > > rake that is slowing batch jobs down in general.
> > >
> > > Although, it's not essential that the email goes out in minutes, those
> > > hours
> > > waiting for the emails to go out are incredibly stressful since the
> > > mailing
> > > list will sometimes break.
> > >
> > > Stephen
> > >
> > > PS:I noticed that SendGrid can now manage newsletter delivery for you,
> > > and I
> > > may explore that option.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to heroku@googlegroups.com
> > (mailto:heroku@googlegroups.com).
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com
> > (mailto:heroku%2bunsubscr...@googlegroups.com).
> > For more options, visit this group at
> > http://groups.google.com/group/heroku?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Heroku" group.
> To post to this group, send email to heroku@googlegroups.com
> (mailto:heroku@googlegroups.com).
> To unsubscribe from this group, send email to
> heroku+unsubscr...@googlegroups.com
> (mailto:heroku+unsubscr...@googlegroups.com).
> For more options, visit this group at
> http://groups.google.com/group/heroku?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
