Re: Resque vs Appoxy SimpleWorker
Hi Stephen, You could definitely get increased throughput using SimpleWorker as you would instantly be going from one server to hundreds to run your jobs. (disclosure: I created SimpleWorker). If you decide to do it, there is one suggestion that will make it work a lot faster/better for your scenario where you have a bunch of small jobs: batch up the work into fewer jobs. For instance, instead of queuing up one email per job, queue up 50 emails per job (pass an array of 50 emails to the worker). So instead of queuing up 20,000 jobs, you'd queue up 400 jobs with 50 emails each. There are a couple of reasons for this: 1) It will take longer to queue up all your jobs since you're sending them to a remote service (unlike Resque) so it's recommended to batch them up. Queuing up 400 jobs takes a fraction of the time compared to queuing up 20,000. 2) There is setup/teardown time involved on SimpleWorker, so if you're jobs only take a couple seconds to run, like sending an email probably does, you'll want to batch them up. If you don't, more time will be spent on the setup/teardown than actually doing the work. But if you do 50 or 100 in a single job, the setup/teardown will be negligible. I hope that helps! Feel free to email me directly if you have any other questions or want some help getting started. Regards, Travis -- You received this message because you are subscribed to the Google Groups "Heroku" group. To view this discussion on the web visit https://groups.google.com/d/msg/heroku/-/XUVNRZUEnbMJ. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Resque vs Appoxy SimpleWorker
Hi Stephen,
Using Resque and Sendgrid (running locally / not on heroku) with 4 resque
workers, we are able to completely mail out 30K emails within an hour or two.
It took practically no time to enqueue.
I may be remembering this incorrectly, as luckily we don't send out slews of
emails every night.
Article.all.each loads everything into memory. You may want to try find_each /
find_in_batches or a pure sql route. (There are ways to do this in rails 2 and
rails 3)
Good Luck,
Keenan
On Sunday, August 28, 2011 at 5:01 AM, Stephen Cremin wrote:
> I created another rake task that pushed jobs onto Resque, and indeed it did
> push thousands of jobs onto the queue and let me ramp through them by
> increasing the number of workers. Perhaps I was hit by the recent worker bug
> outlined on Heroku's status page. If nobody else has direct experience with
> Sendgrid on Resque vs Appoxy, I'll do my own experiment and report back what
> kind of throughput I can get.
>
> Stephen
>
>
> On 28 August 2011 08:50, Gabriel (mailto:ummo...@gmail.com)> wrote:
> > On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in
> > seconds so I think something else is going on here. Maybe it's
> > ActiveRecord taking a while to load a bunch of objects?
> >
> > On Aug 26, 6:10 pm, Stephen Cremin > (mailto:asianf...@gmail.com)> wrote:
> > > I recently introuduced Resque to my application and I've started moving
> > > background tasks into queues. In day-to-day use, I use an :after_save hook
> > > to index articles in the background, but I also have a rake task:
> > >
> > > ===
> > > desc "Indexing articles"
> > >
> > > task :index_articles => :environment do
> > > begin
> > > Article.all.each { |a| Resque.enqueue(IndexArticle, a.id (http://a.id),
> > > a.main_text :
> > > a.raw_main_text), a.class.name (http://a.class.name),
> > > a.publish_date.to_i) }
> > > end
> > > end
> > > ===
> > >
> > > When I ran this, I expected to see a couple of thousand articles appear
> > > almost instantly on the relevant queue on Resque, and be able to increase
> > > the number of workers to speed through them. Instead, I never see more
> > > than
> > > 5-10 at a time on the queue, and the rake task takes quite some time to
> > > run.
> > >
> > > I have 20,000 subscribers on a mailing list that currently takes 2-3 hours
> > > to send via SendGrid in a rake task. I was hoping that queuing these up on
> > > Resque would allow me to send them in minutes rather than hours with
> > > enough
> > > workers, but I'm guessing now that's not the case.
> > >
> > > Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue,
> > > that
> > > could be sent off in minutes rather than hours? Or would Resque handle
> > > this
> > > and I just haven't configured it properly. Or is it something inherent in
> > > rake that is slowing batch jobs down in general.
> > >
> > > Although, it's not essential that the email goes out in minutes, those
> > > hours
> > > waiting for the emails to go out are incredibly stressful since the
> > > mailing
> > > list will sometimes break.
> > >
> > > Stephen
> > >
> > > PS:I noticed that SendGrid can now manage newsletter delivery for you,
> > > and I
> > > may explore that option.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Heroku" group.
> > To post to this group, send email to heroku@googlegroups.com
> > (mailto:heroku@googlegroups.com).
> > To unsubscribe from this group, send email to
> > heroku+unsubscr...@googlegroups.com
> > (mailto:heroku%2bunsubscr...@googlegroups.com).
> > For more options, visit this group at
> > http://groups.google.com/group/heroku?hl=en.
> >
>
> --
> You received this message because you are subscribed to the Google Groups
> "Heroku" group.
> To post to this group, send email to heroku@googlegroups.com
> (mailto:heroku@googlegroups.com).
> To unsubscribe from this group, send email to
> heroku+unsubscr...@googlegroups.com
> (mailto:heroku+unsubscr...@googlegroups.com).
> For more options, visit this group at
> http://groups.google.com/group/heroku?hl=en.
--
You received this message because you are subscribed to the Google Groups
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
Re: Resque vs Appoxy SimpleWorker
I created another rake task that pushed jobs onto Resque, and indeed it did
push thousands of jobs onto the queue and let me ramp through them by
increasing the number of workers. Perhaps I was hit by the recent worker bug
outlined on Heroku's status page. If nobody else has direct experience with
Sendgrid on Resque vs Appoxy, I'll do my own experiment and report back what
kind of throughput I can get.
Stephen
On 28 August 2011 08:50, Gabriel wrote:
> On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in
> seconds so I think something else is going on here. Maybe it's
> ActiveRecord taking a while to load a bunch of objects?
>
> On Aug 26, 6:10 pm, Stephen Cremin wrote:
> > I recently introuduced Resque to my application and I've started moving
> > background tasks into queues. In day-to-day use, I use an :after_save
> hook
> > to index articles in the background, but I also have a rake task:
> >
> > ===
> > desc "Indexing articles"
> >
> > task :index_articles => :environment do
> > begin
> > Article.all.each { |a| Resque.enqueue(IndexArticle, a.id,
> a.main_text :
> > a.raw_main_text), a.class.name, a.publish_date.to_i) }
> > end
> > end
> > ===
> >
> > When I ran this, I expected to see a couple of thousand articles appear
> > almost instantly on the relevant queue on Resque, and be able to increase
> > the number of workers to speed through them. Instead, I never see more
> than
> > 5-10 at a time on the queue, and the rake task takes quite some time to
> run.
> >
> > I have 20,000 subscribers on a mailing list that currently takes 2-3
> hours
> > to send via SendGrid in a rake task. I was hoping that queuing these up
> on
> > Resque would allow me to send them in minutes rather than hours with
> enough
> > workers, but I'm guessing now that's not the case.
> >
> > Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue,
> that
> > could be sent off in minutes rather than hours? Or would Resque handle
> this
> > and I just haven't configured it properly. Or is it something inherent in
> > rake that is slowing batch jobs down in general.
> >
> > Although, it's not essential that the email goes out in minutes, those
> hours
> > waiting for the emails to go out are incredibly stressful since the
> mailing
> > list will sometimes break.
> >
> > Stephen
> >
> > PS:I noticed that SendGrid can now manage newsletter delivery for you,
> and I
> > may explore that option.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Heroku" group.
> To post to this group, send email to heroku@googlegroups.com.
> To unsubscribe from this group, send email to
> heroku+unsubscr...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/heroku?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
Re: Resque vs Appoxy SimpleWorker
On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in
seconds so I think something else is going on here. Maybe it's
ActiveRecord taking a while to load a bunch of objects?
On Aug 26, 6:10 pm, Stephen Cremin wrote:
> I recently introuduced Resque to my application and I've started moving
> background tasks into queues. In day-to-day use, I use an :after_save hook
> to index articles in the background, but I also have a rake task:
>
> ===
> desc "Indexing articles"
>
> task :index_articles => :environment do
> begin
> Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, a.main_text :
> a.raw_main_text), a.class.name, a.publish_date.to_i) }
> end
> end
> ===
>
> When I ran this, I expected to see a couple of thousand articles appear
> almost instantly on the relevant queue on Resque, and be able to increase
> the number of workers to speed through them. Instead, I never see more than
> 5-10 at a time on the queue, and the rake task takes quite some time to run.
>
> I have 20,000 subscribers on a mailing list that currently takes 2-3 hours
> to send via SendGrid in a rake task. I was hoping that queuing these up on
> Resque would allow me to send them in minutes rather than hours with enough
> workers, but I'm guessing now that's not the case.
>
> Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, that
> could be sent off in minutes rather than hours? Or would Resque handle this
> and I just haven't configured it properly. Or is it something inherent in
> rake that is slowing batch jobs down in general.
>
> Although, it's not essential that the email goes out in minutes, those hours
> waiting for the emails to go out are incredibly stressful since the mailing
> list will sometimes break.
>
> Stephen
>
> PS:I noticed that SendGrid can now manage newsletter delivery for you, and I
> may explore that option.
--
You received this message because you are subscribed to the Google Groups
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
Re: Resque vs Appoxy SimpleWorker
On Fri, Aug 26, 2011 at 6:10 PM, Stephen Cremin wrote:
> I recently introuduced Resque to my application and I've started moving
> background tasks into queues. In day-to-day use, I use an :after_save hook
> to index articles in the background, but I also have a rake task:
> ===
> desc "Indexing articles"
> task :index_articles => :environment do
> begin
> Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, a.main_text :
> a.raw_main_text), a.class.name, a.publish_date.to_i) }
> end
> end
> ===
> When I ran this, I expected to see a couple of thousand articles appear
> almost instantly on the relevant queue on Resque, and be able to increase
> the number of workers to speed through them. Instead, I never see more than
> 5-10 at a time on the queue, and the rake task takes quite some time to run.
> I have 20,000 subscribers on a mailing list that currently takes 2-3 hours
> to send via SendGrid in a rake task. I was hoping that queuing these up on
> Resque would allow me to send them in minutes rather than hours with enough
> workers, but I'm guessing now that's not the case.
> Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, that
> could be sent off in minutes rather than hours? Or would Resque handle this
> and I just haven't configured it properly. Or is it something inherent in
> rake that is slowing batch jobs down in general.
+1. We are using Pony mail with sendgrid and what we really need is
pipelining when sending mass emails, which it doesn't support. With
STARTTLS negotiation, if we could send multiple emails through the
same connection (SMTP supports this) this will cut down the start up
time for each email. Maybe a feature request for Pony mail?
> Although, it's not essential that the email goes out in minutes, those hours
> waiting for the emails to go out are incredibly stressful since the mailing
> list will sometimes break.
> Stephen
K.
---
http://blog.mudynamics.com
http://blitz.io
@pcapar
--
You received this message because you are subscribed to the Google Groups
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
Resque vs Appoxy SimpleWorker
I recently introuduced Resque to my application and I've started moving
background tasks into queues. In day-to-day use, I use an :after_save hook
to index articles in the background, but I also have a rake task:
===
desc "Indexing articles"
task :index_articles => :environment do
begin
Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, a.main_text :
a.raw_main_text), a.class.name, a.publish_date.to_i) }
end
end
===
When I ran this, I expected to see a couple of thousand articles appear
almost instantly on the relevant queue on Resque, and be able to increase
the number of workers to speed through them. Instead, I never see more than
5-10 at a time on the queue, and the rake task takes quite some time to run.
I have 20,000 subscribers on a mailing list that currently takes 2-3 hours
to send via SendGrid in a rake task. I was hoping that queuing these up on
Resque would allow me to send them in minutes rather than hours with enough
workers, but I'm guessing now that's not the case.
Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, that
could be sent off in minutes rather than hours? Or would Resque handle this
and I just haven't configured it properly. Or is it something inherent in
rake that is slowing batch jobs down in general.
Although, it's not essential that the email goes out in minutes, those hours
waiting for the emails to go out are incredibly stressful since the mailing
list will sometimes break.
Stephen
PS:I noticed that SendGrid can now manage newsletter delivery for you, and I
may explore that option.
--
You received this message because you are subscribed to the Google Groups
"Heroku" group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
Re: Appoxy SimpleWorker
Hi there, I am with Appoxy, the creators of SimpleWorker, and you should be able to run any kind of job you want. Obviously some jobs need more horsepower so we have plans with bigger servers too. Feel free to contact me directly and I'll see what I can do to make it work for you. Regards, Travis Reeder -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Appoxy SimpleWorker
Ah thanks for that Not clear it is suitable for image manipulation... as that kind of task is quite beefy, requiring memory, grunt and awkward libraries like imagemagick -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Appoxy SimpleWorker
Hey there, Check out http://www.simpleworker.com/. -M On Feb 28, 2011, at 4:28 PM, railsnerd wrote: > This has appeared on the Add-Ons page > > But a google search reveals nothing > > Anyone know who provides this service? > > Would be good to know more details. I'm looking for a cost effecting > way to manage image resizing, and this could be ideal. > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Appoxy SimpleWorker
This has appeared on the Add-Ons page But a google search reveals nothing Anyone know who provides this service? Would be good to know more details. I'm looking for a cost effecting way to manage image resizing, and this could be ideal. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
