Re: Resque vs Appoxy SimpleWorker
Hi Stephen,
Using Resque and Sendgrid (running locally / not on heroku) with 4 resque
workers, we are able to completely mail out 30K emails within an hour or two.
It took practically no time to enqueue.
I may be remembering this incorrectly, as luckily we don't send out slews of
emails every night.
Article.all.each loads everything into memory. You may want to try find_each /
find_in_batches or a pure sql route. (There are ways to do this in rails 2 and
rails 3)
Good Luck,
Keenan
On Sunday, August 28, 2011 at 5:01 AM, Stephen Cremin wrote:
I created another rake task that pushed jobs onto Resque, and indeed it did
push thousands of jobs onto the queue and let me ramp through them by
increasing the number of workers. Perhaps I was hit by the recent worker bug
outlined on Heroku's status page. If nobody else has direct experience with
Sendgrid on Resque vs Appoxy, I'll do my own experiment and report back what
kind of throughput I can get.
Stephen
On 28 August 2011 08:50, Gabriel ummo...@gmail.com
(mailto:ummo...@gmail.com) wrote:
On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in
seconds so I think something else is going on here. Maybe it's
ActiveRecord taking a while to load a bunch of objects?
On Aug 26, 6:10 pm, Stephen Cremin asianf...@gmail.com
(mailto:asianf...@gmail.com) wrote:
I recently introuduced Resque to my application and I've started moving
background tasks into queues. In day-to-day use, I use an :after_save hook
to index articles in the background, but I also have a rake task:
===
desc Indexing articles
task :index_articles = :environment do
begin
Article.all.each { |a| Resque.enqueue(IndexArticle, a.id (http://a.id),
a.main_text :
a.raw_main_text), a.class.name (http://a.class.name),
a.publish_date.to_i) }
end
end
===
When I ran this, I expected to see a couple of thousand articles appear
almost instantly on the relevant queue on Resque, and be able to increase
the number of workers to speed through them. Instead, I never see more
than
5-10 at a time on the queue, and the rake task takes quite some time to
run.
I have 20,000 subscribers on a mailing list that currently takes 2-3 hours
to send via SendGrid in a rake task. I was hoping that queuing these up on
Resque would allow me to send them in minutes rather than hours with
enough
workers, but I'm guessing now that's not the case.
Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue,
that
could be sent off in minutes rather than hours? Or would Resque handle
this
and I just haven't configured it properly. Or is it something inherent in
rake that is slowing batch jobs down in general.
Although, it's not essential that the email goes out in minutes, those
hours
waiting for the emails to go out are incredibly stressful since the
mailing
list will sometimes break.
Stephen
PS:I noticed that SendGrid can now manage newsletter delivery for you,
and I
may explore that option.
--
You received this message because you are subscribed to the Google Groups
Heroku group.
To post to this group, send email to heroku@googlegroups.com
(mailto:heroku@googlegroups.com).
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
(mailto:heroku%2bunsubscr...@googlegroups.com).
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
--
You received this message because you are subscribed to the Google Groups
Heroku group.
To post to this group, send email to heroku@googlegroups.com
(mailto:heroku@googlegroups.com).
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com
(mailto:heroku+unsubscr...@googlegroups.com).
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
--
You received this message because you are subscribed to the Google Groups
Heroku group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
Re: Resque vs Appoxy SimpleWorker
Hi Stephen, You could definitely get increased throughput using SimpleWorker as you would instantly be going from one server to hundreds to run your jobs. (disclosure: I created SimpleWorker). If you decide to do it, there is one suggestion that will make it work a lot faster/better for your scenario where you have a bunch of small jobs: batch up the work into fewer jobs. For instance, instead of queuing up one email per job, queue up 50 emails per job (pass an array of 50 emails to the worker). So instead of queuing up 20,000 jobs, you'd queue up 400 jobs with 50 emails each. There are a couple of reasons for this: 1) It will take longer to queue up all your jobs since you're sending them to a remote service (unlike Resque) so it's recommended to batch them up. Queuing up 400 jobs takes a fraction of the time compared to queuing up 20,000. 2) There is setup/teardown time involved on SimpleWorker, so if you're jobs only take a couple seconds to run, like sending an email probably does, you'll want to batch them up. If you don't, more time will be spent on the setup/teardown than actually doing the work. But if you do 50 or 100 in a single job, the setup/teardown will be negligible. I hope that helps! Feel free to email me directly if you have any other questions or want some help getting started. Regards, Travis -- You received this message because you are subscribed to the Google Groups Heroku group. To view this discussion on the web visit https://groups.google.com/d/msg/heroku/-/XUVNRZUEnbMJ. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Resque vs Appoxy SimpleWorker
I created another rake task that pushed jobs onto Resque, and indeed it did
push thousands of jobs onto the queue and let me ramp through them by
increasing the number of workers. Perhaps I was hit by the recent worker bug
outlined on Heroku's status page. If nobody else has direct experience with
Sendgrid on Resque vs Appoxy, I'll do my own experiment and report back what
kind of throughput I can get.
Stephen
On 28 August 2011 08:50, Gabriel ummo...@gmail.com wrote:
On a weekly basis I load 5 - 600 jobs into a DelayedJob queue in
seconds so I think something else is going on here. Maybe it's
ActiveRecord taking a while to load a bunch of objects?
On Aug 26, 6:10 pm, Stephen Cremin asianf...@gmail.com wrote:
I recently introuduced Resque to my application and I've started moving
background tasks into queues. In day-to-day use, I use an :after_save
hook
to index articles in the background, but I also have a rake task:
===
desc Indexing articles
task :index_articles = :environment do
begin
Article.all.each { |a| Resque.enqueue(IndexArticle, a.id,
a.main_text :
a.raw_main_text), a.class.name, a.publish_date.to_i) }
end
end
===
When I ran this, I expected to see a couple of thousand articles appear
almost instantly on the relevant queue on Resque, and be able to increase
the number of workers to speed through them. Instead, I never see more
than
5-10 at a time on the queue, and the rake task takes quite some time to
run.
I have 20,000 subscribers on a mailing list that currently takes 2-3
hours
to send via SendGrid in a rake task. I was hoping that queuing these up
on
Resque would allow me to send them in minutes rather than hours with
enough
workers, but I'm guessing now that's not the case.
Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue,
that
could be sent off in minutes rather than hours? Or would Resque handle
this
and I just haven't configured it properly. Or is it something inherent in
rake that is slowing batch jobs down in general.
Although, it's not essential that the email goes out in minutes, those
hours
waiting for the emails to go out are incredibly stressful since the
mailing
list will sometimes break.
Stephen
PS:I noticed that SendGrid can now manage newsletter delivery for you,
and I
may explore that option.
--
You received this message because you are subscribed to the Google Groups
Heroku group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
--
You received this message because you are subscribed to the Google Groups
Heroku group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
Resque vs Appoxy SimpleWorker
I recently introuduced Resque to my application and I've started moving
background tasks into queues. In day-to-day use, I use an :after_save hook
to index articles in the background, but I also have a rake task:
===
desc Indexing articles
task :index_articles = :environment do
begin
Article.all.each { |a| Resque.enqueue(IndexArticle, a.id, a.main_text :
a.raw_main_text), a.class.name, a.publish_date.to_i) }
end
end
===
When I ran this, I expected to see a couple of thousand articles appear
almost instantly on the relevant queue on Resque, and be able to increase
the number of workers to speed through them. Instead, I never see more than
5-10 at a time on the queue, and the rake task takes quite some time to run.
I have 20,000 subscribers on a mailing list that currently takes 2-3 hours
to send via SendGrid in a rake task. I was hoping that queuing these up on
Resque would allow me to send them in minutes rather than hours with enough
workers, but I'm guessing now that's not the case.
Would Appoxy Simpleworker let me fire off 20,000 email jobs in a queue, that
could be sent off in minutes rather than hours? Or would Resque handle this
and I just haven't configured it properly. Or is it something inherent in
rake that is slowing batch jobs down in general.
Although, it's not essential that the email goes out in minutes, those hours
waiting for the emails to go out are incredibly stressful since the mailing
list will sometimes break.
Stephen
PS:I noticed that SendGrid can now manage newsletter delivery for you, and I
may explore that option.
--
You received this message because you are subscribed to the Google Groups
Heroku group.
To post to this group, send email to heroku@googlegroups.com.
To unsubscribe from this group, send email to
heroku+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/heroku?hl=en.
Appoxy SimpleWorker
This has appeared on the Add-Ons page But a google search reveals nothing Anyone know who provides this service? Would be good to know more details. I'm looking for a cost effecting way to manage image resizing, and this could be ideal. -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Appoxy SimpleWorker
Ah thanks for that Not clear it is suitable for image manipulation... as that kind of task is quite beefy, requiring memory, grunt and awkward libraries like imagemagick -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Appoxy SimpleWorker
Hi there, I am with Appoxy, the creators of SimpleWorker, and you should be able to run any kind of job you want. Obviously some jobs need more horsepower so we have plans with bigger servers too. Feel free to contact me directly and I'll see what I can do to make it work for you. Regards, Travis Reeder -- You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
