mstump <mrevilgn...@gmail.com> writes:
> I'm curious as to what people are using for workers/queues. We're
> currently using storm, but im interested in evaluating alternatives.
If you can make your workflow fit, it's hard to beat the simplicity of
sending expressions over a message queue like RabbitMQ. This is the
approach Die Roboter uses:
https://github.com/technomancy/die-roboter/
While this particular implementation hasn't seen a lot of real world
use, it's based upon 2 years of experience managing a huge data pipeline
over AMQP. Since Clojure forms are trivial to send over the wire it
turns out you can implement it in under 200 lines:
https://github.com/technomancy/die-roboter/blob/master/src/die/roboter.clj
Here's an example program that uses it to distribute log parsing:
https://github.com/technomancy/logan/blob/master/src/logan/core.clj
IMO asynchronous messaging over queues is often overlooked but has some
pretty compelling advantages to offer, not the least of which is the
simplicity of the execution model. With Rabbit you also get the ability
to retry failed jobs by having workers claim a message but hold off on
acking it until the form has returned a value.
The main gotcha is that long-running tasks need to regularly report
progress in order to avoid timing out, but Die Roboter comes with a
variant of clojure.java.io/copy that periodically resets timeouts.
-Phil
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
