On Feb 6, 2014, at 6:45 PM, Zach Tellman <z...@factual.com> wrote: > At Factual we get a lot of data thrown at us, and often don't have control > over the rate at which it comes in. As such, it's preferable that our buffer > isn't bounded by the process' memory, since a temporary blip in throughput > may cause GC pauses, OOM exceptions, and other things that will only > exacerbate the problem. It's also preferable that if the process dies, we > won't lose any data which hasn't yet escaped the process. A disk-backed > queue satisfies both of these requirements. > > As such, I'm happy to announce that we're open sourcing 'durable-queue': > https://github.com/Factual/durable-queue. It's a small, fast, pure-Clojure > implementation that in our production systems is responsible for processing > billions of entries daily. We believe it has broad applications, and are > excited to see how others will use it.
What excellent timing! I've been looking at ZeroMQ, RabbitMQ, and Kafka for the last week or so. ZMQ is awfully attractive for what I'm trying to do, but there are a few things it doesn't do that I need done. I had begun thinking of building something similar on top of Redis. You mention the idea of batching to reduce the impact of fsync. Is there an API for batching puts? Is there a way to batch a complete! and put! new tasks to the queue? One pattern that keeps coming up is: - take a single task from the queue - execute the task, which might generate a set of new tasks to be queued on the same queue (and likely on other queues too) - signal completion, and put the new tasks Cheers, Bob > > Zach > > P.S. If this sort of work is interesting to you, Factual is hiring: > https://groups.google.com/forum/#!searchin/clojure/factual/clojure/8bPIEnNpfyQ/lvv-9gkVozAJ > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
