Re: [clojure-rabbitmq] Are lazy queues still bound by memory limit?
Thank you! On Sunday, September 17, 2017 at 11:32:17 PM UTC-5, Michael Klishin wrote: > > The point of lazy queues is not that nothing ever is stored in memory. > It's that queues try to move messages > to disk very aggressively (default mode keeps a portion in RAM using > metrics such as ingress/egress rates and various > configurable values). > > On Sun, Sep 17, 2017 at 10:29 PM, Michael Klishin <mkli...@pivotal.io > > wrote: > >> This is a question for rabbitmq-users but I'm happy to answer it here. >> >> Lazy queues per se don't have a limit but RabbitMQ message store still >> does: when messages >> are sent to the message store [as opposed to being embedded into queue >> index], message store index(es) >> come into play and by default the index is in-memory. >> >> There is a plugin [1] that lets you use LevelDB for the index. Even >> though LevelDB is an efficient >> data store, expect a write throughput hit compared to the default >> implementation. >> >> 1. https://github.com/rabbitmq/rabbitmq-msg-store-index-eleveldb >> >> On Sun, Sep 17, 2017 at 8:25 PM, Steve Suehs <skelt...@gmail.com >> > wrote: >> >>> I've been experimenting with Clojure and RabbitMQ using the langohr >>> library. Thank you for building and sharing it. >>> >>> I've been learning more about the care and feeding of a rabbitmq >>> server. I crashed mine installed on my dev laptop several times by filling >>> a queue with messages. If queues fill up and memory gets tight, a server >>> can be overwhelmed and fall over. If I set or arrange for a reasonable >>> memory limit, things go ok. I am now running RabbitMQ in a Docker container >>> with a memory limit. >>> >>> My latest experiments were with lazy queues, which are supposed to write >>> events to disk and not exhaust memory. I still seem to hit..well, not hit, >>> rather, an asymptotic limit at around 261 million events in the queue. Is >>> this considered outrageously and unreasonably large? Publication rate >>> slowed to 700/s. >>> >>> I also notice that if the consumers are caught up with the producer >>> everything seems to scream at twice the rate than when storage is used. If >>> storage is used I'm seeing about 6000r/s. That's cool...just not exactly >>> what I expected from the description of a lazy queue. >>> >>> >>> <https://lh3.googleusercontent.com/-XSzuKvXxMxI/Wb8uHb7LKmI/EYA/G6V5v93_yGcJW6jFPCnQUUBOp1LGCm7nQCLcBGAs/s1600/rabbitmqLimit.png> >>> >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "clojure-rabbitmq" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure-rabbit...@googlegroups.com . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> MK >> >> Staff Software Engineer, Pivotal/RabbitMQ >> > > > > -- > MK > > Staff Software Engineer, Pivotal/RabbitMQ > -- You received this message because you are subscribed to the Google Groups "clojure-rabbitmq" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure-rabbitmq+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [clojure-rabbitmq] Are lazy queues still bound by memory limit?
The point of lazy queues is not that nothing ever is stored in memory. It's that queues try to move messages to disk very aggressively (default mode keeps a portion in RAM using metrics such as ingress/egress rates and various configurable values). On Sun, Sep 17, 2017 at 10:29 PM, Michael Klishin <mklis...@pivotal.io> wrote: > This is a question for rabbitmq-users but I'm happy to answer it here. > > Lazy queues per se don't have a limit but RabbitMQ message store still > does: when messages > are sent to the message store [as opposed to being embedded into queue > index], message store index(es) > come into play and by default the index is in-memory. > > There is a plugin [1] that lets you use LevelDB for the index. Even though > LevelDB is an efficient > data store, expect a write throughput hit compared to the default > implementation. > > 1. https://github.com/rabbitmq/rabbitmq-msg-store-index-eleveldb > > On Sun, Sep 17, 2017 at 8:25 PM, Steve Suehs <skelter@gmail.com> > wrote: > >> I've been experimenting with Clojure and RabbitMQ using the langohr >> library. Thank you for building and sharing it. >> >> I've been learning more about the care and feeding of a rabbitmq server. >> I crashed mine installed on my dev laptop several times by filling a queue >> with messages. If queues fill up and memory gets tight, a server can be >> overwhelmed and fall over. If I set or arrange for a reasonable memory >> limit, things go ok. I am now running RabbitMQ in a Docker container with a >> memory limit. >> >> My latest experiments were with lazy queues, which are supposed to write >> events to disk and not exhaust memory. I still seem to hit..well, not hit, >> rather, an asymptotic limit at around 261 million events in the queue. Is >> this considered outrageously and unreasonably large? Publication rate >> slowed to 700/s. >> >> I also notice that if the consumers are caught up with the producer >> everything seems to scream at twice the rate than when storage is used. If >> storage is used I'm seeing about 6000r/s. That's cool...just not exactly >> what I expected from the description of a lazy queue. >> >> >> <https://lh3.googleusercontent.com/-XSzuKvXxMxI/Wb8uHb7LKmI/EYA/G6V5v93_yGcJW6jFPCnQUUBOp1LGCm7nQCLcBGAs/s1600/rabbitmqLimit.png> >> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "clojure-rabbitmq" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure-rabbitmq+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > MK > > Staff Software Engineer, Pivotal/RabbitMQ > -- MK Staff Software Engineer, Pivotal/RabbitMQ -- You received this message because you are subscribed to the Google Groups "clojure-rabbitmq" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure-rabbitmq+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [clojure-rabbitmq] Are lazy queues still bound by memory limit?
This is a question for rabbitmq-users but I'm happy to answer it here. Lazy queues per se don't have a limit but RabbitMQ message store still does: when messages are sent to the message store [as opposed to being embedded into queue index], message store index(es) come into play and by default the index is in-memory. There is a plugin [1] that lets you use LevelDB for the index. Even though LevelDB is an efficient data store, expect a write throughput hit compared to the default implementation. 1. https://github.com/rabbitmq/rabbitmq-msg-store-index-eleveldb On Sun, Sep 17, 2017 at 8:25 PM, Steve Suehs <skelter@gmail.com> wrote: > I've been experimenting with Clojure and RabbitMQ using the langohr > library. Thank you for building and sharing it. > > I've been learning more about the care and feeding of a rabbitmq server. > I crashed mine installed on my dev laptop several times by filling a queue > with messages. If queues fill up and memory gets tight, a server can be > overwhelmed and fall over. If I set or arrange for a reasonable memory > limit, things go ok. I am now running RabbitMQ in a Docker container with a > memory limit. > > My latest experiments were with lazy queues, which are supposed to write > events to disk and not exhaust memory. I still seem to hit..well, not hit, > rather, an asymptotic limit at around 261 million events in the queue. Is > this considered outrageously and unreasonably large? Publication rate > slowed to 700/s. > > I also notice that if the consumers are caught up with the producer > everything seems to scream at twice the rate than when storage is used. If > storage is used I'm seeing about 6000r/s. That's cool...just not exactly > what I expected from the description of a lazy queue. > > > <https://lh3.googleusercontent.com/-XSzuKvXxMxI/Wb8uHb7LKmI/EYA/G6V5v93_yGcJW6jFPCnQUUBOp1LGCm7nQCLcBGAs/s1600/rabbitmqLimit.png> > > > > -- > You received this message because you are subscribed to the Google Groups > "clojure-rabbitmq" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure-rabbitmq+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- MK Staff Software Engineer, Pivotal/RabbitMQ -- You received this message because you are subscribed to the Google Groups "clojure-rabbitmq" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure-rabbitmq+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[clojure-rabbitmq] Are lazy queues still bound by memory limit?
I've been experimenting with Clojure and RabbitMQ using the langohr library. Thank you for building and sharing it. I've been learning more about the care and feeding of a rabbitmq server. I crashed mine installed on my dev laptop several times by filling a queue with messages. If queues fill up and memory gets tight, a server can be overwhelmed and fall over. If I set or arrange for a reasonable memory limit, things go ok. I am now running RabbitMQ in a Docker container with a memory limit. My latest experiments were with lazy queues, which are supposed to write events to disk and not exhaust memory. I still seem to hit..well, not hit, rather, an asymptotic limit at around 261 million events in the queue. Is this considered outrageously and unreasonably large? Publication rate slowed to 700/s. I also notice that if the consumers are caught up with the producer everything seems to scream at twice the rate than when storage is used. If storage is used I'm seeing about 6000r/s. That's cool...just not exactly what I expected from the description of a lazy queue. <https://lh3.googleusercontent.com/-XSzuKvXxMxI/Wb8uHb7LKmI/EYA/G6V5v93_yGcJW6jFPCnQUUBOp1LGCm7nQCLcBGAs/s1600/rabbitmqLimit.png> -- You received this message because you are subscribed to the Google Groups "clojure-rabbitmq" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure-rabbitmq+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ANN] psq.clj 0.0.2 – Persistent Priority Search Queues
Great work. I love the Clojure state for data-structures. People often
underestimate how much access to quality data-structures is important for a
language. By the way, www.data.avl is not a valid link.
On Sunday, 20 November 2016 12:32:31 UTC-8, Michał Marczyk wrote:
>
> Hi,
>
> I am pleased to announce the 0.0.2 release of psq.clj, a Clojure
> priority search queue library based on Ralf Hinze's priority search
> pennants (see Ralf Hinze, "A Simple Implementation Technique for
> Priority Search Queues"):
>
> https://github.com/michalmarczyk/psq.clj
>
> https://clojars.org/psq.clj
>
> Priority search queues simultaneously behave like sorted maps with
> respect to their keys and like priority queues with respect to their
> entries, with the priority queue ordering based on the values, or
> priorities, attached to the keys. Additionally they support several
> operations that blend the sorted map and priority queue aspects
> together, such as "peek in key range" (retrieve a minimum-priority
> entry – NB. ties on priorities are supported – within the given key
> range) and optimized subseq-like traversals of key ranges that only
> return those entries whose priorities fall below a certain threshold
> (this is much more efficient than subseq + filter on a regular sorted
> map).
>
> psq.clj priority search queues support the full data.avl abstract data
> type (with the exception of the transient API which I am not sure
> would bring a significant return on the additional memory cost in this
> context). The full public API is as follows:
>
> ;;; there is a single public namespace:
> (require '[psq.clj :as psq])
>
>
> ;;; 1. factory functions
>
> ;; the psq.clj counterpart to clojure.core/sorted-map:
> (psq/psqueue key priority …)
> ;= {key priority …}
>
> ;; a version of the above that takes keys+priorities in a seqable:
> (psq/psqueue* [key priority …])
>
> ;; a factory that accepts a map or seqable of map entries:
> (psq/psq {key priority …})
> (psq/psq [[key priority] …])
>
> ;; versions of the above that take *two* custom comparators: the
> ;; first one determines the PSQ's key order, the second one is used
> ;; for priorities:
> (psq/psqueue-by > > key priority …) ; reverse numeric order on keys
> ; and priorities
>
> psq/psqueue-by*, psq/psq-by ; like psq/psqueue*, psq/psq, but with
> ; custom comparators
>
>
> ;;; 2. regular sorted map API; NB. (r)(sub)seq use key order
>
> (seq (psq/psqueue 0 10 1 9))
> ;= ([0 10] [1 9])
>
> ;; also supported: assoc, dissoc, conj, rseq, subseq, rsubseq
>
>
> ;;; 3. nearest neighbour lookups
> (psq/nearest (psq/psq {0 1 4 5 9 10}) > 3)
> ;= [4 5]
>
>
> ;;; 4. nth, rank in key order
>
> (nth (psq/psqueue 0 3 6 -3) 0)
> ;= [0 3]
>
> (psq/rank (psq/psqueue 0 3 6 -3) 6) ; returns long, -1 for not found
> ;= 1
>
>
> ;;; 5. priority queue API based on values/priorities
>
> (peek (psq/psqueue 0 3 1 -3))
> ;= [1 -3]
>
> (pop (psq/psqueue 0 3 1 -3))
> ;= {0 3}
>
> ;; seq over the PSQ in order of non-decreasing priorities
> (psq/priority-seq (psq/psqueue 0 3 1 -3))
> ;= ([1 -3] [0 3])
>
>
> ;;; 6. splits, subranges – with structural sharing in the common
> ;;;parts, but no holding on to keys outside the stated range for
> ;;;GC purposes
>
> ;; return a vector of
> ;;
> ;; 1. a fully independent PSQ comprising the entries of the
> ;; input PSQ to the left of the given key,
> ;;
> ;; 2. the entry at the given key, or nil if not present,
> ;;
> ;; 3. a fully independent PSQ comprising the entries of the
> ;; input PSQ to the right of the given key:
> (psq/split (psq/psqueue* (range 10)) 4)
> ;= [{0 1 2 3} [4 5] {6 7 8 9}]
>
> ;; like subseq, but returns an independent PSQ
> (psq/subrange (psq/psqueue* (range 10)) >= 4 < 8)
> ;= {4 5 6 7}
>
>
> ;;; 7. priority-bounded traversals:
> (psq/seq<= a-psq priority-upper-bound)
>
> ;; also supported: rseq<=, subseq<=, rsubseq<= (the latter two
> ;; with subseq/rsubseq-like arguments to specify key bounds) and
> ;; < variants of all of the above (seq< etc.)
>
> All of the above operations can be performed in O(log n) time, with
> the exception of peek, which takes constant time, and seq-like
> operations, which are sensitive to the number of entries actually
> produced. Priority-bounded traversals take O(r(log n - log r) + r)
> time, where r is the number of entries actually returned.
>
> In practice, psq.c
[ANN] maxiphobe 0.0.1 – Persistent Meldable Priority Queues
Hi, I am pleased to announce the initial release of maxiphobe, a meldable priority queue library based on Okasaki's maxiphobic heaps (see Okasaki, "Alternatives to Two Classic Data Structures"). Maxiphobic heaps are a strikingly simple purely functional approach to implementing priority queues that offers very reasonable performance given the available operations. https://github.com/michalmarczyk/maxiphobe https://clojars.org/maxiphobe Maxiphobe priority queues are, in effect, persistent lists that maintain a non-decreasing order of elements, as determined either by Clojure's default comparator or a custom comparator passed in by the user. Additionally, instances using the same comparator can be "melded" into a single priority queue containing all the elements of the inputs. (require '[maxiphobe.core :as pq]) (pq/pq [0 -1 3 4 2]) ;= (-1 0 2 3 4) (pq/pqueue 0 -1 3 4 2) ;= (-1 0 2 3 4) (pq/pq-by > [0 -1 3 4 2]) ;= (4 3 2 0 -1) (pq/pqueue-by > 0 -1 3 4 2) ;= (4 3 2 0 -1) (peek (pq/pqueue-by > 0 -1 3 4 2)) ; or first ;= 4 (pop (pq/pqueue-by > 0 -1 3 4 2)); or next ;= (3 2 0 -1) (pq/meld (pq/pqueue -1 3 2 8) (pq/pqueue 0 -3 2 4)) ;= (-3 -1 0 2 2 3 4 8) To include maxiphobe in your project: [maxiphobe "0.0.1"] maxiphobe maxiphobe 0.0.1 compile "maxiphobe:maxiphobe:0.0.1" See below for some benchmark results. Cheers, Michał Benchmarks == 1. Compare basic operations to sorted-set-as-PQ and to regular (FIFO) peristent queues to get a high-level picture of maxiphobe performance. FIFO queue operations are included to provide a point of reference – they are, of course, completely different semantically. (let [pq (pq/pq (range 1000)) s (apply sorted-set (range 1000)) q (into clojure.lang.PersistentQueue/EMPTY (range 1000))] (assert (= (seq s) pq q)) (println "peek") (c/quick-bench (peek pq)) (c/quick-bench (first s)) (c/quick-bench (peek q)) (println "pop") (c/quick-bench (pop pq)) (c/quick-bench (disj s (first s))) (c/quick-bench (pop q)) (let [pq (nth (iterate pop pq) 500) s (apply sorted-set pq) q (nth (iterate pop q) 500)] (assert (= (seq s) pq q)) (println "pop2") (c/quick-bench (pop pq)) (c/quick-bench (disj s (first s))) (c/quick-bench (pop q peek ;;; maxiphobe PQ: ;;; (peek pq) Evaluation count : 32624694 in 6 samples of 5437449 calls. Execution time mean : -2.422742 ns Execution time std-deviation : 0.630313 ns Execution time lower quantile : -2.934682 ns ( 2.5%) Execution time upper quantile : -1.681600 ns (97.5%) Overhead used : 21.406378 ns ;;; sorted set: ;;; (first s) Evaluation count : 1944576 in 6 samples of 324096 calls. Execution time mean : 3.262937 µs Execution time std-deviation : 440.634979 ns Execution time lower quantile : 2.728010 µs ( 2.5%) Execution time upper quantile : 3.753706 µs (97.5%) Overhead used : 21.406378 ns ;;; queue ;;; (peek q) Evaluation count : 22712976 in 6 samples of 3785496 calls. Execution time mean : 7.395679 ns Execution time std-deviation : 0.615558 ns Execution time lower quantile : 6.766551 ns ( 2.5%) Execution time upper quantile : 8.050620 ns (97.5%) Overhead used : 21.406378 ns pop ;;; maxiphobe PQ: ;;; (pop pq) Evaluation count : 1107840 in 6 samples of 184640 calls. Execution time mean : 2.143321 µs Execution time std-deviation : 1.064519 µs Execution time lower quantile : 362.028152 ns ( 2.5%) Execution time upper quantile : 3.175097 µs (97.5%) Overhead used : 21.406378 ns ;;; sorted set: ;;; (disj s (first s)) Evaluation count : 508608 in 6 samples of 84768 calls. Execution time mean : 10.498305 µs Execution time std-deviation : 2.658746 µs Execution time lower quantile : 6.260925 µs ( 2.5%) Execution time upper quantile : 12.859465 µs (97.5%) Overhead used : 21.406378 ns ;;; queue: ;;; (pop q) Evaluation count : 8807040 in 6 samples of 1467840 calls. Execution time mean : 730.736239 ns Execution time std-deviation : 122.924195 ns Execution time lower quantile : 561.778923 ns ( 2.5%) Execution time upper quantile : 839.133234 ns (97.5%) Overhead used : 21.406378 ns pop2 ;;; maxiphobe PQ: ;;; (pop pq) Evaluation count : 1816404 in 6 samples of 302734 calls. Execution time mean : 2.874078 µs Execution time std-deviation : 310.794245 ns Execution time lower quantile : 2.614476 µs ( 2.5%) Execution time upper quantile : 3.233927 µs (97.5%) Overhead used : 21.406378 ns ;;; sorted set: ;;; (disj s (first s)) Evaluation count : 554400 in 6 samples of 92400 calls. Execution time mean : 1
[ANN] psq.clj 0.0.2 – Persistent Priority Search Queues
Hi,
I am pleased to announce the 0.0.2 release of psq.clj, a Clojure
priority search queue library based on Ralf Hinze's priority search
pennants (see Ralf Hinze, "A Simple Implementation Technique for
Priority Search Queues"):
https://github.com/michalmarczyk/psq.clj
https://clojars.org/psq.clj
Priority search queues simultaneously behave like sorted maps with
respect to their keys and like priority queues with respect to their
entries, with the priority queue ordering based on the values, or
priorities, attached to the keys. Additionally they support several
operations that blend the sorted map and priority queue aspects
together, such as "peek in key range" (retrieve a minimum-priority
entry – NB. ties on priorities are supported – within the given key
range) and optimized subseq-like traversals of key ranges that only
return those entries whose priorities fall below a certain threshold
(this is much more efficient than subseq + filter on a regular sorted
map).
psq.clj priority search queues support the full data.avl abstract data
type (with the exception of the transient API which I am not sure
would bring a significant return on the additional memory cost in this
context). The full public API is as follows:
;;; there is a single public namespace:
(require '[psq.clj :as psq])
;;; 1. factory functions
;; the psq.clj counterpart to clojure.core/sorted-map:
(psq/psqueue key priority …)
;= {key priority …}
;; a version of the above that takes keys+priorities in a seqable:
(psq/psqueue* [key priority …])
;; a factory that accepts a map or seqable of map entries:
(psq/psq {key priority …})
(psq/psq [[key priority] …])
;; versions of the above that take *two* custom comparators: the
;; first one determines the PSQ's key order, the second one is used
;; for priorities:
(psq/psqueue-by > > key priority …) ; reverse numeric order on keys
; and priorities
psq/psqueue-by*, psq/psq-by ; like psq/psqueue*, psq/psq, but with
; custom comparators
;;; 2. regular sorted map API; NB. (r)(sub)seq use key order
(seq (psq/psqueue 0 10 1 9))
;= ([0 10] [1 9])
;; also supported: assoc, dissoc, conj, rseq, subseq, rsubseq
;;; 3. nearest neighbour lookups
(psq/nearest (psq/psq {0 1 4 5 9 10}) > 3)
;= [4 5]
;;; 4. nth, rank in key order
(nth (psq/psqueue 0 3 6 -3) 0)
;= [0 3]
(psq/rank (psq/psqueue 0 3 6 -3) 6) ; returns long, -1 for not found
;= 1
;;; 5. priority queue API based on values/priorities
(peek (psq/psqueue 0 3 1 -3))
;= [1 -3]
(pop (psq/psqueue 0 3 1 -3))
;= {0 3}
;; seq over the PSQ in order of non-decreasing priorities
(psq/priority-seq (psq/psqueue 0 3 1 -3))
;= ([1 -3] [0 3])
;;; 6. splits, subranges – with structural sharing in the common
;;;parts, but no holding on to keys outside the stated range for
;;;GC purposes
;; return a vector of
;;
;; 1. a fully independent PSQ comprising the entries of the
;; input PSQ to the left of the given key,
;;
;; 2. the entry at the given key, or nil if not present,
;;
;; 3. a fully independent PSQ comprising the entries of the
;; input PSQ to the right of the given key:
(psq/split (psq/psqueue* (range 10)) 4)
;= [{0 1 2 3} [4 5] {6 7 8 9}]
;; like subseq, but returns an independent PSQ
(psq/subrange (psq/psqueue* (range 10)) >= 4 < 8)
;= {4 5 6 7}
;;; 7. priority-bounded traversals:
(psq/seq<= a-psq priority-upper-bound)
;; also supported: rseq<=, subseq<=, rsubseq<= (the latter two
;; with subseq/rsubseq-like arguments to specify key bounds) and
;; < variants of all of the above (seq< etc.)
All of the above operations can be performed in O(log n) time, with
the exception of peek, which takes constant time, and seq-like
operations, which are sensitive to the number of entries actually
produced. Priority-bounded traversals take O(r(log n - log r) + r)
time, where r is the number of entries actually returned.
In practice, psq.clj priority search queues are slower than Clojure's
built-in sorted maps for pure sorted-map operations, typically by a
factor of 2 to 4. In return, the PSQ operations are very performant;
subseq<= can be over three orders of magnitude faster than the
equivalent combination of subseq and filter applied to a regular
sorted map, subrange takes tens of microseconds when the size of the
input is on the order of hundreds of thousands, peek takes less than
10 ns etc. See end of this message for some benchmark results.
Mark Engelberg's excellent data.priority-map is another natural point
of comparison. In short, whereas data.priority-map is hash-map-like,
with extremely fast lookups and no particular ordering of keys,
psq.clj is sorted-map-like, with a much richer abstract data type at
the cost of noticeably slower lookups. assoc is often faster with
psq.cl
Re: What are favored Redis-backed queues?
Also, I should mention that Ruby doesn't have very good built in parallelism support (no true threads when I was using, though this might have changed). As such, I've seen a fair bit of usage of Resque running on a single machine. This would be an insane overcomplication in Clojure given all it's concurrency/parallelism support. So unless you're actually planning to distribute tasks across a cluster, keep it simple and stick to things like the built in Clojure reference types and core.async. Chris -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: What are favored Redis-backed queues?
I think you mean to be asking specifically about *job* queuing using Redis. You don't need anything other than Redis + Carmine to create queues. But obviously, the value of Resque (note spelling) is that it *uses* Redis in a number of non-trivial ways, such that all of the features above are available. There does appear to have been a Clojure port of Resqueue (https://github.com/jxa/resque-clojure) which garnered a bit of attention, but it hasn't been updated in a while, so I don't know what the state of it is. It's worth considering that if you don't *need* something very robust (and depending on your situation), you could just use Redis + Carmine directly to pass messages around (possibly representing jobs), and have workers pull from the message queue. There is nothing else you really need for this; it's quite straight forward. The rub is that you don't get any of the higher level features; if a job dies for instance, there's no way to know about that without building it yourself. It's also worth considering whether a job queue is really the right mode of distributed computing. This area is a real strength of Clojure's, and there are a lot of offerings here. I say this because the offerings are a lot slimmer in Ruby land (at least when I was working with it), and so if you're thinking queues based on prior experience with Ruby, I'd definitely dig a little more to make sure it's the right model for you. It might seem like already having Redis in your system makes it a good goto, but if you're bending the model it'll end up being more work. Some other things to consider: * storm * onyx * quasar (https://github.com/puniverse/quasar) * pulsar (http://docs.paralleluniverse.co/pulsar) Chris -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
What are favored Redis-backed queues?
I'm looking at this old post from Github, that lists the features they were looking for in a message queue: - Persistence - See what's pending - Modify pending jobs in-place - Tags - Priorities - Fast pushing and popping - See what workers are doing - See what workers have done - See failed jobs - Kill fat workers - Kill stale workers - Kill workers that are running too long - Keep Rails loaded / persistent workers - Distributed workers (run them on multiple machines) - Workers can watch multiple (or all) tags - Don't retry failed jobs - Don't release failed jobs https://github.com/blog/542-introducing-resque They ended up creating Rescue, and using Redis in the background. Lately I've been looking at Carmine, but I'm wondering, what are some of the queues that people are using with Clojure, in particular, those using Redis? (Since the subject is potentially immense, I figure I should limit conversation to Redis. I am already using Redis in production, so for me anything using Redis is easy to add in.) -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: STM history queues, when is a snapshot pushed
Having learned a little more about refs and transactions from M. Fogus and
C. Houser The Joy of Clojure, Second Editionhttp://www.manning.com/fogus2/,
I altered the stress-ref function from 10.2.4. Using Clojure 1.5.1:
(defn stress-ref [r]
(let [slow-tries (atom 0)]
(future
(dosync
(swap! slow-tries inc)
(Thread/sleep 200)
@r)
(println (format r is: %s, meta: %s, history: %d, after: %d tries
@r (meta @r) (.getHistoryCount r) @slow-tries)))
(dotimes [i 500]
(Thread/sleep 10)
(dosync (alter r #(let [r0 (first %)]
(condp = r0
:identical %
:= (with-meta % {:mkw (inc (:mkw (meta
%)))})
(with-meta [(inc r0)] {:mkw (inc (:mkw (meta
%)))}))
:done))
Using this function in the following ways
user= (stress-ref (ref (with-meta [:identical] {:mkw 0})))
:done
r is: [:identical], meta: {:mkw 0}, history: 10, after: 26 tries
user= (stress-ref (ref (with-meta [:=] {:mkw 0})))
:done
r is: [:=], meta: {:mkw 500}, history: 10, after: 26 tries
user= (stress-ref (ref (with-meta [0] {:mkw 0})))
:done
r is: [500], meta: {:mkw 500}, history: 10, after: 26 tries
From these results I infer that a snapshot is pushed to history whenever
alter (et al) are used. Any efficiencies need to be implemented in the
Clojure code, eg.
(defn stress-ref-equal [r]
(let [slow-tries (atom 0)]
(future
(dosync
(swap! slow-tries inc)
(Thread/sleep 200)
@r)
(println (format r is: %s, meta: %s, history: %d, after: %d tries
@r (meta @r) (.getHistoryCount r) @slow-tries)))
(dotimes [i 500]
(Thread/sleep 10)
(dosync (let [r0 (first @r)]
(condp = r0
:identical (ensure r)
:= (ensure r)
(alter r #(with-meta [(inc r0)] {:mkw (inc (:mkw (meta
%)))}))
:done))
user= (stress-ref-equal (ref (with-meta [:identical] {:mkw 0})))
r is: [:identical], meta: {:mkw 0}, history: 0, after: 1 tries
:done
user= (stress-ref-equal (ref (with-meta [:=] {:mkw 0})))
r is: [:=], meta: {:mkw 0}, history: 0, after: 1 tries
:done
user= (stress-ref-equal (ref (with-meta [0] {:mkw 0})))
:done
r is: [500], meta: {:mkw 500}, history: 10, after: 26 tries
The documentation at Refs and Transactions http://clojure.org/refs only
refers to change without defining what is meant by change. Given the
admonition not to rely on inferences as made above, what's a coder to do?
Future optimization of refs and transactions may break the code.
As a beginner, I can't tell if I'm just confused, or if this is actually
confusing.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
STM history queues, when is a snapshot pushed
Is a new snapshot pushed onto the history queue of a ref when 1. every time the ref is a target of alter, commute, ref-set, reset, alter-meta!, or reset-meta!, or 2. only when the committed value for the ref is not identical to the value at the end of the history queue? When checking the committability of a ref before the end of a transaction, is the check that 1. the end position of the history queue for that ref is the same as it was at the beginning of the transaction, or 2. the value of the ref at the end of the history queue is identical to the value at the beginning of the transaction? For both questions, choice 1 appears to be unquestionable safe, but could sponsor a higher number of retries. The choice 2s may conserve some retries, but I'm not sure about the safety. For both questions, the implementation complexity and efficiency seem similar. At any rate, the actual implementation is not a part of the documented API, yet could affect how one codes transaction functions. Even if I dug through the Java source to form my own impressions, one is not supposed to rely on such behaviour. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: what is a good book about Java queues?
Two queues cover the 80% case: an implementation of BlockingQueue (array, linked list backed), and clojure.lang.PersistentQueue. The former provides concurrency semantics: there are no 'stale values' issues associated to mutation, and you can choose whether your reads/writes are blocking (which is important, given that typically one wants bounded size queues) or not. The latter, just like the other persistent data structures, must be wrapped in a reference type in order to express change. There are also thread-unsafe queues (i.e. need locking), and SynchronousQueue, which while it's not actually a queue (it can never store a single element!), is a very handy reference type - like Clojure promises but reusable. On Thursday, March 7, 2013 9:32:38 PM UTC+1, larry google groups wrote: At some point on this mailist, someone suggested that to best understand concurrency in Java, once should read: Java Concurrency In Practice http://www.amazon.com/Java-Concurrency-Practice-Brian-Goetz/dp/0321349601/ref=sr_1_1?ie=UTF8qid=1362688125sr=8-1keywords=java+concurrency+in+practice So I read it, but I am surprised that it says fairly little about working with queues. I am curious if anyone would recommend a book for learning more about working with Java queues? I suffer the problem that I learned Clojure without first learning Java, and Clojure leans on Java rather heavily, so I am now trying to come up to speed on Java. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: what is a good book about Java queues?
Yes, the javadoc is quite good, paste a link here: http://docs.oracle.com/javase/7/docs/api/java/util/concurrent/BlockingQueue.html On Friday, March 8, 2013 4:32:38 AM UTC+8, larry google groups wrote: At some point on this mailist, someone suggested that to best understand concurrency in Java, once should read: Java Concurrency In Practice http://www.amazon.com/Java-Concurrency-Practice-Brian-Goetz/dp/0321349601/ref=sr_1_1?ie=UTF8qid=1362688125sr=8-1keywords=java+concurrency+in+practice So I read it, but I am surprised that it says fairly little about working with queues. I am curious if anyone would recommend a book for learning more about working with Java queues? I suffer the problem that I learned Clojure without first learning Java, and Clojure leans on Java rather heavily, so I am now trying to come up to speed on Java. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
what is a good book about Java queues?
At some point on this mailist, someone suggested that to best understand concurrency in Java, once should read: Java Concurrency In Practice http://www.amazon.com/Java-Concurrency-Practice-Brian-Goetz/dp/0321349601/ref=sr_1_1?ie=UTF8qid=1362688125sr=8-1keywords=java+concurrency+in+practice So I read it, but I am surprised that it says fairly little about working with queues. I am curious if anyone would recommend a book for learning more about working with Java queues? I suffer the problem that I learned Clojure without first learning Java, and Clojure leans on Java rather heavily, so I am now trying to come up to speed on Java. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: what is a good book about Java queues?
On Mar 8, 1:32 am, larry google groups lawrencecloj...@gmail.com wrote: At some point on this mailist, someone suggested that to best understand concurrency in Java, once should read: Java Concurrency In Practice http://www.amazon.com/Java-Concurrency-Practice-Brian-Goetz/dp/032134... So I read it, but I am surprised that it says fairly little about working with queues. I am curious if anyone would recommend a book for learning more about working with Java queues? I suffer the problem that I learned Clojure without first learning Java, and Clojure leans on Java rather heavily, so I am now trying to come up to speed on Java. I've found their Javadoc content to be quite informative. Looking at the source code should be useful too. The ConcurrentLinkedQueue doc mentions http://www.cs.rochester.edu/u/michael/PODC96.html as the reference. Shantanu -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: Queues in Clojure
2010/12/3 Andreas Kostler andreas.koestler.le...@gmail.com: Hi All, May I cite an Author of a populer Clojure book: If you find yourself wishing yourself to repeatedly check a work queue to see if there's an item of work to be popped off, or if you want to use a queue to send a task to another thread, you do *not* want the PersistenQueue discussed in this section Why do I not want to use clojure.lang.PersistentQueue for that purpose and what would I want to use instead? Can anyone fill me in please? clojure.lang.PersistentQueue is a collection data structure with queue operations. It is persistent, so it lets you (and other threads) create modified versions of it with elements enqueued or dequeued in a thread-safe manner. java.util.concurrent.LinkedBlockingQueue[1] is not a persistent data structure. It is often used as a communication channel between threads, since it allows its operations to block (potentially with a timeout). One thread can dequeue (and block if the queue is empty) simultaneously as another thread can enqueue (and block if the queue is full) at the other end. It makes it easy to make programs in a producer-consumer style. // raek [1] http://download.oracle.com/javase/6/docs/api/java/util/concurrent/LinkedBlockingQueue.html -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Queues in Clojure
Hi All, May I cite an Author of a populer Clojure book: If you find yourself wishing yourself to repeatedly check a work queue to see if there's an item of work to be popped off, or if you want to use a queue to send a task to another thread, you do *not* want the PersistenQueue discussed in this section Why do I not want to use clojure.lang.PersistentQueue for that purpose and what would I want to use instead? Can anyone fill me in please? -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Queues
Is there any reason to prefer lists over vectors or vice versa for implementing queues? It seems that for both lists and vectors, adding and removing at one end (front for lists, end for vectors) is cheap, whereas it is expensive at the other end. For queues you need to add at one end and remove from the other, so one of the two operations is necessarily expensive. But is there a difference between the two? Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Queues
Konrad Hinsen a écrit : Is there any reason to prefer lists over vectors or vice versa for implementing queues? It seems that for both lists and vectors, adding and removing at one end (front for lists, end for vectors) is cheap, whereas it is expensive at the other end. For queues you need to add at one end and remove from the other, so one of the two operations is necessarily expensive. But is there a difference between the two? Konrad. If you haven't tried it yet, there's clojure.lang.PersistentQueue: user= (def empty-queue clojure.lang.PersistentQueue/EMPTY) #'user/empty-queue user= (conj empty-queue 1) (1) user= (conj *1 2 3 4) (1 2 3 4) user= (pop *1) (2 3 4) user= (peek *2) Christophe -- Professional: http://cgrand.net/ (fr) On Clojure: http://clj-me.blogspot.com/ (en) --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Queues
2009/2/3 Konrad Hinsen konrad.hin...@laposte.net: Is there any reason to prefer lists over vectors or vice versa for implementing queues? It seems that for both lists and vectors, adding and removing at one end (front for lists, end for vectors) is cheap, whereas it is expensive at the other end. For queues you need to add at one end and remove from the other, so one of the two operations is necessarily expensive. But is there a difference between the two? You can use a pair of lists to implement a queue where the first list is used to dequeue items from and the second list is used to enqueue items to. When the first queue is empty, you replace it with a reversed version of the second queue. I don't have time to write this in clojure at the moment but in haskell it would be something like this: -- A queue consists of two lists of a's data Queue a = Queue [a] [a] -- i.e. when enqueueing we add the new item to the front of the second list enqueue (Queue as bs) i = Queue as (i:bs) -- when dequeueing we need to handle two cases: when A is emply and when A is not empty. In -- the first case we replace A with the reverse of B and replace B with the empty list, i.e. we -- move the enqueued elements from B to A and then run dequeue again with the new queue -- structure dequeue (Queue [] bs) = dequeue (Queue (reverse b) []) -- In the second case we simply return the head of A as the dequeued item and update the -- queue structure dequeue (Queue as bs) = (head as, Queue (tail as) bs) The point is that enqueueing and dequeueing are O(1) most cases and the reversing operation which is O(n) is amortised over the lifetime of the qeueue. Konrad. -- ! Lauri --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Queues
I don't know which of these two options are best in general, but I wonder; are persistance and immutability valuable properties of queues? Safe and cheap snap-shotting might be a nice feature of a queue but I (generally) wouldn't want it at the cost of more expensive put and take operations. Java already have concurrent, albeit mutable, queue implementations in the java.util.concurrent package. Since most operations you'd want to execute on a queue are writes, I can't help but think that a mutable datastructure is better suited, and that an attempt with immutable structures would most likely end up simulating mutability with some form of copy-on-write + CAS scheme anyway. Then again, safe snapshots might really be important, and that's where I'd use PersistentQueue as Christophe Grand mentioned. On Tue, Feb 3, 2009 at 2:35 PM, Konrad Hinsen konrad.hin...@laposte.net wrote: Is there any reason to prefer lists over vectors or vice versa for implementing queues? It seems that for both lists and vectors, adding and removing at one end (front for lists, end for vectors) is cheap, whereas it is expensive at the other end. For queues you need to add at one end and remove from the other, so one of the two operations is necessarily expensive. But is there a difference between the two? Konrad. -- Venlig hilsen / Kind regards, Christian Vest Hansen. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Queues
Lauri Pesonen a écrit : You can use a pair of lists to implement a queue where the first list is used to dequeue items from and the second list is used to enqueue items to. When the first queue is empty, you replace it with a reversed version of the second queue. Or you can use a seq and a vector like clojure.lang.PersistentQueue does thus you won't need to reverse the vector but the queue will retain references on dequeued items for some time (until the current seq is empty). Christophe --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Queues
On Feb 3, 2009, at 14:57, Christophe Grand wrote: If you haven't tried it yet, there's clojure.lang.PersistentQueue: I didn't, since it's well hidden - you can even search for PersistentQueue on the Clojure web site without finding anything. But it looks like just what I want - thanks! Konrad. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
Re: Queues
On Tue, Feb 3, 2009 at 2:35 PM, Konrad Hinsen konrad.hin...@laposte.netwrote: For queues you need to add at one end and remove from the other, so one of the two operations is necessarily expensive. No. Look here for hints: http://www.cs.bu.edu/teaching/c/queue/array/types.html http://www.cs.bu.edu/teaching/c/queue/array/types.html But also look here for other approaches: http://www.google.com/search?q=queue+persistent Cheers P. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en -~--~~~~--~~--~--~---
