My guess is that partition-all and partition use non-volatile references because none of the built-in stuff will return control back to the caller at a finer resolution than output value (AFAIK). That's why take needs volatile but partition-all doesn't (because for take the state persists between output values). It does mean that new transducing contexts would need to synchronize though - which core.async does through mutexes.
On Sunday, April 9, 2017 at 8:47:25 AM UTC-5, tbc++ wrote: > > The volatile! is needed for the case where a transducer is only used by > one thread at a time, but the thread executing the transducer may change > from one call to the next. This happens fairly often with core.async. If > you used a non-atomic, non-volatile mutable field, the JVM would be free to > perform several optimizations (like keeping the local in a CPU register) > that would cause the value to not properly propagate to other threads in > the case of a context switch. Using volatile! tells the JVM to flush all > writes to this field by the time the next memory barrier rolls around. It > also tells the JVM to make sure it doesn't cache the reads to this field > across memory barriers. > > It's a tricky subject, and one that's really hard to test, and frankly I > probably got some of the specifics wrong in that last paragraph, but that's > the general idea of why transducers use volatile!. > > Timothy > > On Sun, Apr 9, 2017 at 12:49 AM, Alexander Gunnarson < > alexander...@gmail.com <javascript:>> wrote: > >> EDIT: Transducers are actually not safe in `fold` contexts as I thought: >> >> (let [f (fn [i x] (println (str "i " i " " (Thread/currentThread))) >> (flush) x) >> r-map-indexed #(r/folder %2 (map-indexed %1))] >> (->> [6 7 8 9 10] >> (r-map-indexed f) >> (r/fold 1 (fn ([] (vector)) ([x] x) ([a b] (into a b))) conj))) >> >> Produces: >> >> i 0 Thread[ForkJoinPool-1-worker-2,5,main] >> i 2 Thread[ForkJoinPool-1-worker-1,5,main] >> i 3 Thread[ForkJoinPool-1-worker-1,5,main] >> i 4 Thread[ForkJoinPool-1-worker-1,5,main] >> i 1 Thread[ForkJoinPool-1-worker-3,5,main] >> >> So you would have to be careful to e.g. create different `map-indexed` >> transducers for single-threaded (e.g. `unsynchronized-mutable` box) and >> multi-threaded (e.g. `atom` box) contexts. >> >> On Sunday, April 9, 2017 at 2:10:06 AM UTC-4, Alexander Gunnarson wrote: >>> >>> I was wondering the same thing, shintotomoe. This thread >>> <https://groups.google.com/forum/#!topic/clojure/CjxK7xEsOKQ> talks >>> about it as well. I think it's safe to assume that since `ArrayList` uses >>> unsynchronized mutability internally (a quick review of the GrepCode >>> entry for `ArrayList` confirms this >>> <http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/8u40-b25/java/util/ArrayList.java>), >>> >>> then we can rest assured that a `volatile` box as opposed to a totally >>> unsynchronized mutable variable is unnecessary, even in the context of >>> `fold`. After all, `reduce` (and by extension, `transduce`) is only ever >>> going to be single-threaded unless the data structure in question >>> unexpectedly implements a multithreaded reduce, which should never happen >>> (and if it does, you likely have bigger problems). To be honest, I'm not >>> sure why `volatile` is used in transducers instead of e.g. an >>> `unsynchronized-mutable` box. There may be a good reason, but I'm not >>> seeing it immediately. I'd love to learn. >>> >>> On Thursday, January 1, 2015 at 10:36:13 PM UTC-5, shintotomoe wrote: >>>> >>>> Thank you for the superfast response. I take it implementing your own >>>> transducing process is not something you would usually do unless you have >>>> a >>>> unique use case (my own use case being already implemented by chan taking >>>> a >>>> transducer). >>>> >>>> Still, I was wondering about the use of ArrayList in partition-all, and >>>> the recommendation to use volatiles inside transducers, which seem at >>>> odds. >>>> It seems we don't need to implement transducers in a thread-safe way. Is >>>> that correct? >>>> >>>> On Friday, January 2, 2015 12:58:51 PM UTC+11, tbc++ wrote: >>>>> >>>>> Core.async already has pipeline, pipeline-blocking and pipeline-async. >>>>> In addition you can use a transducer inside a channel. Use those instead. >>>>> >>>>> Timothy >>>>> >>>>> On Thu, Jan 1, 2015 at 6:55 PM, shintotomoe <tste...@atlassian.com> >>>>> wrote: >>>>> >>>>>> I was wondering how to apply a transducer inside a go process. What >>>>>> I've so far is the following >>>>>> >>>>>> (defn pipe-xform [in-ch out-ch xform] >>>>>> (let [tr >>>>>> (let [tr (xform (fn >>>>>> ([result] result) >>>>>> ([result input] (conj! result input))))] >>>>>> (fn >>>>>> ([] (locking tr (persistent! (tr (transient []))))) >>>>>> ([input] (locking tr (persistent! (tr (transient []) >>>>>> input))))))] >>>>>> (go-loop [] >>>>>> (if-some [value (<! in-ch)] >>>>>> (do (doseq [v (tr value)] >>>>>> (>! out-ch v)) >>>>>> (recur)) >>>>>> (do (doseq [v (tr)] >>>>>> (>! out-ch v)) >>>>>> (close! out-ch)))))) >>>>>> >>>>>> Now, I could just do >>>>>> >>>>>> (let [xf-ch (chan 1 xform)] >>>>>> (pipe in-ch xf-ch) >>>>>> (pipe xf-ch out-ch) >>>>>> >>>>>> >>>>>> Or just redesign my code so that I can create in-ch or out-ch with >>>>>> the transducer directly, but I was wondering whether there are any >>>>>> obvious >>>>>> flaws with the pipe-xform implementation. >>>>>> >>>>>> In particular, I was wondering about the locking. At first I was >>>>>> under the impression that transducers are thread-safe due to the use of >>>>>> volatiles, but looking at the partition-all transducer, which uses an >>>>>> ArrayList for its state, It appears that's not the case. >>>>>> >>>>>> Any feedback greatly appreciated. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Clojure" group. >>>>>> To post to this group, send email to clo...@googlegroups.com >>>>>> Note that posts from new members are moderated - please be patient >>>>>> with your first post. >>>>>> To unsubscribe from this group, send email to >>>>>> clojure+u...@googlegroups.com >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/clojure?hl=en >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Clojure" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to clojure+u...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> “One of the main causes of the fall of the Roman Empire was >>>>> that–lacking zero–they had no way to indicate successful termination of >>>>> their C programs.” >>>>> (Robert Firth) >>>>> >>>> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clo...@googlegroups.com >> <javascript:> >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+u...@googlegroups.com <javascript:> >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+u...@googlegroups.com <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > “One of the main causes of the fall of the Roman Empire was that–lacking > zero–they had no way to indicate successful termination of their C > programs.” > (Robert Firth) > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
