On Monday, April 10, 2017 at 1:57:10 PM UTC-5, Léo Noel wrote: > > What you said holds for reduction but not necessarily a parallel fold (see >> clojure.core.reducers/fold). >> > > Exactly, and that's why stateful transducers are explicitly forbidden in > fold and in core.async pipeline functions. > This is not related to memory visibility, this is due to the fact that > stateful transducers force the reducing process to be sequential. > Does it make any sense to parallelize map-indexed ? partition-all ? dedupe > ? >
Parallel transducers is something Rich has thought about some but that's a future effort. > Léo, I definitely agree that you can use unsynchronized mutable stateful >> transducers *as long as you can guarantee they'll be used only in >> single-threaded contexts.* >> > > The problem is at a lower level. The memory model of the JVM >> doesn't guarantee that changes to an unsynchronized non-volatile reference >> are visible to other threads. >> > > The Java Memory Model allows using unsynchronized variables to share data > across threads as long as a memory barrier is set between the writer and > the reader. For example, in the case of core.async, the channel lock sets a > barrier, and there is also (redundant) barriers for each volatile inside > transducers. > The JVM is pretty good at minimizing this stuff - so while you are stating these barriers are redundant and are implying that's an issue, it would not surprise me if the JVM is able to reduce or eliminate the impacts of that. At the very least, it's too difficult to reason about without a real perf test and numbers. > A transducing process could apply each step of the transduce using a >> thread from a pool and also not use a memory barrier > > > >> Transducers included in core cannot make the assumption that they will >> only be used that way. >> > > Yes, that makes sense that you can't make that assumption. > > > This is the key point : what assumptions a transducer can make ? > A transducer can assume it will be invoked by no more than one thread at a time > In my opinion, it is reasonable for a stateful transducer to assume that > the transducing context will fulfill the contract of "always passing the > result of step n to the first argument of step n+1". > > This assumption is powerful because it guarantees that there will always > be a memory barrier between two successive steps. > Proof (reductio ad absurdum) : without a memory barrier, the result of the > step n wouldn't be visible to the (potentially different) thread performing > the step n+1. > You're conflating the stateful values inside the transducer with the state returned by and passed into a transducer. That's a linkage that does not necessarily exist. > So here is my question to the language designers : is it reasonable to > assume that ? > No. > If yes, that means it's ok to use unsynchronized variables in stateful > transducers as long as they stay local. > > If no, that means we'll use synchronization in all stateful transducers, > with an obvious performance penalty and a benefit that remains unclear. > Fast wrong results are still wrong. I do not think it's at all obvious how this affects performance without running some benchmarks. Volatiles do not require flushing values to all cores or anything like that. They just define constraints - the JVM is very good at optimizing these kinds of things. It would not surprise me if an uncontended thread-contained volatile could be very fast (for the single-threaded transducer case) or that a volatile under a lock would be no worse than the lock by itself. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
