The problem is at a lower level. The memory model of the JVM doesn't guarantee that changes to an unsynchronized non-volatile reference are visible to other threads. Transducers don't have to worry about concurrency but they do have to worry about visibility of changes across different threads.
On Mon, Apr 10, 2017 at 8:37 AM, Léo Noel <leo.noel...@gmail.com> wrote: > This topic is of high interest to me as it is at the core of my current > works. I had a similar questioning a while ago and I have to say I'm even > more confused with this : > >> While transducing processes may provide locking to cover the visibility of >> state updates in a stateful transducer, transducers should still use >> stateful constructs that ensure visibility (by using volatile, atoms, etc). > > > I actually tried pretty hard to find a use case that would make > partition-all fail because of its unsynchronized local state, and did not > manage to find one that did not break any contract. I arrived at the > conclusion that it is always safe to use unsynchronized constructs in > stateful transducers. The reason is that you need to ensure that the result > of each step is given to the next, and doing so you will necessarily set a > memory barrier of some sort between each step. Each step happens-before the > next, and therefore mutations performed by the thread at step n are always > visible by the thread performing the step n+1. This is really brilliant : > when designing a transducer, you can be confident that calls to your > reducing function will be sequential and stop worrying about concurrency. > You just have to ensure that mutable state stays local. True encapsulation, > the broken promise of object-oriented programming. > > My point is that the transducer contract "always feed the result of step n > as the first argument of step n+1" is strong enough to safely use local > unsynchronized state. For this reason, switching partition-* transducers to > volatile constructs really sounds like a step backwards to me. However, > after re-reading the documentation on transducers, I found that this > contract is not explicitly stated. It is just *natural* to think this way, > because transducers are all about reducing processes. Is there a plan to > reconsider this principle ? I would be very interested to know what Rich has > in mind that could lead him to advise to overprotect local state of > transducers. > > > > On Monday, April 10, 2017 at 4:44:00 AM UTC+2, Alexander Gunnarson wrote: >> >> Thanks so much for your input Alex! It was a very helpful confirmation of >> the key conclusions arrived at in this thread, and I appreciate the >> additional elaborations you gave, especially the insight you passed on about >> the stateful transducers using `ArrayList`. I'm glad that I wasn't the only >> one wondering about the apparent lack of parity between its unsynchronized >> mutability and the volatile boxes used for e.g. `map-indexed` and others. >> >> As an aside about the stateful `take` transducer, Tesser uses the >> equivalent of one but skirts the issue by not guaranteeing that the first n >> items of the collection will be returned, but rather, n items of the >> collection in no particular order and starting at no particular index. This >> is achievable without Tesser by simply replacing the `volatile` in the >> `core/take` transducer with an `atom` and using it with `fold`. But yes, >> `take`'s contract is broken with this and so still follows the rule of thumb >> you established that `fold` can't use stateful transducers (at least, not >> without weird things like reordering of the indices in `map-indexed` and so >> on). >> >> That's interesting that `fold` can use transducers directly! I haven't >> tried that yet — I've just been wrapping them in an `r/folder`. >> >> On Sunday, April 9, 2017 at 10:22:13 PM UTC-4, Alex Miller wrote: >>> >>> Hey all, just catching up on this thread after the weekend. Rich and I >>> discussed the thread safety aspects of transducers last fall and the >>> intention is that transducers are expected to only be used in a single >>> thread at a time, but that thread can change throughout the life of the >>> transducing process (for example when a go block is passed over threads in a >>> pool in core.async). While transducing processes may provide locking to >>> cover the visibility of state updates in a stateful transducer, transducers >>> should still use stateful constructs that ensure visibility (by using >>> volatile, atoms, etc). >>> >>> The major transducing processes provided in core are transduce, into, >>> sequence, eduction, and core.async. All but core.async are single-threaded. >>> core.async channel transducers may occur on many threads due to interaction >>> with the go processing threads, but never happen on more than one thread at >>> a time. These operations are covered by the channel lock which should >>> guarantee visibility. Transducers used within a go block (via something like >>> transduce or into) occur eagerly and don't incur any switch in threads so >>> just fall back to the same old expectations of single-threaded use and >>> visibility. >>> >>> Note that there are a couple of stateful transducers that use ArrayList >>> (partition-by and partition-all). From my last conversation with Rich, he >>> said those should really be changed to protect themselves better with >>> volatile or something else. I thought I wrote up a ticket for this but looks >>> like maybe I didn't, so I will take care of that. >>> >>> Reducer fold is interesting in that each "bucket" is reduced via its >>> reduce function, which can actually use a transducer (since that produces a >>> reduce function), however, it can't be a stateful transducer (something like >>> take, etc). >>> >>> Hope that helps with respect to intent. >>> > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to a topic in the > Google Groups "Clojure" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/clojure/VQj0E9TJWYY/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
