Re: About transients no longer being safe in 1.7-alpha2
Yes -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
Will someone remember to update http://clojure.org/transients once 1.7.0 is
released?
On Monday, November 3, 2014 at 1:57:58 AM UTC-8, Daniel Marjenburgh wrote:
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even though
it's quite a big change.
With this change the following code is possible:
(let [m (transient {:a 0})
futs (for [n (range 100)]
(future (assoc! m :a (inc (:a m)]
(mapv deref futs) ; wait until futures are done
(persistent! m))
The results will vary per run, where it used to throw
an IllegalAccessError: Transient used by non-owner thread.
I understand the problems of not being able to have 1 go routine access a
transient, even though it would be safe, but this solution feels like it's
throwing out the baby with the bathwater. Basically, it's doing away with
what the following block on clojure.org http://clojure.org/transients
says:
*Transients enforce thread isolation**.* Because each result of a
transient operation shares (mutable) structure with the previous, it would
be very dangerous if more than one thread were to manipulate a transient at
once. In order to prevent this, transients will detect any (read or write)
use from a thread other than the one that created them and throw an
exception.
This may not sound like a concurrency story, but single-thread isolation
is actually a very useful concurrency semantic. The whole point of using
transients is that doing so is safe, because their use is an isolated
implementation detail of otherwise functional code. Having that be enforced
means that some things that would normally be very hard to make safe with
ordinary mutable data structures become easy.
I don't have a good solution for dealing with transients and logical
threads, but I would much prefer keeping the semantics of transients as
they are and maybe pass an option to transient to disable owner checking.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
Hi Atamert, yes, that is also wrong usage that is not caught. I do want to
focus on the current change though.
So the thought behind this change is that the birth-thread check was
unnecessary or too strict from the beginning and people should just use
transients correctly, caveat implementor. No more kids gloves.
There is a paragraph http://clojure.org/transients dedicated to
enforcement of thread-isolation, which I think is great. I also understand
we want to give programmers more flexibility in using transients in other
(mutlithreaded) contexts where the user knows it will be safe. I just think
throwing away the entire safeguard is overkill and it's making unsafe
Clojure code not only really easy, but even the default in the case of
transients. All programs not using core.async could benefit from the
owner-check.
Again, like Alex suggested, a flag like (transient m {} :thread-isolated?
false) or something (with the default to true) would at least ease my mind
a bit. It would also alert programmers in core async when they see it, that
they need to make sure only one logical thread should have access.
But they would run fine, if you used delay's instead of future's. Same
wrong usage, single threaded. So the issue is not really this new change.
(See my code with delay's earlier)
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
I'll be more careful before posted code that doesn't compile :)
Anyway, here's my output using clojure 1.7-alpha3
https://lh4.googleusercontent.com/-MqiKnTcxneE/VFigwvQtzBI/A08/vGvY2waCJdc/s1600/Screen%2BShot%2B2014-11-04%2Bat%2B10.43.17.png
It could be that depending on OS/env the second future starts after the
first one finishes (if you add more futures or Thread sleeps it might be
easier to reproduce).
But this example already shows that 2 threads are mutating the same
transient, otherwise, as Atamert pointed out, the result should have been
45.
Op dinsdag 4 november 2014 08:45:37 UTC+1 schreef Sean Corfield:
On Nov 3, 2014, at 11:13 PM, Daniel Marjenburgh dmarje...@gmail.com
javascript: wrote:
I was a bit too quick there and posted some errors int he code:
That’s still not quite right. I think you mean:
(let [v (transient {:a 0})
f1 (future (reduce #(assoc! % :a (+ (:a %) %2)) v (range 10)))
f2 (future (reduce #(assoc! % :a (+ (:a %) %2)) v (range 10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
Which seems to consistently produce {:a 90} on both Clojure 1.7.0 Alpha 2
and Clojure 1.7.0 Alpha 3.
Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
Perfection is the enemy of the good.
-- Gustave Flaubert, French realist novelist (1821-1880)
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
Hi there,
On Tuesday, November 4, 2014 8:05:03 AM UTC+1, Daniel Marjenburgh wrote:
I know transients aren't bash-in-place, as they may return new references,
but that is not the problem I'm having here.
If you bash in place, you would get unexpected results, but still the same
result every time. The problem here is that the 'value' of the transient is
changing underneath you because of another thread.
To ensure consistency, you must ensure that you update a transient value at
most once – all calls (including reads) on the same transient value after
an update is illegal/undefined behaviour. Transient updates – minus
persistent! – will always return a new transient value, even though the
transient returned may be the same reference. So the code
(let [v (transient {:a 0})
f1 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))
f2 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
will result in undefined behaviour, because the original v is updated twice
(and is potentially read after an update). The correct way would be to do
the following:
(let [v {:a 0}
f1 (future (reduce #(assoc! % (+ (:a %) %2)) (transient v) (range
10)))
f2 (future (reduce #(assoc! % (+ (:a %) %2)) (transient v) (range
10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
I don't see an explicit rule at http://clojure.org/transients stating this,
although I feel it is implicit from the bash in-place rule. Perhaps that's
something worth adding to the page to avoid confusion.
--
Regards,
Jean Niklas L'orange
Op dinsdag 4 november 2014 06:13:00 UTC+1 schreef Alex Miller:
On Monday, November 3, 2014 10:44:19 PM UTC-6, Atamert Ölçgen wrote:
Thanks Alex!
Now that I took a second look at Daniel's code, it seems assoc! is used
like swap!, as if it would modify m in place. So I would expect, if it runs
without errors, result to be {:a 0}.
Right, that is an incorrect usage - it will actually modify with changes
though, but not in expected ways (this is independent of the change we're
discussing - you can get the same behavior in a single thread modifying a
transient without reusing the return).
Given that transients are values (not reference types like ref or atom) I
can't think of a case where they can be modified concurrently.
You can, but not without going pretty far out of normal Clojure code.
On Tue, Nov 4, 2014 at 11:19 AM, Alex Miller al...@puredanger.com
wrote:
On Monday, November 3, 2014 9:00:10 PM UTC-6, Atamert Ölçgen wrote:
On Mon, Nov 3, 2014 at 5:57 PM, Daniel Marjenburgh
dmarje...@gmail.com wrote:
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even though
it's quite a big change.
With this change the following code is possible:
With persistents the result would be the same, every time. If this is
now valid Clojure, I didn't run it myself, we are sacrificing
consistency.
I don't understand what we're getting in return.
Even the simple example in the ticket (with one future) doesn't make a
lot of sense to me.
Am I missing something obvious?
Transients always expect thread isolation. In the past this was locked
to a single thread (the one that made the transient). That restriction now
extends to being used by multiple threads, but isolation should still be
maintained for proper use.
What we are gaining is the ability to use transient collections in go
blocks with core.async, where the thread being used is just one pool out
of
a thread. For example (just typing this not running it, so excuse any
typos):
(defn drain [ch coll]
(let [t (transient []]
(go-loop
(if-some [v (! ch)]
(do (conj! t v) (recur))
(persistent! t)
This doesn't necessarily work in core.async because each time the !
parks, it may be woken up with a different thread from the pool. The
transient change allows this kind of code to succeed.
(let [m (transient {:a 0})
futs (for [n (range 100)]
(future (assoc! m :a (inc (:a m)]
(mapv deref futs) ; wait until futures are done
(persistent! m))
The results will vary per run, where it used to throw
an IllegalAccessError: Transient used by non-owner thread.
I understand the problems of not being able to have 1 go routine
access a transient, even though it would be safe, but this solution
feels
like it's throwing out the baby with the bathwater. Basically, it's
doing
away with what the following block on clojure.org
http://clojure.org/transients says:
*Transients enforce thread isolation**.* Because each result of a
transient operation shares (mutable) structure with the previous, it
would
be very dangerous if more than one thread were to manipulate a
transient at
once. In order to
Re: About transients no longer being safe in 1.7-alpha2
Hi Jean,
In this case, we are dealing with 1 key-value pair, so there is only one
transient reference all the time. It is not my experience that each update
returns a new transient value (here, each update is identical to the
original, as per the identical? predicate).
I know you shouldn't use transients in the way I illustrated, but use it as
if it always returns a new value. In that scenario, you would not get
inconsistent results, as multiple threads would have to coordinate to pass
the reference around (as happens in core.async) and the birth-thread check
is unnecessary.
Still, the onus is now put on the user to ensure correct use of transients
in a multi-threaded environment, as no warning or error is given anymore.
It has always been the case that Clojure makes it hard to write unsafe
mutable code, you had to go out of your way. With this change, I feel we
are giving up this safeguard a bit too easily.
Maybe another solution could be to have the birth-thread explicitly
relinquish ownership of the transient, for example, by calling
(relinquish-transient m). Then the thread that makes the next transient
update would become the new owner. If another thread updates a transient
while another threads owns it, the usual error is thrown. This way we
maintain the safety as before and it becomes a issue for core.async to deal
with transients in logical threads, not clojure.core. I guess a go macro
could recognize transients in go-blocks and auto-relinguish them before
parking operations so that the user can keep thinking in logical threads.
I'd like to know how feasible a solution like that would be.
Op dinsdag 4 november 2014 11:12:08 UTC+1 schreef Jean Niklas L'orange:
Hi there,
On Tuesday, November 4, 2014 8:05:03 AM UTC+1, Daniel Marjenburgh wrote:
I know transients aren't bash-in-place, as they may return new
references, but that is not the problem I'm having here.
If you bash in place, you would get unexpected results, but still the
same result every time. The problem here is that the 'value' of the
transient is changing underneath you because of another thread.
To ensure consistency, you must ensure that you update a transient value
at most once – all calls (including reads) on the same transient value
after an update is illegal/undefined behaviour. Transient updates – minus
persistent! – will always return a new transient value, even though the
transient returned may be the same reference. So the code
(let [v (transient {:a 0})
f1 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))
f2 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
will result in undefined behaviour, because the original v is updated
twice (and is potentially read after an update). The correct way would be
to do the following:
(let [v {:a 0}
f1 (future (reduce #(assoc! % (+ (:a %) %2)) (transient v) (range
10)))
f2 (future (reduce #(assoc! % (+ (:a %) %2)) (transient v) (range
10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
I don't see an explicit rule at http://clojure.org/transients stating
this, although I feel it is implicit from the bash in-place rule. Perhaps
that's something worth adding to the page to avoid confusion.
--
Regards,
Jean Niklas L'orange
Op dinsdag 4 november 2014 06:13:00 UTC+1 schreef Alex Miller:
On Monday, November 3, 2014 10:44:19 PM UTC-6, Atamert Ölçgen wrote:
Thanks Alex!
Now that I took a second look at Daniel's code, it seems assoc! is used
like swap!, as if it would modify m in place. So I would expect, if it
runs
without errors, result to be {:a 0}.
Right, that is an incorrect usage - it will actually modify with changes
though, but not in expected ways (this is independent of the change we're
discussing - you can get the same behavior in a single thread modifying a
transient without reusing the return).
Given that transients are values (not reference types like ref or atom)
I can't think of a case where they can be modified concurrently.
You can, but not without going pretty far out of normal Clojure code.
On Tue, Nov 4, 2014 at 11:19 AM, Alex Miller al...@puredanger.com
wrote:
On Monday, November 3, 2014 9:00:10 PM UTC-6, Atamert Ölçgen wrote:
On Mon, Nov 3, 2014 at 5:57 PM, Daniel Marjenburgh
dmarje...@gmail.com wrote:
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even
though
it's quite a big change.
With this change the following code is possible:
With persistents the result would be the same, every time. If this is
now valid Clojure, I didn't run it myself, we are sacrificing
consistency.
I don't understand what we're getting in return.
Even the simple example in the ticket (with one future) doesn't make
a lot of
Re: About transients no longer being safe in 1.7-alpha2
Hi Daniel,
On Tuesday, November 4, 2014 11:54:30 AM UTC+1, Daniel Marjenburgh wrote:
Hi Jean,
In this case, we are dealing with 1 key-value pair, so there is only one
transient reference all the time. It is not my experience that each update
returns a new transient value (here, each update is identical to the
original, as per the identical? predicate).
I probably worded myself poorly: When I wrote transient value, I meant the
contents of a transient, not the reference to it.
I like to think of transients like this: They work exactly like persistent
data structures, but when any kind of update is performed on the transient,
the runtime should be allowed to deallocate the old transient head if it
wants to. If I use transients correctly, I should never see any segfaults
(or whatever you want to call it) due to transient usage.
Now, the runtime doesn't actually deallocate the transient head, it
(usually) reuses it for performance. Which means that the old transient
references are equal to the new transient references, even though the old
transient contents are overwritten (and therefore invalidated). So even
though you can use the old transient reference to refer to the new
transient value, it's not referring to the old transient value.
If it helps you, you can also think of transients as linear types, like the
ones in Rust. Their constraints are very similar, but correct transient
usage have to be verified by us, not the compiler. Now, whether that's a
good idea or not is another matter.
I know you shouldn't use transients in the way I illustrated, but use it
as if it always returns a new value. In that scenario, you would not get
inconsistent results, as multiple threads would have to coordinate to pass
the reference around (as happens in core.async) and the birth-thread check
is unnecessary.
Still, the onus is now put on the user to ensure correct use of transients
in a multi-threaded environment, as no warning or error is given anymore.
It has always been the case that Clojure makes it hard to write unsafe
mutable code, you had to go out of your way. With this change, I feel we
are giving up this safeguard a bit too easily.
Maybe another solution could be to have the birth-thread explicitly
relinquish ownership of the transient, for example, by calling
(relinquish-transient m). Then the thread that makes the next transient
update would become the new owner. If another thread updates a transient
while another threads owns it, the usual error is thrown. This way we
maintain the safety as before and it becomes a issue for core.async to deal
with transients in logical threads, not clojure.core. I guess a go macro
could recognize transients in go-blocks and auto-relinguish them before
parking operations so that the user can keep thinking in logical threads.
I'd like to know how feasible a solution like that would be.
Op dinsdag 4 november 2014 11:12:08 UTC+1 schreef Jean Niklas L'orange:
Hi there,
On Tuesday, November 4, 2014 8:05:03 AM UTC+1, Daniel Marjenburgh wrote:
I know transients aren't bash-in-place, as they may return new
references, but that is not the problem I'm having here.
If you bash in place, you would get unexpected results, but still the
same result every time. The problem here is that the 'value' of the
transient is changing underneath you because of another thread.
To ensure consistency, you must ensure that you update a transient value
at most once – all calls (including reads) on the same transient value
after an update is illegal/undefined behaviour. Transient updates – minus
persistent! – will always return a new transient value, even though the
transient returned may be the same reference. So the code
(let [v (transient {:a 0})
f1 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))
f2 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
will result in undefined behaviour, because the original v is updated
twice (and is potentially read after an update). The correct way would be
to do the following:
(let [v {:a 0}
f1 (future (reduce #(assoc! % (+ (:a %) %2)) (transient v) (range
10)))
f2 (future (reduce #(assoc! % (+ (:a %) %2)) (transient v) (range
10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
I don't see an explicit rule at http://clojure.org/transients stating
this, although I feel it is implicit from the bash in-place rule. Perhaps
that's something worth adding to the page to avoid confusion.
--
Regards,
Jean Niklas L'orange
Op dinsdag 4 november 2014 06:13:00 UTC+1 schreef Alex Miller:
On Monday, November 3, 2014 10:44:19 PM UTC-6, Atamert Ölçgen wrote:
Thanks Alex!
Now that I took a second look at Daniel's code, it seems assoc! is
used like swap!, as if it would modify m in place. So I would expect, if
it
runs without errors, result to be
Re: About transients no longer being safe in 1.7-alpha2
I think all of these examples violate the usage expectations of transients (regardless of release) and you should expect to get confusing results from them. I do not think it's worth modifying core.async go blocks to do anything special in this regard (eince go is a macro I don't think you can detect at compile time which locals are transients anyways). One possibility I've kicked around with some people is deciding at the time you create the transient whether it is allowed to cross threads (through a flag or something). When you don't need it you could then retain the existing safety check. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
Hi Alex, I understand any concurrent use of transients violates the usage expectations. Of course, if you treat any piece of mutable state sequentially like transients, you would not run into trouble either. My concern is purely about the safety that is taken away when transients 'are' used wrongly. All the examples would've throw exceptions before 1.7-alpha2, and they don't do so anymore. Having a single go-block access a transient is valid. Having 2 go-blocks access the same transient is not and there is no warning. If at a certain point you want to 'scale' up from 1 go-block to 2 and don't realise it was accessing transients, you are potentially in a lot of trouble and there is nothing to warn you. FWIW, a flag on creating transients to make the dangers explicit would have my vote. It would also be in line with being explicit about ^:volatile-mutable and have the default be safe. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
Hi Daniel, On Wed, Nov 5, 2014 at 3:06 PM, Daniel Marjenburgh dmarjenbu...@gmail.com wrote: My concern is purely about the safety that is taken away when transients 'are' used wrongly. All the examples would've throw exceptions before 1.7-alpha2, and they don't do so anymore. But they would run fine, if you used delay's instead of future's. Same wrong usage, single threaded. So the issue is not really this new change. (See my code with delay's earlier) -- Kind Regards, Atamert Ölçgen -+- --+ +++ www.muhuk.com -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
About transients no longer being safe in 1.7-alpha2
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even though
it's quite a big change.
With this change the following code is possible:
(let [m (transient {:a 0})
futs (for [n (range 100)]
(future (assoc! m :a (inc (:a m)]
(mapv deref futs) ; wait until futures are done
(persistent! m))
The results will vary per run, where it used to throw
an IllegalAccessError: Transient used by non-owner thread.
I understand the problems of not being able to have 1 go routine access a
transient, even though it would be safe, but this solution feels like it's
throwing out the baby with the bathwater. Basically, it's doing away with
what the following block on clojure.org http://clojure.org/transients
says:
*Transients enforce thread isolation**.* Because each result of a transient
operation shares (mutable) structure with the previous, it would be very
dangerous if more than one thread were to manipulate a transient at once.
In order to prevent this, transients will detect any (read or write) use
from a thread other than the one that created them and throw an exception.
This may not sound like a concurrency story, but single-thread isolation
is actually a very useful concurrency semantic. The whole point of using
transients is that doing so is safe, because their use is an isolated
implementation detail of otherwise functional code. Having that be enforced
means that some things that would normally be very hard to make safe with
ordinary mutable data structures become easy.
I don't have a good solution for dealing with transients and logical
threads, but I would much prefer keeping the semantics of transients as
they are and maybe pass an option to transient to disable owner checking.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
On Mon, Nov 3, 2014 at 5:57 PM, Daniel Marjenburgh dmarjenbu...@gmail.com
wrote:
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even though
it's quite a big change.
With this change the following code is possible:
With persistents the result would be the same, every time. If this is now
valid Clojure, I didn't run it myself, we are sacrificing consistency. I
don't understand what we're getting in return.
Even the simple example in the ticket (with one future) doesn't make a lot
of sense to me.
Am I missing something obvious?
(let [m (transient {:a 0})
futs (for [n (range 100)]
(future (assoc! m :a (inc (:a m)]
(mapv deref futs) ; wait until futures are done
(persistent! m))
The results will vary per run, where it used to throw
an IllegalAccessError: Transient used by non-owner thread.
I understand the problems of not being able to have 1 go routine access a
transient, even though it would be safe, but this solution feels like it's
throwing out the baby with the bathwater. Basically, it's doing away with
what the following block on clojure.org http://clojure.org/transients
says:
*Transients enforce thread isolation**.* Because each result of a
transient operation shares (mutable) structure with the previous, it would
be very dangerous if more than one thread were to manipulate a transient at
once. In order to prevent this, transients will detect any (read or write)
use from a thread other than the one that created them and throw an
exception.
This may not sound like a concurrency story, but single-thread isolation
is actually a very useful concurrency semantic. The whole point of using
transients is that doing so is safe, because their use is an isolated
implementation detail of otherwise functional code. Having that be enforced
means that some things that would normally be very hard to make safe with
ordinary mutable data structures become easy.
I don't have a good solution for dealing with transients and logical
threads, but I would much prefer keeping the semantics of transients as
they are and maybe pass an option to transient to disable owner checking.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an
email to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Kind Regards,
Atamert Ölçgen
-+-
--+
+++
www.muhuk.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
On Monday, November 3, 2014 9:00:10 PM UTC-6, Atamert Ölçgen wrote:
On Mon, Nov 3, 2014 at 5:57 PM, Daniel Marjenburgh dmarje...@gmail.com
javascript: wrote:
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even though
it's quite a big change.
With this change the following code is possible:
With persistents the result would be the same, every time. If this is now
valid Clojure, I didn't run it myself, we are sacrificing consistency. I
don't understand what we're getting in return.
Even the simple example in the ticket (with one future) doesn't make a lot
of sense to me.
Am I missing something obvious?
Transients always expect thread isolation. In the past this was locked to a
single thread (the one that made the transient). That restriction now
extends to being used by multiple threads, but isolation should still be
maintained for proper use.
What we are gaining is the ability to use transient collections in go
blocks with core.async, where the thread being used is just one pool out of
a thread. For example (just typing this not running it, so excuse any
typos):
(defn drain [ch coll]
(let [t (transient []]
(go-loop
(if-some [v (! ch)]
(do (conj! t v) (recur))
(persistent! t)
This doesn't necessarily work in core.async because each time the ! parks,
it may be woken up with a different thread from the pool. The transient
change allows this kind of code to succeed.
(let [m (transient {:a 0})
futs (for [n (range 100)]
(future (assoc! m :a (inc (:a m)]
(mapv deref futs) ; wait until futures are done
(persistent! m))
The results will vary per run, where it used to throw
an IllegalAccessError: Transient used by non-owner thread.
I understand the problems of not being able to have 1 go routine access a
transient, even though it would be safe, but this solution feels like it's
throwing out the baby with the bathwater. Basically, it's doing away with
what the following block on clojure.org http://clojure.org/transients
says:
*Transients enforce thread isolation**.* Because each result of a
transient operation shares (mutable) structure with the previous, it would
be very dangerous if more than one thread were to manipulate a transient at
once. In order to prevent this, transients will detect any (read or write)
use from a thread other than the one that created them and throw an
exception.
This may not sound like a concurrency story, but single-thread isolation
is actually a very useful concurrency semantic. The whole point of using
transients is that doing so is safe, because their use is an isolated
implementation detail of otherwise functional code. Having that be enforced
means that some things that would normally be very hard to make safe with
ordinary mutable data structures become easy.
I don't have a good solution for dealing with transients and logical
threads, but I would much prefer keeping the semantics of transients as
they are and maybe pass an option to transient to disable owner checking.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.com
javascript:
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com javascript:
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an
email to clojure+u...@googlegroups.com javascript:.
For more options, visit https://groups.google.com/d/optout.
--
Kind Regards,
Atamert Ölçgen
-+-
--+
+++
www.muhuk.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
Thanks Alex!
Now that I took a second look at Daniel's code, it seems assoc! is used
like swap!, as if it would modify m in place. So I would expect, if it runs
without errors, result to be {:a 0}.
Given that transients are values (not reference types like ref or atom) I
can't think of a case where they can be modified concurrently.
On Tue, Nov 4, 2014 at 11:19 AM, Alex Miller a...@puredanger.com wrote:
On Monday, November 3, 2014 9:00:10 PM UTC-6, Atamert Ölçgen wrote:
On Mon, Nov 3, 2014 at 5:57 PM, Daniel Marjenburgh dmarje...@gmail.com
wrote:
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even though
it's quite a big change.
With this change the following code is possible:
With persistents the result would be the same, every time. If this is now
valid Clojure, I didn't run it myself, we are sacrificing consistency. I
don't understand what we're getting in return.
Even the simple example in the ticket (with one future) doesn't make a
lot of sense to me.
Am I missing something obvious?
Transients always expect thread isolation. In the past this was locked to
a single thread (the one that made the transient). That restriction now
extends to being used by multiple threads, but isolation should still be
maintained for proper use.
What we are gaining is the ability to use transient collections in go
blocks with core.async, where the thread being used is just one pool out of
a thread. For example (just typing this not running it, so excuse any
typos):
(defn drain [ch coll]
(let [t (transient []]
(go-loop
(if-some [v (! ch)]
(do (conj! t v) (recur))
(persistent! t)
This doesn't necessarily work in core.async because each time the !
parks, it may be woken up with a different thread from the pool. The
transient change allows this kind of code to succeed.
(let [m (transient {:a 0})
futs (for [n (range 100)]
(future (assoc! m :a (inc (:a m)]
(mapv deref futs) ; wait until futures are done
(persistent! m))
The results will vary per run, where it used to throw
an IllegalAccessError: Transient used by non-owner thread.
I understand the problems of not being able to have 1 go routine access
a transient, even though it would be safe, but this solution feels like
it's throwing out the baby with the bathwater. Basically, it's doing away
with what the following block on clojure.org
http://clojure.org/transients says:
*Transients enforce thread isolation**.* Because each result of a
transient operation shares (mutable) structure with the previous, it would
be very dangerous if more than one thread were to manipulate a transient at
once. In order to prevent this, transients will detect any (read or write)
use from a thread other than the one that created them and throw an
exception.
This may not sound like a concurrency story, but single-thread
isolation is actually a very useful concurrency semantic. The whole point
of using transients is that doing so is safe, because their use is an
isolated implementation detail of otherwise functional code. Having that be
enforced means that some things that would normally be very hard to make
safe with ordinary mutable data structures become easy.
I don't have a good solution for dealing with transients and logical
threads, but I would much prefer keeping the semantics of transients as
they are and maybe pass an option to transient to disable owner checking.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google
Groups Clojure group.
To unsubscribe from this group and stop receiving emails from it, send
an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Kind Regards,
Atamert Ölçgen
-+-
--+
+++
www.muhuk.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an
email to
Re: About transients no longer being safe in 1.7-alpha2
On Monday, November 3, 2014 10:44:19 PM UTC-6, Atamert Ölçgen wrote:
Thanks Alex!
Now that I took a second look at Daniel's code, it seems assoc! is used
like swap!, as if it would modify m in place. So I would expect, if it runs
without errors, result to be {:a 0}.
Right, that is an incorrect usage - it will actually modify with changes
though, but not in expected ways (this is independent of the change we're
discussing - you can get the same behavior in a single thread modifying a
transient without reusing the return).
Given that transients are values (not reference types like ref or atom) I
can't think of a case where they can be modified concurrently.
You can, but not without going pretty far out of normal Clojure code.
On Tue, Nov 4, 2014 at 11:19 AM, Alex Miller al...@puredanger.com
javascript: wrote:
On Monday, November 3, 2014 9:00:10 PM UTC-6, Atamert Ölçgen wrote:
On Mon, Nov 3, 2014 at 5:57 PM, Daniel Marjenburgh dmarje...@gmail.com
wrote:
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even though
it's quite a big change.
With this change the following code is possible:
With persistents the result would be the same, every time. If this is
now valid Clojure, I didn't run it myself, we are sacrificing consistency.
I don't understand what we're getting in return.
Even the simple example in the ticket (with one future) doesn't make a
lot of sense to me.
Am I missing something obvious?
Transients always expect thread isolation. In the past this was locked to
a single thread (the one that made the transient). That restriction now
extends to being used by multiple threads, but isolation should still be
maintained for proper use.
What we are gaining is the ability to use transient collections in go
blocks with core.async, where the thread being used is just one pool out of
a thread. For example (just typing this not running it, so excuse any
typos):
(defn drain [ch coll]
(let [t (transient []]
(go-loop
(if-some [v (! ch)]
(do (conj! t v) (recur))
(persistent! t)
This doesn't necessarily work in core.async because each time the !
parks, it may be woken up with a different thread from the pool. The
transient change allows this kind of code to succeed.
(let [m (transient {:a 0})
futs (for [n (range 100)]
(future (assoc! m :a (inc (:a m)]
(mapv deref futs) ; wait until futures are done
(persistent! m))
The results will vary per run, where it used to throw
an IllegalAccessError: Transient used by non-owner thread.
I understand the problems of not being able to have 1 go routine access
a transient, even though it would be safe, but this solution feels like
it's throwing out the baby with the bathwater. Basically, it's doing away
with what the following block on clojure.org
http://clojure.org/transients says:
*Transients enforce thread isolation**.* Because each result of a
transient operation shares (mutable) structure with the previous, it
would
be very dangerous if more than one thread were to manipulate a transient
at
once. In order to prevent this, transients will detect any (read or
write)
use from a thread other than the one that created them and throw an
exception.
This may not sound like a concurrency story, but single-thread
isolation is actually a very useful concurrency semantic. The whole point
of using transients is that doing so is safe, because their use is an
isolated implementation detail of otherwise functional code. Having that
be
enforced means that some things that would normally be very hard to make
safe with ordinary mutable data structures become easy.
I don't have a good solution for dealing with transients and logical
threads, but I would much prefer keeping the semantics of transients as
they are and maybe pass an option to transient to disable owner checking.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google
Groups Clojure group.
To unsubscribe from this group and stop receiving emails from it, send
an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Kind Regards,
Atamert Ölçgen
-+-
--+
+++
www.muhuk.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email
Re: About transients no longer being safe in 1.7-alpha2
Hi Alex.
I know transients aren't bash-in-place, as they may return new references,
but that is not the problem I'm having here.
If you bash in place, you would get unexpected results, but still the same
result every time. The problem here is that the 'value' of the transient is
changing underneath you because of another thread.
Let me change the example a bit:
(let [v (transient {:a 0})
f1 (future (reduce #(assoc! % (+ (:a %) %2
f2 (future (reduce #(assoc! % (+ (:a %) %2]
@f1 @f2 ; wait for futures
(persistent! f1))
This will also give varying results on each run. If you were to add a
(println (:a %)) in the reducing fn of f1 you would see the results are not
the same each time.
Op dinsdag 4 november 2014 06:13:00 UTC+1 schreef Alex Miller:
On Monday, November 3, 2014 10:44:19 PM UTC-6, Atamert Ölçgen wrote:
Thanks Alex!
Now that I took a second look at Daniel's code, it seems assoc! is used
like swap!, as if it would modify m in place. So I would expect, if it runs
without errors, result to be {:a 0}.
Right, that is an incorrect usage - it will actually modify with changes
though, but not in expected ways (this is independent of the change we're
discussing - you can get the same behavior in a single thread modifying a
transient without reusing the return).
Given that transients are values (not reference types like ref or atom) I
can't think of a case where they can be modified concurrently.
You can, but not without going pretty far out of normal Clojure code.
On Tue, Nov 4, 2014 at 11:19 AM, Alex Miller al...@puredanger.com
wrote:
On Monday, November 3, 2014 9:00:10 PM UTC-6, Atamert Ölçgen wrote:
On Mon, Nov 3, 2014 at 5:57 PM, Daniel Marjenburgh dmarje...@gmail.com
wrote:
Hi,
I just want to address this issue (CLJ-1498
http://dev.clojure.org/jira/browse/CLJ-1498). It was accepted in
1.7-alpha2 and I haven't seen a lot of discussion around it, even though
it's quite a big change.
With this change the following code is possible:
With persistents the result would be the same, every time. If this is
now valid Clojure, I didn't run it myself, we are sacrificing consistency.
I don't understand what we're getting in return.
Even the simple example in the ticket (with one future) doesn't make a
lot of sense to me.
Am I missing something obvious?
Transients always expect thread isolation. In the past this was locked
to a single thread (the one that made the transient). That restriction now
extends to being used by multiple threads, but isolation should still be
maintained for proper use.
What we are gaining is the ability to use transient collections in go
blocks with core.async, where the thread being used is just one pool out of
a thread. For example (just typing this not running it, so excuse any
typos):
(defn drain [ch coll]
(let [t (transient []]
(go-loop
(if-some [v (! ch)]
(do (conj! t v) (recur))
(persistent! t)
This doesn't necessarily work in core.async because each time the !
parks, it may be woken up with a different thread from the pool. The
transient change allows this kind of code to succeed.
(let [m (transient {:a 0})
futs (for [n (range 100)]
(future (assoc! m :a (inc (:a m)]
(mapv deref futs) ; wait until futures are done
(persistent! m))
The results will vary per run, where it used to throw
an IllegalAccessError: Transient used by non-owner thread.
I understand the problems of not being able to have 1 go routine
access a transient, even though it would be safe, but this solution feels
like it's throwing out the baby with the bathwater. Basically, it's doing
away with what the following block on clojure.org
http://clojure.org/transients says:
*Transients enforce thread isolation**.* Because each result of a
transient operation shares (mutable) structure with the previous, it
would
be very dangerous if more than one thread were to manipulate a transient
at
once. In order to prevent this, transients will detect any (read or
write)
use from a thread other than the one that created them and throw an
exception.
This may not sound like a concurrency story, but single-thread
isolation is actually a very useful concurrency semantic. The whole
point
of using transients is that doing so is safe, because their use is an
isolated implementation detail of otherwise functional code. Having that
be
enforced means that some things that would normally be very hard to make
safe with ordinary mutable data structures become easy.
I don't have a good solution for dealing with transients and logical
threads, but I would much prefer keeping the semantics of transients as
they are and maybe pass an option to transient to disable owner checking.
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to
Re: About transients no longer being safe in 1.7-alpha2
I was a bit too quick there and posted some errors int he code:
(let [v (transient {:a 0})
f1 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))
f2 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
FWIW I can confirm the following code produces {:a 90} consistently:
(let [v (transient {:a 0})
f1 (delay (reduce #(assoc! % :a (+ (:a %) %2)) v (range 10)))
f2 (delay (reduce #(assoc! % :a (+ (:a %) %2)) v (range 10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
Note that there are no future calls here. Clojure 1.5.1
Part of me wants the result to be {:a 45}
On Tue, Nov 4, 2014 at 3:13 PM, Daniel Marjenburgh dmarjenbu...@gmail.com
wrote:
I was a bit too quick there and posted some errors int he code:
(let [v (transient {:a 0})
f1 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))
f2 (future (reduce #(assoc! % (+ (:a %) %2)) v (range 10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with
your first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an
email to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Kind Regards,
Atamert Ölçgen
-+-
--+
+++
www.muhuk.com
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: About transients no longer being safe in 1.7-alpha2
On Nov 3, 2014, at 11:13 PM, Daniel Marjenburgh dmarjenbu...@gmail.com wrote:
I was a bit too quick there and posted some errors int he code:
That’s still not quite right. I think you mean:
(let [v (transient {:a 0})
f1 (future (reduce #(assoc! % :a (+ (:a %) %2)) v (range 10)))
f2 (future (reduce #(assoc! % :a (+ (:a %) %2)) v (range 10)))]
@f1 @f2 ; wait for futures
(persistent! @f1))
Which seems to consistently produce {:a 90} on both Clojure 1.7.0 Alpha 2 and
Clojure 1.7.0 Alpha 3.
Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
Perfection is the enemy of the good.
-- Gustave Flaubert, French realist novelist (1821-1880)
--
You received this message because you are subscribed to the Google
Groups Clojure group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
Clojure group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
