Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Marko Rauhamaa
Chris Angelico : > On Mon, Jul 9, 2018 at 5:18 AM, Marko Rauhamaa wrote: >> Chris Angelico : >>> Are you assuming that Python's semantics are defined by the semantics >>> of one possible implementation language? >> >> What are Python's semantics defined by? I've been using these: >> >>https:/

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Chris Angelico
On Mon, Jul 9, 2018 at 5:18 AM, Marko Rauhamaa wrote: > Chris Angelico : >> Are you assuming that Python's semantics are defined by the semantics >> of one possible implementation language? > > What are Python's semantics defined by? I've been using these: > >https://docs.python.org/3/referenc

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Marko Rauhamaa
Chris Angelico : > Are you assuming that Python's semantics are defined by the semantics > of one possible implementation language? What are Python's semantics defined by? I've been using these: https://docs.python.org/3/reference/> https://docs.python.org/3/library/> Unfortunately, neith

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Chris Angelico
On Mon, Jul 9, 2018 at 4:57 AM, Marko Rauhamaa wrote: > Chris Angelico : > >> On Mon, Jul 9, 2018 at 2:11 AM, Marko Rauhamaa wrote: >>> MRAB : In C you'd declare 'quit' as 'volatile' to tell the compiler that it could change unexpectedly, so don't make that assumption. >>> >>> C is an e

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Marko Rauhamaa
Chris Angelico : > On Mon, Jul 9, 2018 at 2:11 AM, Marko Rauhamaa wrote: >> MRAB : >>> In C you'd declare 'quit' as 'volatile' to tell the compiler that it >>> could change unexpectedly, so don't make that assumption. >> >> C is an even tougher case. Even if the compiler kept on checking a >> vol

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Steven D'Aprano
On Sun, 08 Jul 2018 16:37:11 +0100, MRAB wrote: > On 2018-07-08 14:38, Steven D'Aprano wrote: >> On Sun, 08 Jul 2018 14:11:58 +0300, Marko Rauhamaa wrote: >> > [snip] >>> More importantly, this loop may never finish: >>> >>> # Initially >>> quit = False >>> >>> # Thread 1 >>> gl

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Steven D'Aprano
On Sun, 08 Jul 2018 19:35:55 +0300, Marko Rauhamaa wrote: > Steven D'Aprano : >> On Sun, 08 Jul 2018 14:11:58 +0300, Marko Rauhamaa wrote: >>> PS My example with "impossible" being the result of a racy integer >>> operation is of course unlikely but could be the outcome if the Python >>> runtime r

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Chris Angelico
On Mon, Jul 9, 2018 at 2:11 AM, Marko Rauhamaa wrote: > MRAB : >> On 2018-07-08 14:38, Steven D'Aprano wrote: >>> On Sun, 08 Jul 2018 14:11:58 +0300, Marko Rauhamaa wrote: >>> >> [snip] More importantly, this loop may never finish: # Initially quit = False

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Marko Rauhamaa
Steven D'Aprano : > On Sun, 08 Jul 2018 14:11:58 +0300, Marko Rauhamaa wrote: >> PS My example with "impossible" being the result of a racy integer >> operation is of course unlikely but could be the outcome if the Python >> runtime reorganized its object cache on the fly (in a hypothetical >> impl

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Marko Rauhamaa
MRAB : > On 2018-07-08 14:38, Steven D'Aprano wrote: >> On Sun, 08 Jul 2018 14:11:58 +0300, Marko Rauhamaa wrote: >> > [snip] >>> More importantly, this loop may never finish: >>> >>> # Initially >>> quit = False >>> >>> # Thread 1 >>> global quit >>> while not quit: >>>

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread MRAB
On 2018-07-08 14:38, Steven D'Aprano wrote: On Sun, 08 Jul 2018 14:11:58 +0300, Marko Rauhamaa wrote: [snip] More importantly, this loop may never finish: # Initially quit = False # Thread 1 global quit while not quit: time.sleep(1) # Thread 2 global quit

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Steven D'Aprano
On Sun, 08 Jul 2018 14:11:58 +0300, Marko Rauhamaa wrote: > Steven D'Aprano : >> Changing implementations from one which is thread safe to one which is >> not can break people's code, and shouldn't be done on a whim. >> Especially since such breakage could be subtle, hard to notice, harder >> to t

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Marko Rauhamaa
Steven D'Aprano : > Changing implementations from one which is thread safe to one which is > not can break people's code, and shouldn't be done on a whim. > Especially since such breakage could be subtle, hard to notice, harder > to track down, and even harder still to fix. Java's HotSpot does it

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Steven D'Aprano
On Sun, 08 Jul 2018 10:52:15 +0300, Marko Rauhamaa wrote: > You are on the right track, Chris, but you are still deducing behavior > from a particular implementation. For example Java's definition is > approximately the one I give above: > >Without correct synchronization, very strange, confu

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-08 Thread Marko Rauhamaa
Chris Angelico : > On Sun, Jul 8, 2018 at 11:04 AM, Steven D'Aprano > wrote: >>> The only thing Python should guarantee is that the data structures stay >>> "coherent" under race conditions. In other words, there cannot be a >>> segmentation fault. For example, if two threads executed this code i

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Chris Angelico
On Sun, Jul 8, 2018 at 1:58 PM, Steven D'Aprano wrote: > On Sun, 08 Jul 2018 12:23:41 +1000, Chris Angelico wrote: > >>> Some people, when confronted with a problem, say, "I know, I'll use >>> threads". Nothhtwo probw ey ave lems. >> >> Right. Now they have to deal with interleaving, but that's al

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Steven D'Aprano
On Sun, 08 Jul 2018 12:23:41 +1000, Chris Angelico wrote: >> Some people, when confronted with a problem, say, "I know, I'll use >> threads". Nothhtwo probw ey ave lems. > > Right. Now they have to deal with interleaving, but that's all. And > honestly, MOST CODE wouldn't notice interleaving; it'

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Chris Angelico
On Sun, Jul 8, 2018 at 12:12 PM, Steven D'Aprano wrote: > On Sun, 08 Jul 2018 11:15:17 +1000, Chris Angelico wrote: > > [...] >> Python threads don't switch only between lines of code, > > As I understand it, there could be a switch between any two byte codes, > or maybe only between certain bytes

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Steven D'Aprano
On Sun, 08 Jul 2018 11:15:17 +1000, Chris Angelico wrote: [...] > Python threads don't switch only between lines of code, As I understand it, there could be a switch between any two byte codes, or maybe only between certain bytes codes. But certain more fine grained than just between lines of

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Chris Angelico
On Sun, Jul 8, 2018 at 11:04 AM, Steven D'Aprano wrote: >> The only thing Python should guarantee is that the data structures stay >> "coherent" under race conditions. In other words, there cannot be a >> segmentation fault. For example, if two threads executed this code in >> parallel: >> >>

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Steven D'Aprano
On Sun, 08 Jul 2018 00:00:26 +0300, Marko Rauhamaa wrote: > Ian Kelly : >> the leaning of the devs seems to be to refrain from documenting it and >> instead document that *no* operations are guaranteed atomic. > > I believe that to be wise. Otherwise, Python would limit its future > implementatio

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Marko Rauhamaa
Ian Kelly : > the leaning of the devs seems to be to refrain from documenting it and > instead document that *no* operations are guaranteed atomic. I believe that to be wise. Otherwise, Python would limit its future implementation options. The only thing Python should guarantee is that the data s

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Ethan Furman
On 07/06/2018 10:51 AM, INADA Naoki wrote: > On Sat, Jul 7, 2018 at 2:49 AM Steven D'Aprano wrote: >> I have a dict with string keys: >> >> --> D = {'a': None, 'b': None} >> How do I do a thread-safe insertion if, and only if, the key isn't >> already there? > D.setdefault('c', None) This is

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Ian Kelly
On Fri, Jul 6, 2018 at 11:56 AM INADA Naoki wrote: > > D.setdefault('c', None) Not guaranteed to be atomic. I think the only safe way to do it is with a Lock. -- https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Devin Jeanpierre
On Sat, Jul 7, 2018 at 6:49 AM Marko Rauhamaa wrote: > Is that guaranteed to be thread-safe? The documentation ( s://docs.python.org/3/library/stdtypes.html#dict.setdefault>) makes no > such promise. It's guaranteed to be thread-safe because all of Python's core containers are thread safe (in as

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Ian Kelly
On Sat, Jul 7, 2018 at 8:03 AM Stefan Behnel wrote: > > Marko Rauhamaa schrieb am 07.07.2018 um 15:41: > > Steven D'Aprano : > >> On Sat, 07 Jul 2018 02:51:41 +0900, INADA Naoki wrote: > >>> D.setdefault('c', None) > >> > >> Oh that's clever! > > > > Is that guaranteed to be thread-safe? The docum

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Stefan Behnel
Marko Rauhamaa schrieb am 07.07.2018 um 15:41: > Steven D'Aprano : >> On Sat, 07 Jul 2018 02:51:41 +0900, INADA Naoki wrote: >>> D.setdefault('c', None) >> >> Oh that's clever! > > Is that guaranteed to be thread-safe? The documentation ( s://docs.python.org/3/library/stdtypes.html#dict.setdefault

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-07 Thread Marko Rauhamaa
Steven D'Aprano : > On Sat, 07 Jul 2018 02:51:41 +0900, INADA Naoki wrote: >> D.setdefault('c', None) > > Oh that's clever! Is that guaranteed to be thread-safe? The documentation () makes no such promise. At least __collectios_abc.py contains this method definition for MutableMapping: def s

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-06 Thread Steven D'Aprano
On Sat, 07 Jul 2018 02:51:41 +0900, INADA Naoki wrote: > D.setdefault('c', None) Oh that's clever! Thanks! -- Steven D'Aprano "Ever since I learned about confirmation bias, I've been seeing it everywhere." -- Jon Ronson -- https://mail.python.org/mailman/listinfo/python-list

Re: Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-06 Thread INADA Naoki
D.setdefault('c', None) On Sat, Jul 7, 2018 at 2:49 AM Steven D'Aprano wrote: > > I have a dict with string keys: > > D = {'a': None, 'b': None} > > (the values don't matter for this question) and I want to add a key but > only if it isn't already there. If I do the obvious: > > if not 'c' in D: >

Thread-safe way to add a key to a dict only if it isn't already there?

2018-07-06 Thread Steven D'Aprano
I have a dict with string keys: D = {'a': None, 'b': None} (the values don't matter for this question) and I want to add a key but only if it isn't already there. If I do the obvious: if not 'c' in D: D['c'] = None there's a Time Of Check to Time Of Use bug where some other thread could c