[issue16998] Lost updates with multiprocessing.Value

[Richard Oudkerk]

Changes by Richard Oudkerk :


--
resolution:  -> fixed
stage:  -> committed/rejected
status: open -> closed
type: behavior -> 

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16998] Lost updates with multiprocessing.Value

[Roundup Robot]

Roundup Robot added the comment:

New changeset 7aabbe919f55 by Richard Oudkerk in branch '2.7':
Issue 16998: Clarify that += on a shared value is not atomic.
http://hg.python.org/cpython/rev/7aabbe919f55

New changeset 11cafbe6519f by Richard Oudkerk in branch '3.3':
Issue 16998: Clarify that += on a shared value is not atomic.
http://hg.python.org/cpython/rev/11cafbe6519f

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16998] Lost updates with multiprocessing.Value

[Jens Lechtenboerger]

Jens Lechtenboerger added the comment:

> It only says that accesses are synchronized.  The problem is that you were 
> assuming that "+=" involves a single access -- but that is not how python 
> works.

Yes, I understand that by now (actually since your first comment).

> Where in the examples is there "non-process-safe" access?  (Note that waiting 
> for the only process which modifies a value to terminate using join() will 
> prevent races.)

In section "The multiprocessing.sharedctypes module" the assignments in the 
first example (function modify()) are unsafe.  None of them is protected by a 
lock as suggested in your first comment.  Strictly speaking, no lock is 
necessary in the example as there are no race conditions (the processes work in 
an alternating fashion without concurrency).

I certainly did not see that the example (for a *shared* memory data structure) 
is based on the implicit assumption of a single writer.  In contrast, I assumed 
that some "magic" should offer process-safe usage of "+=", which made me file 
this bug.  That assumption has turned out to be wrong.  To prevent others from 
being mislead in the same way I suggest to either protect those operations with 
locks (and comment on their effect) or to state the implicit assumption 
explicitly.

Maybe add the following after "Below is an example where a number of ctypes 
objects are modified by a child process:"
Note that assignments such n.value **= 2 are not executed atomically but 
involve two operations, a load followed by a store.  Each of these operations 
is protected by the Value's lock, which is dropped in between.  Thus, in 
scenarios with concurrent modifying processes you may want to explicitly 
acquire the Value's lock to ensure atomic execution (avoiding race conditions 
and lost updates), e.g.:
with n.get_lock():
n.value **= 2

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16998] Lost updates with multiprocessing.Value

[Richard Oudkerk]

Richard Oudkerk added the comment:

> I see.  Then this is a documentation bug.  The examples in the 
> documentation use such non-thread-safe assignments (combined with the 
> statement "These shared objects will be process and thread safe.").

Are you talking about this documentation:

  If lock is True (the default) then a new lock object is created to 
  synchronize access to the value. If lock is a Lock or RLock object then 
  that will be used to synchronize access to the value. If lock is False 
  then access to the returned object will not be automatically protected 
  by a lock, so it will not necessarily be “process-safe”.

It only says that accesses are synchronized.  The problem is that you were 
assuming that "+=" involves a single access -- but that is not how python works.

Where in the examples is there "non-process-safe" access?  (Note that waiting 
for the only process which modifies a value to terminate using join() will 
prevent races.)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16998] Lost updates with multiprocessing.Value

[Jens Lechtenboerger]

Jens Lechtenboerger added the comment:

> Loads and stores are both atomic.  But "+=" is made up of two operations, a 
> load followed by a store, and the lock is dropped between the two.

I see.  Then this is a documentation bug.  The examples in the documentation 
use such non-thread-safe assignments (combined with the statement "These shared 
objects will be process and thread safe.").

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16998] Lost updates with multiprocessing.Value

[Richard Oudkerk]

Richard Oudkerk added the comment:

> I thought that access to the value field of Value instances was 
> protected by locks to avoid lost updates.

Loads and stores are both atomic.  But "+=" is made up of two operations, a 
load followed by a store, and the lock is dropped between the two.

The same lack of atomicity applies when using "+=" to modify an attribute of a 
normal python object in a multithreaded program.

If you want an atomic increment you could try

def do_inc(integer):
with integer.get_lock():
integer.value += 1

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16998] Lost updates with multiprocessing.Value

[Antoine Pitrou]

Changes by Antoine Pitrou :


--
nosy: +sbt

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue16998] Lost updates with multiprocessing.Value

[Jens Lechtenboerger]

New submission from Jens Lechtenboerger:

Maybe I'm misreading the documentation of multiprocessing.Value and
multiprocessing.sharedctypes.Value.
I thought that access to the value field of Value instances was protected by
locks to avoid lost updates.

Specifically, for multiprocessing.Value(typecode_or_type, *args[, lock]) and
multiprocessing.sharedctypes.Value(typecode_or_type, *args[, lock]) the
documentation states:
> By default the return value is actually a synchronized wrapper for the
> object. [...]
> If lock is True (the default) then a new lock object is created to
> synchronize access to the value. If lock is a Lock or RLock object then that
> will be used to synchronize access to the value. If lock is False then
> access to the returned object will not be automatically protected by a lock,
> so it will not necessarily be “process-safe”.

(By the way, I'm not sure why both, multiprocessing.Value and
multiprocessing.sharedctypes.Value are documented.  They appear to be the same
thing.)

The following tests (also attached as file) show that lost updates may occur
if several instances of multiprocessing.Process increment the same Value that
is passed as args parameter.

def do_inc(integer):
"""Increment integer.value for multiprocessing.Value integer."""
integer.value += 1

def do_test(notasks):
"""Create notasks processes, each incrementing the same Value.

As the Value is initialized to 0, its final value is expected to be
notasks.
"""
tasks = list()
integer = multiprocessing.sharedctypes.Value("i", 0)
for run in range(notasks):
proc = multiprocessing.Process(target=do_inc, args=(integer,))
proc.start()
tasks.append(proc)
for proc in tasks:
proc.join()
if integer.value != notasks:
logging.error(
"Unexpected value: %d (expected: %d)", integer.value, notasks)

if __name__ == "__main__":
do_test(100)


Sample invocations and results:

Note that on a single CPU machine the error is not reported for every
execution but only for about every third run.
$ python --version
Python 2.6.5
$ uname -a
Linux ubuntu-desktop 2.6.32.11+drm33.2 #2 Fri Jun 18 20:30:49 CEST 2010 i686 
GNU/Linux
$ python test_multiprocessing.py
ERROR:root:Unexpected value: 99 (expected: 100)

On a quadcore, the error occurs almost every time.
$ uname -a
Linux PC 2.6.35.13 #4 SMP Tue Dec 20 15:22:02 CET 2011 x86_64 GNU/Linux
$ ~/local/Python-2.7.3/python test_multiprocessing.py
ERROR:root:Unexpected value: 95 (expected: 100)
$ ~/local/Python-3.3.0/python test_multiprocessing.py
ERROR:root:Unexpected value: 86 (expected: 100)

--
components: Library (Lib)
files: test_multiprocessing.py
messages: 180251
nosy: lechten
priority: normal
severity: normal
status: open
title: Lost updates with multiprocessing.Value
type: behavior
versions: Python 3.3
Added file: http://bugs.python.org/file28782/test_multiprocessing.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com