[issue39959] (Possible) bug on multiprocessing.shared_memory

[Diogo Flores]

Diogo Flores  added the comment:

Follow up - tested on Linux (The first solution).

The solution presented below will fix the problem with the caveat that the base 
process (the one that creates the shared-memory obj) must outlive any process 
that use the shared-memory.
The rationale is that unless *this* process is creating a new shared-memory 
object (as opposed to attaching itself to an already existing one), then there 
is no point to register itself to be tracked. By making this small change, the 
problem I mentioned when I opened this issue disappears.

#
# 
https://github.com/python/cpython/blob/master/Lib/multiprocessing/shared_memory.py#L116

by changing:

from .resource_tracker import register
register(self._name, "shared_memory")

# To:

if create:
from .resource_tracker import register
register(self._name, "shared_memory")

#

To retain the ability for the base process to be able to exit before those that 
use the shared-memory obj that the base process itself created (the 
current/problematic implementation), as well as fix the issue, I suggest the 
following approach:

When (and only when) a new shared-memory obj is created, such is registered on 
a new class variable of the resource-tracker, hence it can always be accessed 
and closed/unlinked by any process later on - this differs from the current 
approach, where each process that wants to access the shared-memory obj is 
being registered on the resource-tracker.

I look forward for any discussion on the subject.

Thank you,
Diogo

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39959] (Possible) bug on multiprocessing.shared_memory

[Jeff Fischer]

Jeff Fischer  added the comment:

I've run into the same problem. It appears that the SharedMemory class is 
assuming that all clients of a segment are child processes from a single 
parent, and that they inherit the same resource_tracker. If you run separate, 
unrelated processes, you get a separate resource_tracker for each process. 
Then, when a process does a close() followed by a sys.exit(), the resource 
tracker detects a leak and unlinks the segment.

In my application, segment names are stored on the local filesystem and a 
specific process is responsible for unlinking the segment when it is shut down. 
I was able to get this model to work with the current SharedMemory 
implementation by having processes which are just doing a close() also call 
resource_tracker.unregister() directly to prevent their local resource trackers 
from destroying the segment.

I imagine the documentation needs some discussion of the assumed process model 
and either: 1) a statement that you need to inherit the resource tracker from a 
parent process, 2) a blessed way to call the resource tracker to manually 
unregister, or 3) a way to disable the resource tracker when creating the 
SharedMemory object.

--
nosy: +jfischer

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39959] (Possible) bug on multiprocessing.shared_memory

[Raymond Hettinger]

Change by Raymond Hettinger :


--
nosy: +davin, pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue39959] (Possible) bug on multiprocessing.shared_memory

[Diogo Flores]

New submission from Diogo Flores :

Hello,

I came across with what seems like a bug (or at least a disagreement with the 
current documentation).

Discussion:

I expected that after creating a numpy-array (on terminal 1), which is backed 
by shared memory, I would be able to use it in other terminals until I would 
call `shm.unlink()` (on terminal 1), at which point, the memory block would be 
released and no longer accessible.

What happened is that after accessing the numpy-array from terminal 2, I called 
'close()' on the local 'existing_shm' instance and exited the interpreter, 
which displayed the `warning` seen below. 

After, I tried to access the same shared memory block from terminal 3, and a 
FileNotFoundError was raised. (The same error was also raised when I tried to 
call 'shm.unlink()' on terminal 1, after calling 'close()' on terminal 2.)

It seems that calling `close()` on an instance, destroys further access to the 
shared memory block from any point, while what I expected was to be able to 
access the array (i.e. on terminal 2), modify it, "close" my access to it, and 
after be able to access the modified array on i.e. terminal 3.

If the error is on my side I apologize for raising this issue and I would 
appreciate for clarification on what I am doing wrong.

Thank you.
Diogo


Please check below for the commands issued:

## Terminal 1

>>> from multiprocessing import shared_memory
>>> import numpy as np
>>> 
>>> a = np.array([x for x in range(10)])
>>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
>>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
>>> b[:] = a[:]
>>> 
>>> shm.name
'psm_592ec635'


## Terminal 2

>>> from multiprocessing import shared_memory
>>> import numpy as np
>>> 
>>> existing_shm = shared_memory.SharedMemory('psm_592ec635')
>>> c = np.ndarray((10,), dtype=np.int64, buffer=existing_shm.buf)
>>> 
>>> c
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> 
>>> del c
>>> existing_shm.close()
>>> 
>>> exit()
~: /usr/lib/python3.8/multiprocessing/resource_tracker.py:203: UserWarning: 
resource_tracker: There appear to be 1 leaked shared_memory objects to clean up 
at shutdown
  warnings.warn('resource_tracker: There appear to be %d '


## Finally, on terminal 3

>>> from multiprocessing import shared_memory
>>> import numpy as np
>>> 
>>> existing_shm = shared_memory.SharedMemory('psm_592ec635')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.8/multiprocessing/shared_memory.py", line 100, in 
__init__
self._fd = _posixshmem.shm_open(
FileNotFoundError: [Errno 2] No such file or directory: '/psm_592ec635'

--
components: Library (Lib)
messages: 364121
nosy: dxflores
priority: normal
severity: normal
status: open
title: (Possible) bug on multiprocessing.shared_memory
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com