[issue37754] Persistence of Shared Memory Segment after process exits

[Vinay Sharma]

Change by Vinay Sharma :


--
keywords: +patch
pull_requests: +15154
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/15460

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Persistence of Shared Memory Segment after process exits

[Vinay Sharma]

Change by Vinay Sharma :


--
versions: +Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Persistence of Shared Memory Segment after process exits

[Eryk Sun]

Eryk Sun  added the comment:

> register the shared the shared_memory only when it's created and 
> not when it's attached.

In Windows, the section object is reference counted. I haven't looked into the 
Unix implementation, but maybe it could use advisory locking. After opening 
shared memory via shm_open, a process would try to acquire a shared lock on the 
fd. If it can't acquire the lock, close the fd and fail the open. When exiting, 
a process would remove its shared lock and try to acquire an exclusive lock. If 
it acquires an exclusive lock, it should call shm_unlink because it's the last 
reference.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Persistence of Shared Memory Segment after process exits

[Vinay Sharma]

Vinay Sharma  added the comment:

> In terms of providing "consistent behavior across platforms that can be
> reasonably supported", the behavior suggested above could not
> reasonably be supported in Windows.

I understand that persistence of a shared memory segment after all the 
processes using it exit, can be very difficult on Windows.

But, after testing shared_memory on Windows, the behavior on Windows and Unix 
is not consistent at the moment.

For instance:
Let's say a three processes P1, P2 and P3 are trying to communicate using 
shared memory.
 --> P1 creates the shared memory block, and waits for P2 and P3 to access it.
 --> P2 starts and attaches this shared memory segment, writes some data to it 
and exits.
 --> Now in case of Unix, shm_unlink is called as soon as P2 exits.
 --> Now, P3 starts and tries to attach the shared memory segment.
 --> P3 will not be able to attach the shared memory segment in Unix, because 
shm_unlink has been called on that segment.
 --> Whereas, P3 will be able to attach to the shared memory segment in Windows

One possible solution can be, to register the shared the shared_memory only 
when it's created and not when it's attached.

I think that might make Unix's implementation more consistent with windows.

Any thoughts on the same will be very helpful.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Persistence of Shared Memory Segment after process exits

[Eryk Sun]

Eryk Sun  added the comment:

> PS: I personally believe from my experience that shared memory 
> segments should outlive the process, unless specified otherwise. 
> Also, a argument persist=True, can be added which can ensure 
> that the shared_memory segment outlives the process, and can be 
> used by processes which are spawned later.

In terms of providing "consistent behavior across platforms that can be 
reasonably supported", the behavior suggested above could not reasonably be 
supported in Windows. 

The Section (shared memory) object itself is not a file. It gets created in the 
object namespace, either globally in "\BaseNamedObjects" or in an interactive 
session's "\Sessions\\BaseNamedObjects". 

By default, kernel objects are temporary. Creating permanent named objects 
requires SeCreatePermanentPrivilege, which by default is only granted to SYSTEM 
(sort of like root in Unix). For an object to be accessible across sessions and 
outlive an interactive session, it needs to be global. Creating global Section 
objects requires SeCreateGlobalPrivilege, which by default is only granted to 
administrators and service accounts.

Also, the Windows API has no capability to create permanent objects, so this 
would require the NT API functions NtMakePermanentObject (undocumented, added 
in NT 5.1) and NtMakeTemporaryObject.

--
nosy: +eryksun

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Persistence of Shared Memory Segment after process exits

[Vinay Sharma]

Change by Vinay Sharma :


--
type: enhancement -> behavior

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Persistence of Shared Memory Segment after process exits

[Vinay Sharma]

Vinay Sharma  added the comment:

Hi Davin,
Thanks for replying!

As you said I went through the issue, and now understand why segments should 
not be automatically created if they don't exist.

But, after reading that thread I got to know that shared memory is supposed to 
exist even if the process exits, and it can only be freed by unlink, which I 
also believe is an important functionality required by many use cases as you 
mentioned.

But, this is not the behaviour currently.
As soon as the process exists, all the shared memory created is unlinked.

Also, the documentation currently mentions: "shared memory blocks may outlive 
the original process that created them", which is not the case at all.

Currently, the resource_tracker, unlinks the shared memory, by calling unlink 
as specified here:
```
if os.name == 'posix':
import _multiprocessing
import _posixshmem

_CLEANUP_FUNCS.update({
'semaphore': _multiprocessing.sem_unlink,
'shared_memory': _posixshmem.shm_unlink,
})
```

So, is this an expected behaviour, if yes documentation should be updated, and 
if not the code base should be.

I will be happy to submit a patch in both the cases.

PS: I personally believe from my experience that shared memory segments should 
outlive the process, unless specified otherwise. Also, a argument persist=True, 
can be added which can ensure that the shared_memory segment outlives the 
process, and can be used by processes which are spawned later.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Persistence of Shared Memory Segment after process exits

[Vinay Sharma]

Change by Vinay Sharma :


--
title: alter size of segment using multiprocessing.shared_memory -> Persistence 
of Shared Memory Segment after process exits

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com