[issue37754] Consistency of Unix's shared_memory implementation with windows

[Davin Potts]

Davin Potts  added the comment:

I have created issue38119 to track a fix to the inappropriate use of resource 
tracker with shared memory segments, but this does not replace or supersede 
what is discussed here.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Consistency of Unix's shared_memory implementation with windows

[Vinay Sharma]

Vinay Sharma  added the comment:

Hi @davin,
I researched on lots of approaches to solve this problem, and I have listed 
down some of the best ones.

1. As Eryk Sun suggested initially to use advisory locking to implement a 
reference count type of mechanism. I implemented this in the current Pull 
Request already open. And it works great on most platforms, but fails on MacOS. 
This is because MacOS makes no file-system entry like Linux in /dev/shm. In 
fact it makes no filesystem entry for the created shared memory segment. 
Therefore this won't work.



2. I thought of creating a manual file entry for the created shared memory 
segment in /tmp in MacOS. This will work just fine unless the user manually 
changes the permissions of /tmp directory on MacOS. And we will have to rely on 
the fact that /tmp is writable if we use this approach.



3. Shared Semaphores: This is a very interesting approach to implement 
reference count, where a semaphore is created corresponding to every shared 
memory segment, which keeps reference count of the shared memory.
Resource Tracker will clear the shared memory segment and the shared semaphore 
as soon as the value of the shared semaphore becomes 0. 

The only problem with this approach is to decide the name of shared semaphore. 
We will have to define a standardised way to get extract shared semaphore's 
name from shared segment's name.

For instance, shared_semaphore_name = 'psem' + shared_segment_name.
This can cause problems if a semaphore with shared_semaphore_name is already 
present.

This could be solved by taking any available name and storing it inside shared 
memory segment upon creation, and then extracting this name to open the shared 
semaphore.



4. Another way could be to initialize a semaphore by sem_init() inside the 
shared memory segment. For example the first 4 bytes can be reserved for 
semaphore. But, since MacOS has deprecated sem_init(), it wouldn't be a good 
practice.



5. Atomic Calls: I think this is the most simple and best way to solve the 
above problem. We can reserve first 4 bytes for an integer which is nothing but 
the reference count of shared memory, and to prevent data races we could use 
atomic calls to update it.
gcc has inbuilt support for some atomic operations. I have tested them using 
processes by updating them inside shared memory. And they work great.

Following are some atomic calls:

type __sync_add_and_fetch (type *ptr, type value, ...)
type __sync_sub_and_fetch (type *ptr, type value, ...)
type __sync_or_and_fetch (type *ptr, type value, ...)

Documentation of these can be found at below links:
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fsync-Builtins.html#g_t_005f_005fsync-Builtins
https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html#g_t_005f_005fatomic-Builtins

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Consistency of Unix's shared_memory implementation with windows

[Davin Potts]

Davin Potts  added the comment:

A shared semaphore approach for the resource tracker sounds appealing as a way 
to make the behavior on Windows and posix systems more consistent.  However 
this might get implemented, we should not artificially prevent users from 
having some option to persist beyond the last Python process's exit.

I like the point that @eryksun makes that we could instead consider using 
NtMakePermanentObject on Windows to permit more posix-like behavior instead, 
but I do not think we want to head down a path of using undocumented NT APIs.

In the current code, the resource tracker inappropriately triggers 
_posixshmem.shm_unlink; we need to fix this in the immediate short term (before 
3.8 is released) as it breaks the expected behavior @vinay0410 describes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Consistency of Unix's shared_memory implementation with windows

[Vinay Sharma]

Vinay Sharma  added the comment:

Since, advisory locking doesn't work on integer file descriptors which are 
returned by shm_open on macos, I was thinking of an alternative way of fixing 
this.

I was thinking of using a shared semaphore, which will store the reference 
count of the processes using the shared memory segment.
resource_tracker will unlink the shared_memory and the shared semaphore, when 
the count stored by shared semaphore becomes 0.
This will ensure that neither the shared memory segment nor the shared 
semaphore leaks.

Does this sound good ?
Any suggestions would be very helpful.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Consistency of Unix's shared_memory implementation with windows

[Vinay Sharma]

Vinay Sharma  added the comment:

Also, shm_open returns an integer file descriptor.
And when this file descriptor is passed too fcntl.flock (in macOS) it throws 
the following error:
OSError: [Errno 45] Operation not supported

Whereas, the same code works fine on linux.

Therefore, I have doubts whether Macos flock implementation support locking 
shared_memory files.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue37754] Consistency of Unix's shared_memory implementation with windows

[Vinay Sharma]

Vinay Sharma  added the comment:

Hi,
I just opened a PR implementing a fix very similar to your suggestions. I am 
using advisory locking using fcntl.flock.
And I am locking on file descriptors.
If you see my PR, in resource tracker I am opening a file 
"/dev/shm/", and trying to acquire exclusive lock on the same.
And it's working great on Linux.
Since, resource_tracker is spawned as a different process, I can't directly use 
file descriptors.

But macOS doesn't have any memory mapped files created by shm_open in /dev/shm. 
In fact, it doesn't store any reference to memory mapped files in the 
filesystem.

Therefore it get's difficult to get the file descriptor in resource tracker.
Also, is there a good way to pass file descriptors between processes.

Any ideas on the above issue will be much appreciated.

--
title: Persistence of Shared Memory Segment after process exits -> Consistency 
of Unix's shared_memory implementation with windows

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com