[Python-ideas] Re: How to prevent shared memory from being corrupted ?

I forgot that there's also Ray: https://github.com/ray-project/ray Ray uses Apache Arrow (and Plasma) under the hood. It seems Plasma was originally developed by Ray team. Don't know how they solve the GC problem. Maybe they disable it. ___

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

How is this a different problem than the cache coherency problem? https://en.wikipedia.org/wiki/Cache_coherence Perhaps that's an unhelpful abstraction? This hasn't gone anywhere: https://en.wikipedia.org/wiki/Distributed_shared_memory#Directory_memory_coherence Here's a great comparison chart

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

On 8/2/2020 12:20 PM, Eric V. Smith wrote: On Sat, 1 Aug 2020 at 22:42, Eric V. Smith > wrote: While they're immutable at the Python level, strings (and all other objects) are mutated at the C level, due to reference count updates. You need to

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

On Sat, 1 Aug 2020 at 22:42, Eric V. Smith > wrote: While they're immutable at the Python level, strings (and all other objects) are mutated at the C level, due to reference count updates. You need to consider this if you're sharing objects without

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

sharedctypes can only be used with related processes. There is no way that you can pass a sharedctype to an unrelated process. multiprocessing.shared_memory was created to handle this i.e. allow usage of shared memory IPC across unrelated processes. > On 02-Aug-2020, at 9:42 PM, Marco Sulla

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

There's also the possibility to use shared ctypes: https://docs.python.org/3/library/multiprocessing.html#shared-ctypes-objects Operations like += which involve a read and write are not atomic. So if, > for instance, you want to atomically increment a shared value it is > insufficient to just do

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

It's best to avoid those synchronization barriers if possible. If you have all of the data in SHM (RAM) on one node, and you need to notify processes / wait for other workers to be available to perform a task that requires that data, you need a method for IPC: a queue, channel subscriptions, a

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

I understand that I won’t need locks with immutable objects at some level, but I don’t understand how they can be used to synchronise shared memory segments. For every change in an immutable object, a copy is created which will have a different address. Now, for processes to use this updated

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

https://docs.dask.org/en/latest/shared.html#known-limitations : > Known Limitations > The shared memory scheduler has some notable limitations: > > - It works on a single machine > - The threaded scheduler is limited by the GIL on Python code, so if your operations are pure python functions, you

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

PyArrow Plasma object ids, "sealing" makes an object immutable, pyristent https://arrow.apache.org/docs/python/plasma.html#object-ids https://arrow.apache.org/docs/python/plasma.html#creating-an-object-buffer > Objects are created in Plasma in two stages. First, they are created, which allocates

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

On 8/1/2020 1:25 PM, Marco Sulla wrote: You don't need locks with immutable objects. Since they're immutable, any operation that usually will mutate the object, generate another immutable instead. The most common example is str: the sum of two strings in Python (and in many other languages)

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

You don't need locks with immutable objects. Since they're immutable, any operation that usually will mutate the object, generate another immutable instead. The most common example is str: the sum of two strings in Python (and in many other languages) produces a new string. This is usually slower

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

> On 01-Aug-2020, at 1:31 AM, Marco Sulla wrote: > > On Thu, 30 Jul 2020 at 12:57, Vinay Sharma via Python-ideas > mailto:python-ideas@python.org>> wrote: > Python has support for atomic types, I guess: > Atomic Int: >

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

On Thu, 30 Jul 2020 at 12:57, Vinay Sharma via Python-ideas < python-ideas@python.org> wrote: > Python has support for atomic types, I guess: > Atomic Int: > https://github.com/python/cpython/blob/master/Include/internal/pycore_atomic.h#L80 > Atomic Store: >

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

I think you are talking about shared/named semaphores. The problem with them is that python doesn’t have support for shared semaphores, so the first step would be to build an API providing access to shared semaphores, and then this API can be used to synchronise shared memory. > On

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

> On 30 Jul 2020, at 11:55, Vinay Sharma via Python-ideas > wrote: > > > >> On 28-Jul-2020, at 5:19 AM, Robert Collins > > wrote: >> >> On Mon, 27 Jul 2020 at 23:24, Vinay Sharma > > wrote: >>> >>> Hi, Thanks for replying.

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

> On 28-Jul-2020, at 5:19 AM, Robert Collins wrote: > > On Mon, 27 Jul 2020 at 23:24, Vinay Sharma wrote: >> >> Hi, Thanks for replying. >> >>> One thing that is worth thinking about is the safety of the API that >>> is put together. A memory segment plus a separate detached semaphore >>>

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

On Mon, 27 Jul 2020 at 23:24, Vinay Sharma wrote: > > Hi, Thanks for replying. > > > One thing that is worth thinking about is the safety of the API that > > is put together. A memory segment plus a separate detached semaphore > > or mutex can be used to build a safe API, but is not itself a safe

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

Hi, Thanks for replying. > One thing that is worth thinking about is the safety of the API that > is put together. A memory segment plus a separate detached semaphore > or mutex can be used to build a safe API, but is not itself a safe > API. Agreed. That’s why I am more inclined to the second

[Python-ideas] Re: How to prevent shared memory from being corrupted ?

On Sun, 26 Jul 2020 at 19:11, Vinay Sharma via Python-ideas wrote: > > Problem: > Currently, let’s say I create a shared_memory segment using > mulitprocessing.shared_memory.SharedMemory in Process 1 and open the same in > Process 2. > Then, I try to write some data to the shared memory segment