[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Ethan Furman]

On 06/05/2020 07:32 AM, Mark Shannon wrote:


3. It can be implemented on top of the multiprocessing module, for testing. A 
more efficient implementation can be developed once sub-interpreters prove 
useful.


Isn't part of the impetus for in-process sub-interpreters the 
Python-embedded-in-language-X use-case?  Isn't multiprocessing a poor solution 
then?

--
~Ethan~
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BEGS7PQANGGSDLIPE7NIGNEGIQWNGSL3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Eric V. Smith]

On 6/5/2020 11:11 AM, Edwin Zimmerman wrote:

Advantages of the one-to-one model
--

1. It's less bug prone. It is much easier to reason about code working
in a single address space. Most code assumes

I'm curious where reasoning about address spaces comes into writing Python 
code?  I can't say that address space has ever been a
concern to me when coding in Python.


I don't know enough about Python code with subinterpreters to comment 
there. But for the C code that makes up much of CPython: it's very 
difficult to inspect code and know you aren't accidentally sharing 
objects between interpreters.


Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZETIVQERA6Q37BTTOIWYXBKXO432OR2L/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Jim J. Jewett]

I don't think that sharing data only by copying is the final plan.  Proxied 
objects seem like a fairly obvious extension.

I am also a bit suspicious of that great timing; perhaps latency is also 
important for startup?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/QTNMF22BFGKUQELM6XICSQ5PCHVUZIRJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Nick Coghlan]

Multiprocessing serialisation overheads are abysmal. With enough OS support
you can attempt to mitigate that via shared memory mechanisms (which Davin
added to the standard library), but it's impossible to get the overhead of
doing that as low as actually using the address space of one OS process.

For the rest of the email... multiprocessing isn't going anywhere.

Within-process parallelism is just aiming to provide another trade-off
point in design space for CPU bound workloads (one roughly comparable to
the point where JS web workers sit).

Cheers,
Nick.

On Sat., 6 Jun. 2020, 12:39 am Mark Shannon,  wrote:

> Hi,
>
> There have been a lot of changes both to the C API and to internal
> implementations to allow multiple interpreters in a single O/S process.
>
> These changes cause backwards compatibility changes, have a negative
> performance impact, and cause a lot of churn.
>
> While I'm in favour of PEP 554, or some similar model for parallelism in
> Python, I am opposed to the changes we are currently making to support it.
>
>
> What are sub-interpreters?
> --
>
> A sub-interpreter is a logically independent Python process which
> supports inter-interpreter communication built on shared memory and
> channels. Passing of Python objects is supported, but only by copying,
> not by reference. Data can be shared via buffers.
>
>
> How can they be implemented to support parallelism?
> ---
>
> There are two obvious options.
> a) Many sub-interpreters in a single O/S process. I will call this the
> many-to-one model (many interpreters in one O/S process).
> b) One sub-interpreter per O/S process. This is what we currently have
> for multiprocessing. I will call this the one-to-one model (one
> interpreter in one O/S process).
>
> There seems to be an assumption amongst those working on PEP 554 that
> the many-to-one model is the only way to support sub-interpreters that
> can execute in parallel.
> This isn't true. The one-to-one model has many advantages.
>
>
> Advantages of the one-to-one model
> --
>
> 1. It's less bug prone. It is much easier to reason about code working
> in a single address space. Most code assumes
>
> 2. It's more secure. Separate O/S processes provide a much stronger
> boundary between interpreters. This is why some browsers use separate
> processes for browser tabs.
>
> 3. It can be implemented on top of the multiprocessing module, for
> testing. A more efficient implementation can be developed once
> sub-interpreters prove useful.
>
> 4. The required changes should have no negative performance impact.
>
> 5. Third party modules should continue to work as they do now.
>
> 6. It takes much less work :)
>
>
> Performance
> ---
>
> Creating O/S processes is usually considered to be slow. Whilst
> processes are undoubtedly slower to create than threads, the absolute
> time to create a process is small; well under 1ms on linux.
>
> Creating a new sub-interpreter typically requires importing quite a few
> modules before any useful work can be done.
> The time spent doing these imports will dominate the time to create an
> O/S process or thread.
>
> If sub-interpreters are to be used for parallelism, there is no need to
> have many more sub-interpreters than CPU cores, so the overhead should
> be small. For additional concurrency, threads or coroutines can be used.
>
> The one-to-one model is faster as it uses the hardware for interpreter
> separation, whereas the many-to-one model must use software.
> Process separation by the hardware virtual memory system has zero cost.
> Separation done in software needs extra memory reads when doing
> allocation or deallocation.
>
> Overall, for any interpreter that runs for a second or more, it is
> likely that the one-to-one model would be faster.
>
>
> Timings of multiprocessing & threads on my machine (6-core 2019 laptop)
> ---
>
> #Threads
>
> def foo():
>  pass
>
> def spawn_and_join(count):
>  threads = [ Thread(target=foo, args=()) for _ in range(count) ]
>  for t in threads:
>  t.start()
>  for t in threads:
>  t.join()
>
> spawn_and_join(1000)
>
> # Processes
>
> def spawn_and_join(count):
>  processes = [ Process(target=foo, args=()) for _ in range(count) ]
>  for p in processes:
>  p.start()
>  for p in processes:
>  p.join()
>
> spawn_and_join(1000)
>
> Wall clock time for threads:
> 86ms. Less than 0.1ms per thread.
>
> Wall clock time for processes:
> 370ms. Less than 0.4ms per process.
>
> Processes are slower, but plenty fast enough.
>
>
> Cheers,
> Mark.
>
>
>
>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message arc

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Guido van Rossum]

Has anybody brought up the problem yet that if one subinterpreter
encounters a hard crash (say, it segfaults due to a bug in a C extension
module), all subinterpreters active at that moment in the same process are
likely to lose all their outstanding work, without a chance of recovery?

(Of course once we have locks in shared memory, a crashed process leaving a
lock behind may also screw up everybody else, though perhaps less severely.)

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AMX6KO7GGGAAHTVRP34OMUA7ROCDHKSM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Edwin]

That's so, but threads have this problem too.  I don't think this discussion is 
about finding a "perfect" solution or an "ultimate" way of doing things, rather 
it is about the varying opinions on certain design tradeoffs.  If I'm satisfied 
that subinterpreters are the correct solution to my particular need, why 
shouldn't I have the privilege of doing so?



--Edwin

- Original Message -
From: Guido van Rossum (gu...@python.org)
Date: 06/16/20 13:30
To: Python Dev (python-dev@python.org)
Subject: [Python-Dev] Re: Should we be making so many changes in pursuit of PEP 
554?


Has anybody brought up the problem yet that if one subinterpreter encounters a 
hard crash (say, it segfaults due to a bug in a C extension module), all 
subinterpreters active at that moment in the same process are likely to lose 
all their outstanding work, without a chance of recovery?

(Of course once we have locks in shared memory, a crashed process leaving a 
lock behind may also screw up everybody else, though perhaps less severely.)
--


--Guido van Rossum (python.org/~guido)
Pronouns: he/him (why is my pronoun here?)

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AMX6KO7GGGAAHTVRP34OMUA7ROCDHKSM/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/BN2KKUVRMPBZGHLAGUZK5TCOBYTYBMVV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Guido van Rossum]

On Tue, Jun 16, 2020 at 10:52 AM Edwin  wrote:

>
> That's so, but threads have this problem too.  I don't think this
> discussion is about finding a "perfect" solution or an "ultimate" way of
> doing things, rather it is about the varying opinions on certain design
> tradeoffs.  If I'm satisfied that subinterpreters are the correct solution
> to my particular need, why shouldn't I have the privilege of doing so?
>

Interesting choice of word. This is open source, no feature is free, you
are not entitled to anything in particular.

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/5CRKAFLGWUVLX7U2ZSCQYJCWMJR635RX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Mark Shannon]

On 16/06/2020 1:24 pm, Nick Coghlan wrote:
Multiprocessing serialisation overheads are abysmal. With enough OS 
support you can attempt to mitigate that via shared memory mechanisms 
(which Davin added to the standard library), but it's impossible to get 
the overhead of doing that as low as actually using the address space of 
one OS process.


What does "multiprocessing serialisation" even mean? I assume you mean 
the overhead of serializing objects for communication between processes.


The cost of serializing an object has absolutely nothing to do with 
which process the interpreter is running in.


Separate interpreters within a single process will still need to 
serialize objects for communication.


The overhead of passing data through shared memory is the same for 
threads and processes. It's just memory.


Can we please stick to facts and not throw around terms like "abysmal" 
with no data whatsoever to back it up.




For the rest of the email... multiprocessing isn't going anywhere.

Within-process parallelism is just aiming to provide another trade-off 
point in design space for CPU bound workloads (one roughly comparable to 
the point where JS web workers sit).


Cheers,
Nick.

On Sat., 6 Jun. 2020, 12:39 am Mark Shannon, > wrote:


Hi,

There have been a lot of changes both to the C API and to internal
implementations to allow multiple interpreters in a single O/S process.

These changes cause backwards compatibility changes, have a negative
performance impact, and cause a lot of churn.

While I'm in favour of PEP 554, or some similar model for
parallelism in
Python, I am opposed to the changes we are currently making to
support it.


What are sub-interpreters?
--

A sub-interpreter is a logically independent Python process which
supports inter-interpreter communication built on shared memory and
channels. Passing of Python objects is supported, but only by copying,
not by reference. Data can be shared via buffers.


How can they be implemented to support parallelism?
---

There are two obvious options.
a) Many sub-interpreters in a single O/S process. I will call this the
many-to-one model (many interpreters in one O/S process).
b) One sub-interpreter per O/S process. This is what we currently have
for multiprocessing. I will call this the one-to-one model (one
interpreter in one O/S process).

There seems to be an assumption amongst those working on PEP 554 that
the many-to-one model is the only way to support sub-interpreters that
can execute in parallel.
This isn't true. The one-to-one model has many advantages.


Advantages of the one-to-one model
--

1. It's less bug prone. It is much easier to reason about code working
in a single address space. Most code assumes

2. It's more secure. Separate O/S processes provide a much stronger
boundary between interpreters. This is why some browsers use separate
processes for browser tabs.

3. It can be implemented on top of the multiprocessing module, for
testing. A more efficient implementation can be developed once
sub-interpreters prove useful.

4. The required changes should have no negative performance impact.

5. Third party modules should continue to work as they do now.

6. It takes much less work :)


Performance
---

Creating O/S processes is usually considered to be slow. Whilst
processes are undoubtedly slower to create than threads, the absolute
time to create a process is small; well under 1ms on linux.

Creating a new sub-interpreter typically requires importing quite a few
modules before any useful work can be done.
The time spent doing these imports will dominate the time to create an
O/S process or thread.

If sub-interpreters are to be used for parallelism, there is no need to
have many more sub-interpreters than CPU cores, so the overhead should
be small. For additional concurrency, threads or coroutines can be used.

The one-to-one model is faster as it uses the hardware for interpreter
separation, whereas the many-to-one model must use software.
Process separation by the hardware virtual memory system has zero cost.
Separation done in software needs extra memory reads when doing
allocation or deallocation.

Overall, for any interpreter that runs for a second or more, it is
likely that the one-to-one model would be faster.


Timings of multiprocessing & threads on my machine (6-core 2019 laptop)
---

#Threads

def foo():
      pass

def spawn_and_join(count):
      threads = [ Thread(target=foo, args=()) for _ in range(count) ]
      for t in threads:
       

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Brett Cannon]

I wanted to let people know that the four of us on the SC not driving this work 
-- i.e. everyone but Victor -- talked about this at our last meeting and we 
support the work to isolate interpreter state from being global. There are 
benefits for the situation where you have to integrate CPython with other code 
which does its own thread management (i.e. the embedded scenario). It also 
helps from an organizational perspective of the code and thus we believe leads 
to easier maintainability long-term. We are okay with the performance trade-off 
required for this work.

I will also say that while this work is a prerequisite for PEP 554 as currently 
proposed, it does not mean the SC believes PEP 554 will ultimately be accepted. 
We view this work as independently motivated from PEP 554.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YRDOIQ5UOXLUDK7EXCBZYBBXHJDIXG3W/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Petr Viktorin]

On 2020-06-16 19:20, Guido van Rossum wrote:
Has anybody brought up the problem yet that if one subinterpreter 
encounters a hard crash (say, it segfaults due to a bug in a C extension 
module), all subinterpreters active at that moment in the same process 
are likely to lose all their outstanding work, without a chance of recovery?


(Of course once we have locks in shared memory, a crashed process 
leaving a lock behind may also screw up everybody else, though perhaps 
less severely.)


Not really.
Asyncio has the same problem; has anyone brought this issue up there? 
(Granted, asyncio probably didn't uncover too many issues in extension 
modules, but if it did, I assume they would get fixed.)


If you're worried about segfaults, then you should use multiple 
processes. That will always give you better isolation. But I don't think 
it's a reason to stop improving interpreter isolation.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/AGMIWNEGH2PJN473EGGB7J4LY4RYFQA5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Petr Viktorin]

On 2020-06-16 20:28, Mark Shannon wrote:


On 16/06/2020 1:24 pm, Nick Coghlan wrote:
Multiprocessing serialisation overheads are abysmal. With enough OS 
support you can attempt to mitigate that via shared memory mechanisms 
(which Davin added to the standard library), but it's impossible to 
get the overhead of doing that as low as actually using the address 
space of one OS process.


What does "multiprocessing serialisation" even mean? I assume you mean 
the overhead of serializing objects for communication between processes.


The cost of serializing an object has absolutely nothing to do with 
which process the interpreter is running in.


Separate interpreters within a single process will still need to 
serialize objects for communication.


The overhead of passing data through shared memory is the same for 
threads and processes. It's just memory.


Can we please stick to facts and not throw around terms like "abysmal" 
with no data whatsoever to back it up.



I'd like to get back to the facts. Let me quote the original mail from 
this thread:


On 2020-06-05 16:32, Mark Shannon wrote:

While I'm in favour of PEP 554, or some similar model for parallelism in 
Python, I am opposed to the changes we are currently making to support it.


Which changes?
There are several efforts in this general space. Personally, I also 
don't agree with them all. And I think the reason I wasn't able to 
formulate too many replies to you is that we don't have a common 
understanding of what is being discussed, and of the modivations behind 
the changes.




You seem to try convince everyone that multiple processes are better (at 
isolation, and at performance) than multiple interpreters in one 
process. And I see the point: if you can live with the restriction of 
multiple processes, they probably are a better choice!
But I don't think PEPs 554, 489, 573, etc. are about choosing between 
multiprocessing and multiple interpreters; they're about making multiple 
interpreters better than they currently are.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MTPO3VZRTRR6R23X462QXJ2X74E5YX22/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Nick Coghlan]

On Wed., 17 Jun. 2020, 4:28 am Mark Shannon,  wrote:

>
> On 16/06/2020 1:24 pm, Nick Coghlan wrote:
> > Multiprocessing serialisation overheads are abysmal. With enough OS
> > support you can attempt to mitigate that via shared memory mechanisms
> > (which Davin added to the standard library), but it's impossible to get
> > the overhead of doing that as low as actually using the address space of
> > one OS process.
>
> What does "multiprocessing serialisation" even mean? I assume you mean
> the overhead of serializing objects for communication between processes.
>
> The cost of serializing an object has absolutely nothing to do with
> which process the interpreter is running in.
>
> Separate interpreters within a single process will still need to
> serialize objects for communication.
>
> The overhead of passing data through shared memory is the same for
> threads and processes. It's just memory.
>

No, it's not. With multiple processes, you have to instruct the OS to poke
holes in the isolated-by-default behavior in order to give multiple Python
interpreters access to a common memory store.

When the interpreters are in the same process, that isn't true - to give
multiple Python interpreters access, you just give them all a pointer to
the common data.

This will work most easily when the state being shared is not itself a
Python object. PEP 3118 buffers will be one example of that (including when
using pickle protocol 5 for data passing between interpreters), but the
application embedding use case (where there's no real "main" interpreter,
just multiple subinterpreters manipulating the application state) is the
other one I expect to be reasonably common.

This is the Ceph/mod_wsgi/hexchat plugin use case, which is beneficial
enough for people to have pursued it *despite* the significant usability
problems with the current state of the subinterpreter support.

Doing full blown zero-copy ownership transfer of actual Python objects
would be more difficult, since the current plan is to have separate memory
allocation pools per interpreter to avoid excessive locking overhead, so I
don't currently expect to see that any time soon, even if PEP 554 is
accepted. Assuming that remains the case, I'd expect multiprocessing to
remain the default choice for CPU bound use cases where all the interesting
state is held in Python objects (if you're going to have to mess about with
a separate heap of shared objects anyway, you may as well also enjoy the
benefits of greater process isolation).

Cheers,
Nick.

>

>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GFBTU7MLX5V4KQYDSO6WYGLUH6XO2SIA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Emily Bowman]

On Wed, Jun 17, 2020 at 5:56 AM Nick Coghlan  wrote:

>
> Doing full blown zero-copy ownership transfer of actual Python objects
> would be more difficult, since the current plan is to have separate memory
> allocation pools per interpreter to avoid excessive locking overhead, so I
> don't currently expect to see that any time soon, even if PEP 554 is
> accepted. Assuming that remains the case, I'd expect multiprocessing to
> remain the default choice for CPU bound use cases where all the interesting
> state is held in Python objects (if you're going to have to mess about with
> a separate heap of shared objects anyway, you may as well also enjoy the
> benefits of greater process isolation).
>

So most likely there wouldn't be any way to share something like a
bytearray or another buffer interface-compatible type for some time. That's
too bad, I was hoping to have shared arrays that I could put a memoryview
on in each thread/interpreter and deal with locking if I need to, but I
suppose I can work through an extension once the changes stabilize.
Packages like NumPy have had their own opaque C types and C-only routines
to handle all the big threading outside of Python as a workaround for a
long time now.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7MR4MJDTFQIEEMY6WQ2IE4EYFYU6PPKJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Eric Snow]

On Wed, Jun 17, 2020 at 11:42 AM Emily Bowman  wrote:
> So most likely there wouldn't be any way to share something like a bytearray 
> or another
> buffer interface-compatible type for some time. That's too bad, I was hoping 
> to have
> shared arrays that I could put a memoryview on in each thread/interpreter and 
> deal with
> locking if I need to,

Earlier versions of PEP 554 did have a "SendChannel.send_buffer()"
method for this but we tabled it in the interest of simplifying.  That
said, I expect we'll add something like that separately later.

> but I suppose I can work through an extension once the changes stabilize.

Yep.  This should be totally doable in an extension and hopefully
without much effort.

> Packages like NumPy have had their own opaque C types and C-only routines to 
> handle all the big threading outside of Python as a workaround for a long 
> time now.

As a workaround for what?  This sounds interesting. :)

-eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/D2APLOLR4UL7VXLNRFGFWOUN5MPIO2BV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: Should we be making so many changes in pursuit of PEP 554?

[Nick Coghlan]

On Thu., 18 Jun. 2020, 6:06 am Eric Snow, 
wrote:

> On Wed, Jun 17, 2020 at 11:42 AM Emily Bowman 
> wrote:
> > So most likely there wouldn't be any way to share something like a
> bytearray or another
> > buffer interface-compatible type for some time. That's too bad, I was
> hoping to have
> > shared arrays that I could put a memoryview on in each
> thread/interpreter and deal with
> > locking if I need to,
>
> Earlier versions of PEP 554 did have a "SendChannel.send_buffer()"
> method for this but we tabled it in the interest of simplifying.  That
> said, I expect we'll add something like that separately later.
>

Right, buffers are different because the receiving interpreter can set up a
memoryview that refers to storage allocated by the source interpreter.

So the Python objects aren't shared (avoiding refcounting complications),
but the expensive data copying step can still be avoided.


> > Packages like NumPy have had their own opaque C types and C-only
> routines to handle all the big threading outside of Python as a workaround
> for a long time now.
>
> As a workaround for what?  This sounds interesting. :)
>

For the GIL - lots of NumPy operations are in pure C or FORTRAN and will
happily use as many CPUs as you have available.

Cheers,
Nick.



> -eric
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/VXLNERRZLDAFX725JCYEQ5WX3MK7DEE3/
Code of Conduct: http://python.org/psf/codeofconduct/