On Tue, 2022-08-23 at 03:16 +0300, Matti Picus wrote: > > On 22/8/22 18:59, Eric Snow wrote: > > Hi all, > >
<snip> > > devs than just me. Do you have any preference for or against any > > particular venue? > > > > Thanks! > > > > -eric > > _______________________________________________ > > NumPy-Discussion mailing list -- numpy-discussion@python.org > > To unsubscribe send an email to numpy-discussion-le...@python.org > > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > > Member address: matti.pi...@gmail.com > > Thanks for starting the conversation. I would personally prefer the > discussion about NumPy be here, general discussions could be > elsewhere. > > > Please correct me if I am wrong: I understand that multiple > interpreters > would require us to (at least): These days, I was somewhat hoping that the HPy effort might give us subinterpreters without having two seperate efforts going on at the same time. Since much of the refactors are probably identical between the two and it seemed some significant effort might go into that. But of course starting with subinterpreter support without HPy probably also helps the HPy effort. > > - refactor all the static module global state in NumPy and make it > re-entrant or immortal including converting stack-allocated > PyTypeObjects to heap types. What is the status of immortality? None of these seem forbidding on first sight, so long that we can get the state everywhere. Having immortal object seems convenient, but probably not particularly necessary. Most of our state is currently in static variables in functions (usually filled in dynamically at first call). That is very convenient since it doesn't require a global list anywhere. I suppose moving it to module-state may well require a global list (or is there a nice other pattern?). But while tedious, it doesn't seem problematic. Switching to heap types should not be a big deal I suspect. > > - find a mechanism to access the per-interpreter module state > One thing that I am not clear about are e.g. creation functions. They are public C-API so they have no way of getting a "self" or type/module passed in. How will such a function get the module state? Now, we could likely replace those functions in the long run (or even just remove many). But it seems to me that we may need a `PyType_GetModuleByDef()` that is passed _only_ the `module_def`? > - carefully consider places in the code that we steal references > either > intentionally or because that is the CPython C-API we are using > This is an issue for HPy that needs to be cleared up, although I am wondering how important it is for subinterpreters as such? > - measure the performance implications of the necessary changes > > - plan forward/backward compatibility > One other thing I am not quite sure about right now is GIL grabbing. `PyGILState_Ensure()` will continue to work reliably? This used to be one of my main worries. It is also something we can fix-up (pass through additional information), but where a fallback seems needed. Cheers, Sebastian > > This seems like a significant undertaking, and is why we have > rejected > casual calls for supporting multiple interpreters in the past [2], > [3], > [4]. Supporting multiple interpreters is currently not on the NumPy > roadmap [0]. Priorities can be changed, through dialog with the NumPy > community, and others can propose changes to NumPy via NEPs, PRs, and > issues, but we are unlikely to engage directly in the work if it is > not > an agreed upon goal. There are other initiatives around NumPy that > may > dovetail with multiple interpreters. For instance the HPy group hit > many > of the issues above when creating a port of NumPy [5]. It would be > good > to get like-minded people talking about this and to pool resources, > maybe someone on this list has a strong opinion and would be willing > to > put in some work on the subject. > > > One thing CPython could do is to provide clear documentation how to > port > a small c-extension module [1] > > > Matti > > > [0] https://numpy.org/neps/roadmap.html > > [1] https://github.com/python/cpython/issues/79601 > > [2] https://github.com/numpy/numpy/issues/665 > > [3] https://github.com/numpy/numpy/issues/14384 > > [4] https://github.com/numpy/numpy/issues/16963 > > [5] > https://github.com/hpyproject/numpy-hpy/tree/graal-team/hpy#readme > > _______________________________________________ > NumPy-Discussion mailing list -- numpy-discussion@python.org > To unsubscribe send an email to numpy-discussion-le...@python.org > https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ > Member address: sebast...@sipsolutions.net _______________________________________________ NumPy-Discussion mailing list -- numpy-discussion@python.org To unsubscribe send an email to numpy-discussion-le...@python.org https://mail.python.org/mailman3/lists/numpy-discussion.python.org/ Member address: arch...@mail-archive.com
