Hi!
Well, the plugins would be created by third-parties and I'd like them
to enable bunding of modules with their plugins.
I am afraid of modules with the same name, but being different, or
different versions of modules being used by different plugins. If
plugins share an interpreter, the module with a given name that is
imported first sticks around forever and for all plugins.
I am thinking about this design:
- Plugins don't maintain state in their Python world. They expose
functions, my application calls them.
- Everytime I call into them, they are presented with a clean global
namespace. After the call, the namespace (dict) is thrown away. That
releases any objects the plugin code has created.
- So, then I could also actively unload modules they loaded. But I do
know that this is problematic in particular for modules that use
native code.
I am interested in both a short-term and a long-term solution.
Actually, making subinterpreters work better is pretty sexy ...
because it's hard. :-)
Stephan
Am Mi., 23. Jan. 2019 um 11:30 Uhr schrieb Petr Viktorin <encu...@gmail.com>:
>
> On 1/23/19 3:33 AM, Stephan Reiter wrote:
> > Thanks for the answers so far. I appreciate them!
> >
> > Nathaniel, I'd like to allow Python plugins in my application. A
> > plugin should be allowed to bring its own modules along (i.e.
> > plugin-specific subdir is in sys.path when the plugin is active) and
> > hence some isolation of them will be needed, so that they can use
> > different versions of a given module. That's my main motivation for
> > using subinterpreters.
> > I thought about running plugins out-of-processes - a separate process
> > for every plugin - and allow them to communicate with my application
> > via RPC. But that makes it more complex to implement the API my
> > application will offer and will slow down things due to the need to
> > copy data.
> > Maybe you have another idea for me? :)
>
> Try to make the plugins work together. Look into using pip/PyPI for your
> plugins. Try to make it so each package ("plugin") would have only one
> module/package, and dependencies would be other packages that can be
> installed individually and shared. And keep in mind you can set up your
> own package index, or distribute/install individual package files.
>
> If that's not possible, and you want things to work now, go with subprocess.
>
> If you want to help make subinterpreters work better, there are several
> people scratching at the problem from different angles. Most/all would
> welcome help, but don't expect any short-term benefits.
> (FWIW, my own effort is currently blocked on PEP 580, and I hope to move
> forward after a Council is elected.)
>
>
> > Henry, Antoine, thanks for your input; I'll check out the tests and
> > see what I can learn from issue 10915.
> >
> > Stephan
> >
> > Am Di., 22. Jan. 2019 um 22:39 Uhr schrieb Nathaniel Smith <n...@pobox.com>:
> >>
> >> There are currently numerous incompatibilities between numpy and
> >> subinterpreters, and no concrete plan for fixing them. The numpy team does
> >> not consider subinterpreters to be a supported configuration, and can't
> >> help you with any issues you run into. I know the concept of
> >> subinterpreters is really appealing, but unfortunately the CPython
> >> implementation is not really mature or widely supported... are you
> >> absolutely certain you need to use subinterpreters for your application?
> >>
> >> On Tue, Jan 22, 2019, 08:27 Stephan Reiter <stephan.rei...@gmail.com wrote:
> >>>
> >>> Hi all!
> >>>
> >>> I am new to the list and arriving with a concrete problem that I'd
> >>> like to fix myself.
> >>>
> >>> I am embedding Python (3.6) into my C++ application and I would like
> >>> to run Python scripts isolated from each other using sub-interpreters.
> >>> I am not using threads; everything is supposed to run in the
> >>> application's main thread.
> >>>
> >>> I noticed that if I create an interpreter, switch to it and execute
> >>> code that imports numpy (1.13), my application will hang.
> >>>
> >>> ntdll.dll!NtWaitForSingleObject() Unknown
> >>> KernelBase.dll!WaitForSingleObjectEx() Unknown
> >>>> python36.dll!_PyCOND_WAIT_MS(_PyCOND_T * cv=0x00000000748a67a0,
> >>>> _RTL_CRITICAL_SECTION * cs=0x00000000748a6778, unsigned long ms=5) Line
> >>>> 245 C
> >>> [Inline Frame] python36.dll!PyCOND_TIMEDWAIT(_PyCOND_T *) Line 275 C
> >>> python36.dll!take_gil(_ts * tstate=0x0000023251cbc260) Line 224 C
> >>> python36.dll!PyEval_RestoreThread(_ts * tstate=0x0000023251cbc260)
> >>> Line 370 C
> >>> python36.dll!PyGILState_Ensure() Line 855 C
> >>> umath.cp36-win_amd64.pyd!00007ff8c6306ab2() Unknown
> >>> umath.cp36-win_amd64.pyd!00007ff8c630723c() Unknown
> >>> umath.cp36-win_amd64.pyd!00007ff8c6303a1d() Unknown
> >>> umath.cp36-win_amd64.pyd!00007ff8c63077c0() Unknown
> >>> umath.cp36-win_amd64.pyd!00007ff8c62ff926() Unknown
> >>> [Inline Frame] python36.dll!_PyObject_FastCallDict(_object *) Line
> >>> 2316 C
> >>> [Inline Frame] python36.dll!_PyObject_FastCallKeywords(_object *) Line
> >>> 2480 C
> >>> python36.dll!call_function(_object * * *
> >>> pp_stack=0x00000048be5f5e40, __int64 oparg, _object * kwnames) Line
> >>> 4822 C
> >>>
> >>> Numpy's extension umath calls PyGILState_Ensure(), which in turn calls
> >>> PyEval_RestoreThread on the (auto) threadstate of the main
> >>> interpreter. And that's wrong.
> >>> We are already holding the GIL with the threadstate of our current
> >>> sub-interpreter, so there's no need to switch.
> >>>
> >>> I know that the GIL API is not fully compatible with sub-interpreters,
> >>> as issues #10915 and #15751 illustrate.
> >>>
> >>> But since I need to support calls to PyGILState_Ensure - numpy is the
> >>> best example -, I am trying to improve the situation here:
> >>> https://github.com/stephanreiter/cpython/commit/d9d3451b038af2820f500843b6a88f57270e1597
> >>>
> >>> That change may be naive, but it does the trick for my use case. If
> >>> totally wrong, I don't mind pursuing another alley.
> >>>
> >>> Essentially, I'd like to ask for some guidance in how to tackle this
> >>> problem while keeping the current GIL API unchanged (to avoid breaking
> >>> modules).
> >>>
> >>> I am also wondering how I can test any changes I am proposing. Is
> >>> there a test suite for interpreters, for example?
> >>>
> >>> Thank you very much,
> >>> Stephan
> >>> _______________________________________________
> >>> Python-Dev mailing list
> >>> Python-Dev@python.org
> >>> https://mail.python.org/mailman/listinfo/python-dev
> >>> Unsubscribe:
> >>> https://mail.python.org/mailman/options/python-dev/njs%40pobox.com
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev@python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com
> >
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
