On 11/06/2020 2:50 pm, Riccardo Ghetta wrote:
Hello Mark,
and thanks for your suggestions. However, I'm afraid I haven't explained
our use of python well enough.
On 11/06/2020 12:59, Mark Shannon wrote:
If you need to share objects across threads, then there will be
contention, regardless of how many interpreters there are, or which
processes they are in.
As a rule, we don't use that many python objects. Most of the time a
script calls C++ functions, operating on C++ data.
Perhaps with a small snippet I will explain myself better :
hcpi='INFLEUR'
n_months=3
base_infl=hs_base(hcpi, n_months, 0)
im=hs_fs(hcpi,'sia','m',n_months,0)
ip=hs_fs(hcpi,'sia','m',n_months-1,0)
ir=im+(hs_range()[1].day-1)/month_days(hs_range()[1])*(ip-im)
return ir/base_infl # double
this is a part of a inflation estimation used in pricing an
inflation-linked bond.
hcpi and n_months are really parameters of the script and the hs_
functions are all implemented in C++.
Some are very small and fast like hs_range, others are much more complex
and slow (hs_fs), so we wrap them with
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS.
As you see, here python is used more to direct C++, than manipulate
objects. At GUI level things work a bit differently, but here we just
tried to avoid building and destroying a lot of ephemeral python objects
(unneeded anyway, because all subsequent processing is done by C++).
This python script is only a part of a larger processing done in
parallel by several threads, each operating in distinct instruments.
Evaluating an instrument could involve zero, one, or several of those
scripts.
During evaluation an instrument is bound to a single thread, so from the
point of view of python threads share nothing.
If the additional resource consumption is irrelevant, what's the
objection to spinning up a new processes?
The additional resource consumption of a new python interpreter is
irrelevant, but the process as a whole needs a lot of extra data making
a new process rather costly.
Starting a new process is cheap. On my machine, starting a new Python
process takes under 1ms and uses a few Mbytes.
The overhead largely comes from what you do with the process. The
additional cost of starting a new interpreter is the same regardless of
whether it is in the same process or not. There should be no need to
start a new application process for a new Python interpreter.
Plus there are issues of licensing, synchronization and load balancing
that are much easier to resolve (for our system, at least) with threads
than processes.
Would this prevent CPython starting new processes, or is this just for
processes managed by your application?
Still, we /do/ use multiple processes, but those tend to be across
administrative boundaries, or for very specific issues.
Ciao,
Riccardo
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/2AG665AKKLNYOXO2RNE53TOQ626PBIM7/
Code of Conduct: http://python.org/psf/codeofconduct/
