-------- Weitergeleitete Nachricht --------
Betreff: Re: [pytest-dev] Future of pytest-xdist/execnet?
Datum: Thu, 24 Oct 2019 16:25:43 -0400
Von: pytest-dev-ow...@python.org
An: i...@ronnypfannschmidt.de
Your message has been rejected, probably because you are not
subscribed to the mailing list and the list's policy is to prohibit
non-members from posting to it. If you think that your messages are
being rejected in error, contact the mailing list owner at
pytest-dev-ow...@python.org.
--- Begin Message ---
Hi Bruno,
i wouldn't mind to loose xdist as it is today,
* looponfail should probably get a restartonfail replacement in core
* the execnet bound hooks are tricky
* new types of scheduling are tricky
I would like to see the next generation of the plugin with a focus on
"concurrent/parallel running" of pytest,
so we can focus on parallel execution and scheduling protocols.
(a key reason why i would like to see a execnet/mitogen backed only for
starting processes of the same python)
so that we don't accidentally generate a undesirable coupling.
Am 24.10.19 um 14:45 schrieb Bruno Oliveira:
Hi everyone,
For some time now |execnet| has been in maintenance only mode, and
even so very few people are willing to maintain it; lately just myself
and I’m not a good choice given that I don’t know the codebase at all,
plus I have tons on my plate already. This poses a problem because
often we have bug reports or feature requests in |pytest-xdist| which
would require fixing or improving |execnet|, and are currently left
without solution.
Ronny and I have on occasion discussed the possibility of changing
|execnet|‘s backend , with the current contenders being
|multiprocessing| for local distribution and |mitogen| for remote
distribution.
I would like to write down some thoughts and gather opinions from the
mailing list members.
multiprocessing
Aside from not needing to maintain |execnet| anymore, being able to
use |multiprocessing| would bring several benefits:
*
|pytest -s| would work just fine, because |multiprocessing| does
not use |stdout/stderr| for communication. This is a often
requested feature but which we can’t (with current expertise)
implement on |execnet|.
*
It would automatically support anything that can be picked to be
transmitted across the wire. Currently execnet does not support
custom user types and some builtins (frozen sets for example).
Without |execnet| we would lose the ability of running the tests in
different interpreter versions, but I believe this has been supplanted
by |tox| in the ecosystem.
actually there wa a floating idea of integrating pytest tox and execnet,
so that one could run pytest against a ssh host, docker containers and
similar and have a correct setup on the remote side
(as opposed to the rsync + manual setup mess we have now)
mitogen
Using |mitogen| for remote execution would provide automatic bootstrap
of the Python environment, which is not currently supported by
|xdist|‘s ssh remote execution, greatly limiting its usefulness in
real-time scenarios.
This is Ronny’s idea and he can add more here if wanted.
an interesting property of mitogen is that it can import from the
controller, which makes for some interesting theoretical cappabilities,
unfortunately i didnt get to explore those, and likely wont in near future.
pytest-xdist2?
All the changes mentioned above would break backward compatibility
*hard*, because details of the |execnet| implementation are currently
exposed to users in the command-line:
|pytest -d --tx popen//python |
And in several hooks:
|def pytest_xdist_newgateway(gateway): """ called on new raw gateway
creation. """ |
(|gateway| is an |execnet| object)
To incorporate the new backends, we would need to severely break both
command-line and existing hooks. Given the level of breakage this
could cause, just bumping the major version in this case doesn’t seem
enough, it would be a disservice to users given that probably nobody
pins pytest-xdist to |<=2|, and it would break the world with hard to
understand error messages once a first major release was released.
Because of this, we have been discussing creating a new package,
|pytest-xdist2| (other suggestions are welcome), without any backward
compatibility guarantees with |pytest-xdist|.
|pytest-xdist2| would, at first, only support local test distribution
using |multiprocessing| (|-n| flag), no new hooks, and no remote
execution. I’ve made a proof of concept here:
https://github.com/pytest-dev/pytest-xdist/pull/479
So it definitely seems possible. One immediate benefit is that the
proof of concept above supports |pytest -s| already, and can transfer
anything that’s pickled over the wire.
So I would like to know what people think of the above ideas, if they
are good/bad, and/or suggest alternatives that we are not seeing right
now.
Ronny, please feel free to add to this email and correct anything I
got wrong.
I believe the Pull request is very instructive, because the points where
you replaced the communication/synchronization primitives mark the key
interaction points we need to manage
abstraction,
as things look in that pr, its not unthinkably hard to do a basic multi
backend thing as a experiment and base for the new stuff
i dare to say that its even thinkable to port xdist to the new model and
have it provide execnet specific hooks/backend
(in case excnet maintenance picks up again)
cheers, Ronny
Cheers,
Bruno
_______________________________________________
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev
--- End Message ---
_______________________________________________
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev
