Hi Freddy,
unfortunately pytest-concurrent is fundamentally broken for managing
fixtures and other details,
as things are my suggestion is to avoid it.
-- Ronny
Am 24.10.19 um 15:22 schrieb Rietdijk:
Hi,
I think having a simple `pytest-multiprocessing` or
`pytest-concurrent` would be extremely useful and cover a lot of
people's use cases and would strongly recommend to have that as a
separate plugin from something bigger that does feature hooks and/or
remote execution.. It seems though as such a package already exists,
https://github.com/reverbc/pytest-concurrent, which offers --concmode
and --concworkers instead.
Frederik
On Thu, Oct 24, 2019 at 2:46 PM Bruno Oliveira <nicodde...@gmail.com
<mailto:nicodde...@gmail.com>> wrote:
Hi everyone,
For some time now |execnet| has been in maintenance only mode, and
even so very few people are willing to maintain it; lately just
myself and I’m not a good choice given that I don’t know the
codebase at all, plus I have tons on my plate already. This poses
a problem because often we have bug reports or feature requests in
|pytest-xdist| which would require fixing or improving |execnet|,
and are currently left without solution.
Ronny and I have on occasion discussed the possibility of changing
|execnet|‘s backend , with the current contenders being
|multiprocessing| for local distribution and |mitogen| for remote
distribution.
I would like to write down some thoughts and gather opinions from
the mailing list members.
multiprocessing
Aside from not needing to maintain |execnet| anymore, being able
to use |multiprocessing| would bring several benefits:
*
|pytest -s| would work just fine, because |multiprocessing|
does not use |stdout/stderr| for communication. This is a
often requested feature but which we can’t (with current
expertise) implement on |execnet|.
*
It would automatically support anything that can be picked to
be transmitted across the wire. Currently execnet does not
support custom user types and some builtins (frozen sets for
example).
Without |execnet| we would lose the ability of running the tests
in different interpreter versions, but I believe this has been
supplanted by |tox| in the ecosystem.
mitogen
Using |mitogen| for remote execution would provide automatic
bootstrap of the Python environment, which is not currently
supported by |xdist|‘s ssh remote execution, greatly limiting its
usefulness in real-time scenarios.
This is Ronny’s idea and he can add more here if wanted.
pytest-xdist2?
All the changes mentioned above would break backward compatibility
*hard*, because details of the |execnet| implementation are
currently exposed to users in the command-line:
|pytest -d --tx popen//python |
And in several hooks:
|def pytest_xdist_newgateway(gateway): """ called on new raw
gateway creation. """ |
(|gateway| is an |execnet| object)
To incorporate the new backends, we would need to severely break
both command-line and existing hooks. Given the level of breakage
this could cause, just bumping the major version in this case
doesn’t seem enough, it would be a disservice to users given that
probably nobody pins pytest-xdist to |<=2|, and it would break the
world with hard to understand error messages once a first major
release was released.
Because of this, we have been discussing creating a new package,
|pytest-xdist2| (other suggestions are welcome), without any
backward compatibility guarantees with |pytest-xdist|.
|pytest-xdist2| would, at first, only support local test
distribution using |multiprocessing| (|-n| flag), no new hooks,
and no remote execution. I’ve made a proof of concept here:
https://github.com/pytest-dev/pytest-xdist/pull/479
So it definitely seems possible. One immediate benefit is that the
proof of concept above supports |pytest -s| already, and can
transfer anything that’s pickled over the wire.
So I would like to know what people think of the above ideas, if
they are good/bad, and/or suggest alternatives that we are not
seeing right now.
Ronny, please feel free to add to this email and correct anything
I got wrong.
Cheers,
Bruno
_______________________________________________
pytest-dev mailing list
pytest-dev@python.org <mailto:pytest-dev@python.org>
https://mail.python.org/mailman/listinfo/pytest-dev
_______________________________________________
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev
_______________________________________________
pytest-dev mailing list
pytest-dev@python.org
https://mail.python.org/mailman/listinfo/pytest-dev
