Hi, I think having a simple `pytest-multiprocessing` or `pytest-concurrent` would be extremely useful and cover a lot of people's use cases and would strongly recommend to have that as a separate plugin from something bigger that does feature hooks and/or remote execution.. It seems though as such a package already exists, https://github.com/reverbc/pytest-concurrent, which offers --concmode and --concworkers instead.
Frederik On Thu, Oct 24, 2019 at 2:46 PM Bruno Oliveira <nicodde...@gmail.com> wrote: > Hi everyone, > > For some time now execnet has been in maintenance only mode, and even so > very few people are willing to maintain it; lately just myself and I’m not > a good choice given that I don’t know the codebase at all, plus I have tons > on my plate already. This poses a problem because often we have bug reports > or feature requests in pytest-xdist which would require fixing or > improving execnet, and are currently left without solution. > > Ronny and I have on occasion discussed the possibility of changing execnet‘s > backend , with the current contenders being multiprocessing for local > distribution and mitogen for remote distribution. > > I would like to write down some thoughts and gather opinions from the > mailing list members. > multiprocessing > > Aside from not needing to maintain execnet anymore, being able to use > multiprocessing would bring several benefits: > > - > > pytest -s would work just fine, because multiprocessing does not use > stdout/stderr for communication. This is a often requested feature but > which we can’t (with current expertise) implement on execnet. > - > > It would automatically support anything that can be picked to be > transmitted across the wire. Currently execnet does not support custom user > types and some builtins (frozen sets for example). > > Without execnet we would lose the ability of running the tests in > different interpreter versions, but I believe this has been supplanted by > tox in the ecosystem. > mitogen > > Using mitogen for remote execution would provide automatic bootstrap of > the Python environment, which is not currently supported by xdist‘s ssh > remote execution, greatly limiting its usefulness in real-time scenarios. > > This is Ronny’s idea and he can add more here if wanted. > pytest-xdist2? > > All the changes mentioned above would break backward compatibility *hard*, > because details of the execnet implementation are currently exposed to > users in the command-line: > > pytest -d --tx popen//python > > And in several hooks: > > def pytest_xdist_newgateway(gateway): > """ called on new raw gateway creation. """ > > (gateway is an execnet object) > > To incorporate the new backends, we would need to severely break both > command-line and existing hooks. Given the level of breakage this could > cause, just bumping the major version in this case doesn’t seem enough, it > would be a disservice to users given that probably nobody pins pytest-xdist > to <=2, and it would break the world with hard to understand error > messages once a first major release was released. > > Because of this, we have been discussing creating a new package, > pytest-xdist2 (other suggestions are welcome), without any backward > compatibility guarantees with pytest-xdist. > > pytest-xdist2 would, at first, only support local test distribution using > multiprocessing (-n flag), no new hooks, and no remote execution. I’ve > made a proof of concept here: > > https://github.com/pytest-dev/pytest-xdist/pull/479 > > So it definitely seems possible. One immediate benefit is that the proof > of concept above supports pytest -s already, and can transfer anything > that’s pickled over the wire. > > So I would like to know what people think of the above ideas, if they are > good/bad, and/or suggest alternatives that we are not seeing right now. > > Ronny, please feel free to add to this email and correct anything I got > wrong. > > Cheers, > Bruno > _______________________________________________ > pytest-dev mailing list > pytest-dev@python.org > https://mail.python.org/mailman/listinfo/pytest-dev >
_______________________________________________ pytest-dev mailing list pytest-dev@python.org https://mail.python.org/mailman/listinfo/pytest-dev
