NightOwl888 commented on issue #306: URL: https://github.com/apache/lucenenet/issues/306#issuecomment-722408852
The Lucene [test framework](https://www.nuget.org/packages/Lucene.Net.TestFramework) is unique in that it is a component that is meant for use by end users as well as our own tests. I did some research, and I cannot find another example of this on the .NET platform - it is unprecedented. In Java, the test framework uses a custom runner called [Randomized Testing](https://github.com/randomizedtesting/randomizedtesting) that is built on top of JUnit using its extensibility. The framework is inheritance-based, uses its own random seed functionality, and has its own attributes it needs to scan for during the test life cycle. Putting in enough functionality to actually debug the random tests (by outputting the random seed that was used during a failure so it can be plugged back in to debug) is a problem that still [hasn't been completely solved yet](https://github.com/apache/lucenenet/issues/288). I [asked the NUnit team](https://github.com/nunit/nunit/issues/3405), and they don't currently support the extensibility points to build your own test runner. So, in the spirit of not re-inventing the wheel we are trying to make do with only NUnit's features out of the box. Not all of the features of the test framework are supported this way (such as running the tests in a random order), but I believe we have enough of them to get by without our own test runner. NUnit is the only framework that comes close to being able to support an inheritance-based framework without a ton of research and tradeoffs. In case you haven't been following along for the past few years, @conniey worked on converting to xUnit for several months back in 2016-2017 and ultimately came to the conclusion that it wasn't possible. But I believe she was trying to take advantage of xUnit's parallel features. In 2019, we tried converting the test framework to both [xUnit](https://github.com/apache/lucenenet/tree/6ea55d1bcacd9322f10b696e3eee4462905a1479/src/Lucene.Net.Tests.TestFramework.xUnit) and [MSTest](https://github.com/apache/lucenenet/tree/6ea55d1bcacd9322f10b696e3eee4462905a1479/src/Lucene.Net.Tests.TestFramework.MSTest) in addition to NUnit. MSTest was completely impossible to support due to the fact that it doesn't scan for its own attributes in base classes. Maybe that has changed, but it wasn't possible in 2019. xUnit forces you to inject state into the constructor in order to get any control over what would ordinarily be `[BeforeClass]` and `[AfterClass]` attributes. This puts an unnecessary burden on end users who then have to pass your state object in order to inherit your base class. I did determine that it may be possible to make our tests run with xUnit by turning off its parallel feature and it may even be possible to support parallel testing by refactoring several of Lucene's classes to inject state rather than relying on static variables. However, I am not sure if all of the test framework features we need could be supported. xUnit wasn't designed with an inheritance-based test framework that is meant for 3rd parties in mind, and you have to manually do a lot of the things that are included right out of the box with NUnit, such as scanning for custom-made attributes. Ultimately, xUnit will be a lot more work to support than NUnit, and the fact that xUnit doesn't support an inheritance-based model that "just works" without forcing end users to override your constructor makes it less desirable than NUnit. So, after over a year of research and trial and err, we determined that: 1. For the short term at least through to the 4.8.0 release, NUnit is our only viable choice of the top 3 test frameworks 2. For the long term, it would be better to [port Randomized Testing](https://github.com/apache/lucenenet/issues/288) so we can fully support debugging random tests, but we would have to work closely with the NUnit team to ensure there is enough extensibility to support it ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
