Hi iilya, This is really great, I've found writing tests in Elixir to be quite pleasant whereas when I use eunit I spend a lot of time shouting at my screen. How would the current elixir test be integrated in? Would we rewrite them to use your new adapter setup so we can test again the various interfaces?
I took a quick look at the code and it looks really good, my own concern is maintaining the three different adapters for testing. That could be a lot of work. From what I remember we are looking at removing support for the backdoor ports (*5986), so maybe we can remove the need to maintain that one. Otherwise I'm +1 on this. Cheers Garren On Wed, May 22, 2019 at 10:16 PM Joan Touzet <woh...@apache.org> wrote: > Hi Ilya, thanks for starting this thread. Comments inline. > > On 2019-05-22 14:42, Ilya Khlopotov wrote: > > The eunit testing framework is very hard to maintain. In particular, it > has the following problems: > > - the process structure is designed in such a way that failure in setup > or teardown of one test affects the execution environment of subsequent > tests. Which makes it really hard to locate the place where the problem is > coming from. > > I've personally experienced this a lot when reviewing failed logfiles, > trying to find the *first* failure where things go wrong. It's a huge > problem. > > > - inline test in the same module as the functions it tests might be > skipped > > - incorrect usage of ?assert vs ?_assert is not detectable since it > makes tests pass > > - there is a weird (and hard to debug) interaction when used in > combination with meck > > - https://github.com/eproxus/meck/issues/133#issuecomment-113189678 > > - https://github.com/eproxus/meck/issues/61 > > - meck:unload() must be used instead of meck:unload(Module) > > Eep! I wasn't aware of this one. That's ugly. > > > - teardown is not always run, which affects all subsequent tests > > Have first-hand experienced this one too. > > > - grouping of tests is tricky > > - it is hard to group tests so individual tests have meaningful > descriptions > > > > We believe that with ExUnit we wouldn't have these problems: > > Who's "we"? > I think I am one of the "we", I've chatted to iilya a few times around finding a better way to write and run tests in CouchDB. > > > - on_exit function is reliable in ExUnit > > - it is easy to group tests using `describe` directive > > - code-generation is trivial, which makes it is possible to generate > tests from formal spec (if/when we have one) > > Can you address the timeout question w.r.t. EUnit that I raised > elsewhere for cross-platform compatibility testing? I know that > Peng ran into the same issues I did here and was looking into extending > timeouts. > > Many of our tests suffer from failures where CI resources are slow and > simply fail due to taking longer than expected. Does ExUnit have any > additional support here? > > A suggestion was made (by Jay Doane, I believe, on IRC) that perhaps we > simply remove all timeout==failure logic (somehow?) and consider a > timeout a hung test run, which would eventually fail the entire suite. > This would ultimately lead to better deterministic testing, but we'd > probably uncover quite a few bugs in the process (esp. against CouchDB > <= 4.0). > > > > > Here are a few examples: > > > > # Test adapters to test different interfaces using same test suite > > This is neat. I'd like someone else to comment whether this the approach > you define will handle the polymorphic interfaces gracefully, or if the > effort to parametrise/DRY out the tests will be more difficulty than > simply maintaining 4 sets of tests. > > > > # Using same test suite to compare new implementation of the same > interface with the old one > > > > Imagine that we are doing a major rewrite of a module which would > implement the same interface. > > *tries to imagine such a 'hypothetical' rewrite* :) > > > How do we compare both implementations return the same results for the > same input? > > It is easy in Elixir, here is a sketch: > > Sounds interesting. I'd again like an analysis (from someone else) as to > how straightforward this would be to implement. > > -Joan > >
