Hi everyone, I am not exactly sure how to proceed with an RFC https://github.com/apache/couchdb-documentation/pull/415 about using ExUnit to write unit tests. I am using information from https://couchdb.apache.org/bylaws.html#rfc - introduction of a new testing framework is a technical decission and doesn't need an RFC - on the other hand ExUnit makes Elixir dependency mandatory which is something that need to be agreed upon - also it seems that this thread is not in correct one to discuss an RFC since it doesn't include [DISCUSSION] prefix.
Please advice how to classify introduction of Elixir based testing framework into unit testing. Best regards, iilyak On 2019/05/22 18:42:03, Ilya Khlopotov <iil...@apache.org> wrote: > Hi everyone, > > With the upgrade of supported Erlang version and introduction of Elixir into > our integration test suite we have an opportunity to replace currently used > eunit (for new tests only) with Elixir based ExUnit. > The eunit testing framework is very hard to maintain. In particular, it has > the following problems: > - the process structure is designed in such a way that failure in setup or > teardown of one test affects the execution environment of subsequent tests. > Which makes it really hard to locate the place where the problem is coming > from. > - inline test in the same module as the functions it tests might be skipped > - incorrect usage of ?assert vs ?_assert is not detectable since it makes > tests pass > - there is a weird (and hard to debug) interaction when used in combination > with meck > - https://github.com/eproxus/meck/issues/133#issuecomment-113189678 > - https://github.com/eproxus/meck/issues/61 > - meck:unload() must be used instead of meck:unload(Module) > - teardown is not always run, which affects all subsequent tests > - grouping of tests is tricky > - it is hard to group tests so individual tests have meaningful descriptions > > We believe that with ExUnit we wouldn't have these problems: > - on_exit function is reliable in ExUnit > - it is easy to group tests using `describe` directive > - code-generation is trivial, which makes it is possible to generate tests > from formal spec (if/when we have one) > > Here are a few examples: > > # Test adapters to test different interfaces using same test suite > > CouchDB has four different interfaces which we need to test. These are: > - chttpd > - couch_httpd > - fabric > - couch_db > > There is a bunch of operations which are very similar. The only differences > between them are: > - setup/teardown needs different set of applications > - we need to use different modules to test the operations > > This problem is solved by using testing adapter. We would define a common > protocol, which we would use for testing. > Then we implement this protocol for every interface we want to use. > > ``` > defmodule Couch.Test.CRUD do > use ExUnit.Case > alias Couch.Test.Adapter > alias Couch.Test.Utils, as: Utils > > alias Couch.Test.Setup > > require Record > > test_groups = [ > "using Clustered API": Adapter.Clustered, > "using Backdoor API": Adapter.Backdoor, > "using Fabric API": Adapter.Fabric, > ] > > for {describe, adapter} <- test_groups do > describe "Database CRUD #{describe}" do > @describetag setup: %Setup{} > |> Setup.Start.new([:chttpd]) > |> Setup.Adapter.new(adapter) > |> Setup.Admin.new(user: "adm", password: "pass") > |> Setup.Login.new(user: "adm", password: "pass") > test "Create", %{setup: setup} do > db_name = Utils.random_name("db") > setup_ctx = setup |> Setup.run() > assert {:ok, resp} = Adapter.create_db(Setup.get(setup_ctx, > :adapter), db_name) > assert resp.body["ok"] > end > end > end > end > ``` > > # Using same test suite to compare new implementation of the same interface > with the old one > > Imagine that we are doing a major rewrite of a module which would implement > the same interface. > How do we compare both implementations return the same results for the same > input? > It is easy in Elixir, here is a sketch: > ``` > defmodule Couch.Test.Fabric.Rewrite do > use ExUnit.Case > alias Couch.Test.Utils, as: Utils > > # we cannot use defrecord here because we need to construct > # record at compile time > admin_ctx = {:user_ctx, Utils.erlang_record( > :user_ctx, "couch/include/couch_db.hrl", roles: ["_admin"])} > > test_cases = [ > {"create database": {create_db, [:db_name, []]}}, > {"create database as admin": {create_db, [:db_name, [admin_ctx]]}} > ] > module_a = :fabric > module_b = :fabric3 > > describe "Test compatibility of '#{module_a}' with '#{module_b}'" do > for {description, {function, args}} <- test_cases do > test "#{description}" do > result_a = unquote(module_a).unquote(function)(unquote_splicing(args)) > result_b = unquote(module_b).unquote(function)(unquote_splicing(args)) > assert result_a == result_b > end > end > end > > end > ``` > As a result we would get following tests > ``` > Couch.Test.Fabric.Rewrite > * test Test compatibility of 'fabric' with 'fabric3' create database > (0.01ms) > * test Test compatibility of 'fabric' with 'fabric3' create database as > admin (0.01ms) > ``` > > The prototype of integration is in this draft PR > https://github.com/apache/couchdb/pull/2036. I am planing to write formal RFC > after first round of discussions on ML. > > Best regards, > iilyak >
