Hi,

Starting formal vote on RFC according to bylaws. RFC 
https://github.com/apache/couchdb-documentation/pull/415

Best regards,
iilyak

On 2019/05/22 18:42:03, Ilya Khlopotov <iil...@apache.org> wrote: 
> Hi everyone,
> 
> With the upgrade of supported Erlang version and introduction of Elixir into 
> our integration test suite we have an opportunity to replace currently used 
> eunit (for new tests only) with Elixir based ExUnit. 
> The eunit testing framework is very hard to maintain. In particular, it has 
> the following problems:
> - the process structure is designed in such a way that failure in setup or 
> teardown of one test affects the execution environment of subsequent tests. 
> Which makes it really hard to locate the place where the problem is coming 
> from.
> - inline test in the same module as the functions it tests might be skipped
> - incorrect usage of ?assert vs ?_assert is not detectable since it makes 
> tests pass 
> - there is a weird (and hard to debug) interaction when used in combination 
> with meck 
>    - https://github.com/eproxus/meck/issues/133#issuecomment-113189678
>    - https://github.com/eproxus/meck/issues/61
>    - meck:unload() must be used instead of meck:unload(Module)
> - teardown is not always run, which affects all subsequent tests
> - grouping of tests is tricky
> - it is hard to group tests so individual tests have meaningful descriptions
> 
> We believe that with ExUnit we wouldn't have these problems:
> - on_exit function is reliable in ExUnit
> - it is easy to group tests using `describe` directive
> - code-generation is trivial, which makes it is possible to generate tests 
> from formal spec (if/when we have one)
> 
> Here are a few examples:
> 
> # Test adapters to test different interfaces using same test suite
> 
> CouchDB has four different interfaces which we need to test. These are:
> - chttpd
> - couch_httpd
> - fabric
> - couch_db
> 
> There is a bunch of operations which are very similar. The only differences 
> between them are:
> - setup/teardown needs different set of applications
> - we need to use different modules to test the operations
> 
> This problem is solved by using testing adapter. We would define a common 
> protocol, which we would use for testing.
> Then we implement this protocol for every interface we want to use.
> 
> ```
> defmodule Couch.Test.CRUD do
>   use ExUnit.Case
>   alias Couch.Test.Adapter
>   alias Couch.Test.Utils, as: Utils
> 
>   alias Couch.Test.Setup
> 
>   require Record
> 
>   test_groups = [
>     "using Clustered API": Adapter.Clustered,
>     "using Backdoor API": Adapter.Backdoor,
>     "using Fabric API": Adapter.Fabric,
>   ]
> 
>   for {describe, adapter} <- test_groups do
>     describe "Database CRUD #{describe}" do
>       @describetag setup: %Setup{}
>         |> Setup.Start.new([:chttpd])
>         |> Setup.Adapter.new(adapter)
>         |> Setup.Admin.new(user: "adm", password: "pass")
>         |> Setup.Login.new(user: "adm", password: "pass")
>       test "Create", %{setup: setup} do
>         db_name = Utils.random_name("db")
>         setup_ctx = setup |> Setup.run()
>         assert {:ok, resp} = Adapter.create_db(Setup.get(setup_ctx, 
> :adapter), db_name)
>         assert resp.body["ok"]
>       end
>     end
>   end
> end
> ```
> 
> # Using same test suite to compare new implementation of the same interface 
> with the old one
> 
> Imagine that we are doing a major rewrite of a module which would implement 
> the same interface.
> How do we compare both implementations return the same results for the same 
> input?
> It is easy in Elixir, here is a sketch:
> ```
> defmodule Couch.Test.Fabric.Rewrite do
>   use ExUnit.Case
>   alias Couch.Test.Utils, as: Utils
> 
>   # we cannot use defrecord here because we need to construct
>   # record at compile time
>   admin_ctx = {:user_ctx, Utils.erlang_record(
>     :user_ctx, "couch/include/couch_db.hrl", roles: ["_admin"])}
> 
>   test_cases = [
>     {"create database": {create_db, [:db_name, []]}},
>     {"create database as admin": {create_db, [:db_name, [admin_ctx]]}}
>   ]
>   module_a = :fabric
>   module_b = :fabric3
> 
>   describe "Test compatibility of '#{module_a}' with '#{module_b}'" do
>     for {description, {function, args}} <- test_cases do
>       test "#{description}" do
>         result_a = unquote(module_a).unquote(function)(unquote_splicing(args))
>         result_b = unquote(module_b).unquote(function)(unquote_splicing(args))
>         assert result_a == result_b
>       end
>     end
>   end
> 
> end
> ```
> As a result we would get following tests
> ```
> Couch.Test.Fabric.Rewrite
>   * test Test compatibility of 'fabric' with 'fabric3' create database 
> (0.01ms)
>   * test Test compatibility of 'fabric' with 'fabric3' create database as 
> admin (0.01ms)
> ```
> 
> The prototype of integration is in this draft PR 
> https://github.com/apache/couchdb/pull/2036. I am planing to write formal RFC 
> after first round of discussions on ML.
> 
> Best regards,
> iilyak
> 

Reply via email to