Howdy folks!
The testing of CouchDB is something that has seen focus and improvements
for the last several years, for instance migrating the etap suite to eunit,
and updating the JS suite to run against clusters in 2.x. There's still
improvements to be made, and that was one of the topics of the CouchDB dev
summit early in the year [1].
Before we go further, I want to clarify some nomenclature. I'm by no means
going to try and define unit testing vs integration testing vs quantum
phase shift testing, but instead I want to focus on the distinction of
where the testing takes place. Fundamentally, we have two places we test
CouchDB: 1) at the Erlang VM level where we conduct assertions against
module functions or process states; 2) at the HTTP level where we test the
behavior of CouchDB at the user level API. This post focuses entirely on
the latter; that's not to say the former doesn't also merit attention, just
that the two are different enough that we can focus on them in isolation.
So with that, let's chat about the current HTTP test suite in CouchDB. This
is the "JS suite" I referred to above, which is a custom built test suite
written in Javascript and executed in the aging SpiderMonkey. The JS suite
has put in work for years, but it's showing it's age, and is a bit awkward
to work with and improve. However, I think the biggest issue with the JS
suite is that it's utilized far less than it should be, and folks seem to
avoid extending it or adding additional tests to it. There's been
discussion for years about replacing said suite, but the discussions
invariably got blocked on the bike shed of whether to rewrite the suite in
Javascript or Python. This thread provides a third option, with code!
I started hacking on a replacement for the JS suite, this time written in
Elixir. Overall I'm quite impressed with how it's come along, and have some
good examples to show. This is basically an Elixir app that has an HTTP
client and then runs a series of tests that conduct tests against the
CouchDB HTTP API and make assertions therein.
You can find the current code in [2], and a comparison of the changes in
[3]. The core HTTP client is only a handful of lines of codes and works
quite well [4]. The utility functions used across all tests are located in
[5], and the tests themselves are in [6]. The existing test modules have a
1:1 correspondence with the associated JS suite test modules, and in
general are as direct of a port as possible.
The test modules ported in their entirety or most of the way are:
* all_docs.js
* basics.js
* config.js
* reduce.js
* rewrite.js
* uuids.js
* view_collation.js
Paul has dove in and is responsible for a few of those test modules and
he's almost completed porting the replication.js suite as well. We started
with the hard ones first, so for the most part the rest of the ports should
be fairly smooth sailing.
Here's an example of a very basic test:
```erlang
defmodule WelcomeTest do
use CouchTestCase
test "Welcome endpoint" do
assert Couch.get("/").body["couchdb"] == "Welcome", "Should say welcome"
end
end
```
As you can see, the `Couch` client is very simple HTTP client with
easy HTTP verb based methods. Let's look at a more complicated test
for asserting we can create documents in a database:
```erlang
@tag :with_db
test "Create a document and save it to the database", context do
resp = Couch.post("/#{context[:db_name]}", [body: %{:_id => "0",
:a => 1, :b => 1}])
assert resp.status_code == 201, "Should be 201 created"
assert resp.body["id"], "Id should be present"
assert resp.body["rev"], "Rev should be present"
resp2 = Couch.get("/#{context[:db_name]}/#{resp.body["id"]}")
assert resp2.body["_id"] == resp.body["id"], "Ids should match"
assert resp2.body["_rev"] == resp.body["rev"], "Revs should match"
end
```
This is fairly straightforward code to POST a new doc, make assertions
on the response, and then fetch the doc to make sure everything
matches up. What I really wanted to highlight here is the `@tag
:with_db` decorator. We can easily add custom "tags" to the tests to
simplify setup and teardown. That `:with_db` tag does two things, it
dynamically generates a random database name, and then takes care of
setup/teardown for creating and deleting said database for that
particular test. This is really useful and has been very nice to work
with so far. We also have tag functionality in place for executing a
test with a particular set of config options:
```erlang
@tag config: [
{"uuids", "algorithm", "utc_random"}
]
test "utc_random uuids are roughly random" do
resp = Couch.get("/_uuids", query: %{:count => 1000})
assert resp.status_code == 200
uuids = resp.body["uuids"]
assert String.length(Enum.at(uuids, 1)) == 32
# Assert no collisions
assert length(Enum.uniq(uuids)) == length(uuids)
# Assert rough ordering of UUIDs
u1 = String.slice(Enum.at(uuids, 1), 0..13)
u2 = String.slice(Enum.at(uuids, -1), 0..13)
assert u1 < u2
end
```
The tag system really simplifies a lot of the standard auxiliary
actions needed to conduct tests.
To test out the suite, you'll need to spin up the dev server in one window with:
```
./dev/run --admin=adm:pass
```
and then in another window go into the relevant CouchDB src directory and run:
```
cd ~/src/couchdb/elixir_suite/
mix deps.get
mix test --trace
```
The `--trace` flag makes the nice line item output per test, which I
greatly prefer over a slew of periods. You can run an individual test
with `mix test --trace tests/basics_test.exs`. I've pasted the output
from running the basics suite at the bottom of this email so you can
see what the real output looks like.
Overall I'm quite impressed with the toolkit we've been able to put
together in a short amount of time, and I propose we migrate fully to
this test suite by porting all remaining JS suite tests and then
removing the JS suite entirely. Given we've already ported most of the
"hard suites", I think a full port is reasonable to do and just
requires some leg work. Again, I'm impressed with how simple the
tooling here is and how quickly we've been able to run with things,
turns out the Elixir dev experience is actually quite nice! I hope
others have similar opinions after diving in! Let me know what you
think.
-Russell
[1] https://github.com/janl/couchdb-next/issues/39
[2] https://github.com/apache/couchdb/tree/elixir-suite
[3] https://github.com/apache/couchdb/compare/elixir-suite
[4]
https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/lib/couch.ex
[5]
https://github.com/apache/couchdb/blob/elixir-suite/elixir_suite/test/test_helper.exs
[6] https://github.com/apache/couchdb/tree/elixir-suite/elixir_suite/test
vagrant@contrib-jessie:~/src/couchdb/elixir_suite$ mix test --trace
test/basics_test.exs
Excluding tags: [pending: true]
BasicsTest
* test Session contains adm context (66.8ms)
* test Creating a new DB with slashes should return Location header
(COUCHDB-411) (85.8ms)
* test oops, the doc id got lost in code nirwana (82.1ms)
* test Welcome endpoint (7.6ms)
* test POST doc with an _id field isn't overwritten by uuid (102.7ms)
* test On restart, a request for creating an already existing db can
not override (skipped)
* test Creating a new DB should return location header (118.7ms)
* test _bulk_docs POST error when body not an object (95.0ms)
* test Empty database should have zero docs (161.0ms)
* test _all_docs POST error when multi-get is not a {'key': [...]}
structure (104.3ms)
* test Regression test for COUCHDB-954 (skipped)
* test DELETE'ing a non-existent doc should 404 (100.0ms)
* test Revs info status is good (127.3ms)
* test PUT on existing DB should return 412 instead of 500 (97.6ms)
* test Database should be in _all_dbs (117.7ms)
* test Check for invalid document members (122.4ms)
* test Can create several documents (213.0ms)
* test Make sure you can do a seq=true option (99.1ms)
* test PUT doc has a Location header (skipped)
* test Create a document and save it to the database (116.3ms)
* test Created database has appropriate db info name (99.7ms)
* test PUT error when body not an object (89.5ms)
* test Simple map functions (473.0ms)
* test POST doc response has a Location header (117.1ms)
CouchTestCase
Finished in 3.3 seconds
24 tests, 0 failures, 3 skipped
Randomized with seed 936284
