Closing the loop, a change was checked into master that reduces the
Elixir suite to run against an n=1 CouchDB cluster, permanently. This
should help significantly for now.
Any remaining Elixir test failures should have issues opened on them.
On 2019-12-14 12:29 p.m., Garren Smith wrote:
I’m a bit nervous to remove the full elixir test suite. The only partition
tests are in the elixir suite. So we would have no test coverage for any of
that if we remove this. Given that we want to make this the best CouchDB
release ever it feels counter-intuitive to them be removing tests.
I also think there is a few JS tests we have disabled because they are
covered in Elixir so we would need to re-enable those in the same PR that
we remove the elixir tests.
Cheers
Garren
On Sat, Dec 14, 2019 at 2:51 AM Jay Doane <jaydo...@apache.org> wrote:
On Thu, Dec 12, 2019 at 11:27 PM Joan Touzet <woh...@apache.org> wrote:
Jay, consider this a +1 to remove those if you can get to them before I
do.
Joan, I don't think I was clear, but when I said
I'm +1 to removing these tests temporarily.
by "these tests" I actually meant all the elixir tests, not just the two I
mentioned. I don't think we'd get very far by just disabling those two, as
there are many others that fail intermittently as well. I'm +1 to remove
them, and here's a PR in case someone wants it to happen sooner rather than
later:
https://github.com/apache/couchdb/pull/2362
Thanks,
Jay
If this makes the Elixir tests useful again, then it's a good compromise.
-Joan
On 2019-12-13 2:18 a.m., Jay Doane wrote:
I've been trying to merge into master for the last 24+ hours, and have
restarted the build about 10 times so far, but it has yet to pass.
I kept track of failures, and these came up numerous times:
1) test GET /dbname/_design_docs (PartitionDDocTest)
test/elixir/test/partition_ddoc_test.exs:160
** (RuntimeError) timed out after 10294 ms
code: retry_until(
stacktrace:
(couchdbtest) test/elixir/lib/couch/db_test.ex:301:
Couch.DBTest.retry_until/4
test/elixir/test/partition_ddoc_test.exs:163: (test)
1) test All Docs tests (AllDocsTest)
test/elixir/test/all_docs_test.exs:15
** (RuntimeError) timed out after 30097 ms
code: retry_until(fn ->
stacktrace:
(couchdbtest) test/elixir/lib/couch/db_test.ex:301:
Couch.DBTest.retry_until/4
test/elixir/test/all_docs_test.exs:75: (test)
which both seem to be timing out.
I'm +1 to removing these tests temporarily.
Jay
On Thu, Dec 12, 2019 at 1:39 PM Joan Touzet <woh...@apache.org> wrote:
Hi again,
As I've been looking more closely at the CI suite for the Jenkins
transition, I've noticed that our Elixir test cases are actually the
most likely to fail. In 6 consecutive CI runs, 5 runs failed due to
failures in the Elixir suite. (The 6th failed due to a JS test
failure.)
We started the Elixir effort to retire the JS suite. We reached a
decision some months ago to put it into `make check` so that people
would pay attention to its output, and work to fix those tests,
accelerating our chances to get rid of the JS suite.
Unfortunately, that's not materialised. Our Elixir test porters seem
to
have stopped their work for a while now, and no one is systematically
addressing the failures in that suite. I've also heard other
developers
mention (via IRC) that some of the test cases hold invalid assumptions
about how CouchDB works, especially with the Erlang-based clustering
code. It sounds to me like the effort needs a full code review.
With 3.0 around the corner, I want people to be able to trust the
output
of `make check` when downloading the tarball. If there is no
objection,
when I merge the Erlang version / CI changes on Monday, I will also
comment out the call to `make elixir` as part of make check.
When the Elixir porting team is more confident in the reliability and
completeness of their work, and we can successfully retire the JS
suite,
we can reconsider.
-Joan "really wanting to see green, but only seeing red" Touzet
