Bug#942688: NMU for elixir-lang
> >> As elixir-lang is going to be AUTORM if nobody takes care of it, I > >> already added a bit of code in debian/rules to make it retry unit tests > >> if they fail. It will retry 3 times, meaning that if it used to fail 10% > >> of the time, making it retry 3 times makes the unit tests fail 0.01% of > >> the times, which is IMO acceptable. The recent failures on some arch > >> shows that my debian/rules hack works well! :) > > > > Please don't do that sort of thing. It makes things gratuitously painful > > for people like me doing QA. > > > > Either a unit test is to be trusted or it's not (by "trusted" I mean > > here that a failure in the test means the package is unsuitable for > > release). > > > > If the test is trusted but it fails, accept that it fails and do not > > let the package to propagate to testing. > > > > If it's not trusted, there is no point in making it to fail with any > > probability. Not 10%, and not 0.01%. Just disable the test. > > > > Thanks. > > I don't agree with this view. Yes, this makes your work harder. But > running tests improve the quality of the package. Removing them is in no > way an improvement. My complain here is specifically about artificially reducing the probability of a random test to fail. Remember that tests only improve the quality of the package when you actually do something when they fail. If you artificially reduce the probability and do nothing about failing tests, there is no increase of quality at all. [ Please Cc me on any replies, I'm not monitoring this bug ] Thanks.
Bug#942688: NMU for elixir-lang
Hello Thomas I really appreciate your actions intended to keep the package in testing. However, the removed tests show that there are some problems on arm64 which we can't simply ignore. If we do that, there will be other bug reports, making us do something about it. I have to confess that I remove some tests from the package as well, but in 2 specific cases: * rebar3: we haven't still got rebar3 in Debian, so there's nothing to test. * backports: sometimes the tests from backported Elixir reveals some bugs[1] in Erlang from stable, so there are two solutions in the cases like that: 1) remove the corresponding tests or 2) backport Erlang along with Elixir. Removing tests is cheaper. Coming back to Elixir and the problems related to arm64. I believe we have to find a better solution. It’s not a big deal if the package is removed from testing right now. It will turn out in testing again 1) when we figure out the root of the problem and fix it; or 2) when I (or someone else) update the package to the newest version which (probably) will be sustainable to the problem. [1] https://sources.debian.org/src/elixir-lang/1.9.1.dfsg-1~bpo10+1/debian/patches/remove_runs_eval_and_version_commands_test.patch/ -- Evgeny
Bug#942688: NMU for elixir-lang
On 11/4/19 12:11 PM, Santiago Vila wrote: > On Mon, Nov 04, 2019 at 11:40:08AM +0100, Thomas Goirand wrote: >> Hi Evgeny, >> >> As elixir-lang is going to be AUTORM if nobody takes care of it, I >> already added a bit of code in debian/rules to make it retry unit tests >> if they fail. It will retry 3 times, meaning that if it used to fail 10% >> of the time, making it retry 3 times makes the unit tests fail 0.01% of >> the times, which is IMO acceptable. The recent failures on some arch >> shows that my debian/rules hack works well! :) > > Please don't do that sort of thing. It makes things gratuitously painful > for people like me doing QA. > > Either a unit test is to be trusted or it's not (by "trusted" I mean > here that a failure in the test means the package is unsuitable for > release). > > If the test is trusted but it fails, accept that it fails and do not > let the package to propagate to testing. > > If it's not trusted, there is no point in making it to fail with any > probability. Not 10%, and not 0.01%. Just disable the test. > > Thanks. I don't agree with this view. Yes, this makes your work harder. But running tests improve the quality of the package. Removing them is in no way an improvement. Thomas Goirand (zigo)
Bug#942688: NMU for elixir-lang
On Mon, Nov 04, 2019 at 11:40:08AM +0100, Thomas Goirand wrote: > Hi Evgeny, > > As elixir-lang is going to be AUTORM if nobody takes care of it, I > already added a bit of code in debian/rules to make it retry unit tests > if they fail. It will retry 3 times, meaning that if it used to fail 10% > of the time, making it retry 3 times makes the unit tests fail 0.01% of > the times, which is IMO acceptable. The recent failures on some arch > shows that my debian/rules hack works well! :) Please don't do that sort of thing. It makes things gratuitously painful for people like me doing QA. Either a unit test is to be trusted or it's not (by "trusted" I mean here that a failure in the test means the package is unsuitable for release). If the test is trusted but it fails, accept that it fails and do not let the package to propagate to testing. If it's not trusted, there is no point in making it to fail with any probability. Not 10%, and not 0.01%. Just disable the test. Thanks.
Bug#942688: NMU for elixir-lang
Hi Evgeny,
As elixir-lang is going to be AUTORM if nobody takes care of it, I
already added a bit of code in debian/rules to make it retry unit tests
if they fail. It will retry 3 times, meaning that if it used to fail 10%
of the time, making it retry 3 times makes the unit tests fail 0.01% of
the times, which is IMO acceptable. The recent failures on some arch
shows that my debian/rules hack works well! :)
Now, I noticed that the build reliably fails on arm64 and mips64el:
https://buildd.debian.org/status/fetch.php?pkg=elixir-lang=arm64=1.9.1.dfsg-1.2=1572476904=0
https://buildd.debian.org/status/fetch.php?pkg=elixir-lang=mips64el=1.9.1.dfsg-1.2=1572491696=0
Therefore, I've removed the failing test.
Attached to this email, you'll find the debdiff for the change between
version 1.9.1.dfsg-1.1 and 1.9.1.dfsg-1.3.
Please let me know if you want to revert these upload. I can always do
that. The reason why I've done a non-delayed NMU is because otherwise,
it would be AUTORM, but of course, everything can be reverted if you
feel like it. Anyways, I'm always available for anything related to
these uploads, so just let me know.
Cheers,
Thomas Goirand (zigo)
diff -Nru elixir-lang-1.9.1.dfsg/debian/changelog
elixir-lang-1.9.1.dfsg/debian/changelog
--- elixir-lang-1.9.1.dfsg/debian/changelog 2019-10-13 08:39:24.0
+0200
+++ elixir-lang-1.9.1.dfsg/debian/changelog 2019-11-04 11:27:07.0
+0100
@@ -1,3 +1,18 @@
+elixir-lang (1.9.1.dfsg-1.3) unstable; urgency=medium
+
+ * Non-maintainer upload.
+ * Add remove-test-executes-rpc-instructions.patch, as this test reliably
+fails on arm64 and mips64el.
+
+ -- Thomas Goirand Mon, 04 Nov 2019 11:27:07 +0100
+
+elixir-lang (1.9.1.dfsg-1.2) unstable; urgency=medium
+
+ * Non-maintainer upload.
+ * Re-attempt unit testing (up to 3 times) if failed (Closes: #942688).
+
+ -- Thomas Goirand Wed, 30 Oct 2019 12:23:33 +0100
+
elixir-lang (1.9.1.dfsg-1.1) unstable; urgency=medium
* Non-maintainer upload.
diff -Nru
elixir-lang-1.9.1.dfsg/debian/patches/remove-test-executes-rpc-instructions.patch
elixir-lang-1.9.1.dfsg/debian/patches/remove-test-executes-rpc-instructions.patch
---
elixir-lang-1.9.1.dfsg/debian/patches/remove-test-executes-rpc-instructions.patch
1970-01-01 01:00:00.0 +0100
+++
elixir-lang-1.9.1.dfsg/debian/patches/remove-test-executes-rpc-instructions.patch
2019-11-04 11:26:42.0 +0100
@@ -0,0 +1,36 @@
+Description: Removes test "executes rpc instructions"
+ This test fails reliably on arm64 and misp64el.
+Author: Thomas Goirand
+Forwarded: no
+Last-Update: 2019-11-04
+
+--- elixir-lang-1.9.1.dfsg.orig/lib/mix/test/mix/tasks/release_test.exs
elixir-lang-1.9.1.dfsg/lib/mix/test/mix/tasks/release_test.exs
+@@ -268,27 +268,6 @@ defmodule Mix.Tasks.ReleaseTest do
+ end)
+ end
+
+- @tag :epmd
+- test "executes rpc instructions" do
+-in_fixture("release_test", fn ->
+- config = [releases: [permanent1: [include_erts: false]]]
+-
+- Mix.Project.in_project(:release_test, ".", config, fn _ ->
+-root = Path.absname("_build/dev/rel/permanent1")
+-Mix.Task.run("release")
+-script = Path.join(root, "bin/permanent1")
+-
+-open_port(script, ['start'])
+-wait_until_decoded(Path.join(root, "RELEASE_BOOTED"))
+-assert System.cmd(script, ["rpc", "ReleaseTest.hello_world"]) ==
{"hello world\n", 0}
+-assert System.cmd(script, ["stop"]) == {"", 0}
+-
+-assert {pid, 0} = System.cmd(script, ["pid"])
+-assert pid != "\n"
+- end)
+-end)
+- end
+-
+ test "runs eval and version commands" do
+ # In some Windows setups (mostly with Docker), `System.cmd/3` fails
because
+ # the path to the command/executable and one or more arguments contain
spaces.
diff -Nru elixir-lang-1.9.1.dfsg/debian/patches/series
elixir-lang-1.9.1.dfsg/debian/patches/series
--- elixir-lang-1.9.1.dfsg/debian/patches/series2019-10-13
08:39:24.0 +0200
+++ elixir-lang-1.9.1.dfsg/debian/patches/series2019-11-04
11:25:44.0 +0100
@@ -1,3 +1,4 @@
remove-utf8-warning.patch
remove_rebar3_related_tests.patch
use_rebar_instead_of_rebar3_in_tests.patch
+remove-test-executes-rpc-instructions.patch
diff -Nru elixir-lang-1.9.1.dfsg/debian/rules
elixir-lang-1.9.1.dfsg/debian/rules
--- elixir-lang-1.9.1.dfsg/debian/rules 2019-10-13 08:39:24.0 +0200
+++ elixir-lang-1.9.1.dfsg/debian/rules 2019-10-30 22:22:46.0 +0100
@@ -8,8 +8,18 @@
%:
dh $@
+
override_dh_auto_test: .home
- rc=0; HOME=$(CURDIR)/.home dh_auto_test || rc=$$?; epmd -kill; exit $$rc
+ CNT=3 ; rc=1 ; while [ $$CNT != 0 ] && [ $$rc != 0 ] ; do \
+ if HOME=$(CURDIR)/.home dh_auto_test ; then \
+ rc=0 ; \
+ else \
+ rc=1 ; \
+ epmd -kill ; \
+ fi ; \
+ CNT=$$(( $$CNT - 1)) ; \
+ done ; \
