Re: [Pkg-javascript-devel] Bug#853035: node-liftoff: Non-determistically FTBFS due to unreliable timing in tests

[Paolo Greppi]

Hi Graham and thanks for bringing this up again.

Il 07/04/2018 18:13, Graham Inggs ha scritto:
> Hi
> 
> This situation seems to get worse with each new version of nodejs.
> 
> In Ubuntu, where autopkgtests are run for every release architecture,
> node-liftoff failed on armhf [1] and arm64 [2] since the upload of
> nodejs 8.9.3.
> 
> We resorted to increasing the timeout from 5000 to 1 in
> debian/rules and debian/tests/control to allow nodejs to migrate.
> 
> Chris, you wrote "...the test would still be unreliable whatever value
> you choose", can you propose something other than increasing the
> timeout?
> 
> Regards
> Graham
> 
> [1] http://autopkgtest.ubuntu.com/packages/node-liftoff/bionic/armhf
> [2] http://autopkgtest.ubuntu.com/packages/node-liftoff/bionic/arm64

Upstream's CI does not test above node.js 6. I have raised that here:
https://github.com/js-cli/js-liftoff/issues/96

Nonetheless it may be worth updating liftoff to 2.5.0; I am going to try that.

Two new packages are required, see:
https://lists.alioth.debian.org/pipermail/pkg-javascript-devel/2018-April/025746.html
Any help with sponsoring them is welcome.

I also need to update:
- node-findup-sync 0.4.3-1 -> 2.0.0
- node-micromatch 2.3.11-1 -> 3.0.4
- node-detect-file 0.1.0-1 -> 1.0.0
- node-resolve-dir 1.0.0-1 -> 1.0.1
- node-flagged-respawn -> 0.3.2-1 -> 1.0.0

The latter in particular sounds promising for the "Liftoff launch should 
respawn if process.argv has values from v8flags in it" test failures.
flagged-respawn also has a test suite itself which can be enabled since we now 
have node-v8flags in Debian.
Maybe we catch it there !

Expect more RFSs over the next few days ...

Paolo

-- 
Pkg-javascript-devel mailing list
Pkg-javascript-devel@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel

[Pkg-javascript-devel] Bug#853035: node-liftoff: Non-determistically FTBFS due to unreliable timing in tests

[Chris Lamb]

Hi Graham,

> > I'm pretty hardcore on tests being determinstic, so merely increasing
> > the wait time feels especially gross to me. :)
> 
> Would you please explain your objection to timeouts in tests?

Well, we should probably separate the cases of using /timing/ in tests as
part of, say, checking whether something is "fast enough" or an algorithm is
O(n) vs O(n^2) and timing that is essentially relying on the machine not
being under load or is otherwise slow.

I am sure you would agree the former is problematic and not even remotely
determinstic! The latter is borderline, but when you can't rely on test
failures being genuine test failures, you psychologically start to demote
them in your mind in terms of importance and whether you should care about
the,.

> I think it is practical to give up waiting for a response some point.
> Even Debian's buildds give up after 150 minutes.

Mm, as a sanity check for the integrity of the test/build platform, timing
out after X hours or so makes sense and I have no problem with this. It's
just the stuff on the order of Y seconds that is a little bit more difficult
to justify. :)

(Can this test really not reliably detect whether something was killed/
spawned...?)


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

-- 
Pkg-javascript-devel mailing list
Pkg-javascript-devel@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel

[Pkg-javascript-devel] Bug#853035: node-liftoff: Non-determistically FTBFS due to unreliable timing in tests

[Graham Inggs]

Hi Chris

On 7 April 2018 at 18:22, Chris Lamb  wrote:
> Well, just to be 100% clear, I don't propose increasing the timeout at
> all: I'm pretty hardcode on tests being determinstic so merely increasing
> the wait time feels especially gross to me. :)

Would you please explain your objection to timeouts in tests?

I think it is practical to give up waiting for a response some point.
Even Debian's buildds give up after 150 minutes.

Regards
Graham

-- 
Pkg-javascript-devel mailing list
Pkg-javascript-devel@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel

[Pkg-javascript-devel] Bug#853035: node-liftoff: Non-determistically FTBFS due to unreliable timing in tests

[Chris Lamb]

[Resending including 853...@bugs.debian.org in CC this time..]

Hi Graham,

Thanks for the followup.

> Chris, you wrote "...the test would still be unreliable whatever value
> you choose", can you propose something other than increasing the
> timeout?

Well, just to be 100% clear, I don't propose increasing the timeout at
all: I'm pretty hardcode on tests being determinstic so merely increasing
the wait time feels especially gross to me. :)

So, indeed, another solution is really required but alas I'm really not
familiar with this particular test nor this package in general so I would
not be able to comment with any authority.

Have you tried speaking to upstream?


Best wishes,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-

-- 
Pkg-javascript-devel mailing list
Pkg-javascript-devel@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel

[Pkg-javascript-devel] Bug#853035: node-liftoff: Non-determistically FTBFS due to unreliable timing in tests

[Graham Inggs]

Hi

This situation seems to get worse with each new version of nodejs.

In Ubuntu, where autopkgtests are run for every release architecture,
node-liftoff failed on armhf [1] and arm64 [2] since the upload of
nodejs 8.9.3.

We resorted to increasing the timeout from 5000 to 1 in
debian/rules and debian/tests/control to allow nodejs to migrate.

Chris, you wrote "...the test would still be unreliable whatever value
you choose", can you propose something other than increasing the
timeout?

Regards
Graham


[1] http://autopkgtest.ubuntu.com/packages/node-liftoff/bionic/armhf
[2] http://autopkgtest.ubuntu.com/packages/node-liftoff/bionic/arm64

-- 
Pkg-javascript-devel mailing list
Pkg-javascript-devel@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel

[Pkg-javascript-devel] Bug#853035: node-liftoff: Non-determistically FTBFS due to unreliable timing in tests

[Praveen Arimbrathodiyil]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

control: severity -1 important

> In the log you attached this line:

 > [0m Error: timeout of 5000ms exceeded

> looks different from what you report in the bug:

>   [0m Error: timeout of 1ms exceeded

> The test suite is run by upstream (and by us) with this command:

 > mocha -t 5000 -b -R spec test/index

 > this is where the timeout of 5 s comes from.

> Not sure if we should worry about this one.
> And above all: I'm clueless at why it should fail.

> Where did it fail ?
> What is special with your test environment ?

> Paolo

Chris,

Can you try to increase the timeout to say 1 or 2 and see
if it succeeds.

-BEGIN PGP SIGNATURE-

iQIzBAEBCAAdFiEEKnl0ri/BUtd4Z9pKzh+cZ0USwioFAliavBMACgkQzh+cZ0US
wiolzA//ftAwk30i6FjAPDV6xu1rZgjIM2ha+nIwVvEWQLcupH36V/pkgXHFF6rX
xcXUQy4xnKOYvpza2+D26nx9qw8BleYpjqxJaNVt/n5CVgfDFGw3ZYoiPhnftBuV
4Qj5DWdQ0RXSK929QBjMZeG9IZ//k0oHIOQjWkNjjdh7nz70zPOlEbkBkeJQP81y
julNTqi6xT4umAXc90tRlM0WuzZJtUD89yaEoFCmqWSSHGHvBVfS/L8CXGxAnDA3
7LKB1O9vFi3od7+bcHYdx16JNE2IEMGlB48xLWuw6rvgQBQS40fHM3sSD9vmur0y
fLneyAdqrbdCEAcXS557gLBOPeX/LoPs6ARPVMzTvsiWRvTpj5nzXivj2IwG3Wbi
v0Ib+doSmA4S637FX9Ztyj+7w0lmQiZi+IDth+GLJhCvFaLrBsg+Z49Cszco9OfD
qJfwt0cpiQR6I4sGk8V2tmT3Vrt7VAjxgb1iVxyHoDo7uACIvF7xZxEiRkjm8Vnb
6bsuLSU6nDgTs38HyJRJwIwXmNgLhM/FR7RhGi0p/y4FUAqyB029RsPMIyWTQ6JU
WgFhog2wbrErvZwIZvAr2KGppKD6hH7PEuAMzbh2WaexlBkHU5ZQmhmwMNWBYow0
eSm3UK8dneDmPEgWaObRrZoZ/bwnLAbW2NYD2fsSjui82ikV57o=
=zqes
-END PGP SIGNATURE-
-- 
Pkg-javascript-devel mailing list
Pkg-javascript-devel@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel

[Pkg-javascript-devel] Bug#853035: node-liftoff: Non-determistically FTBFS due to unreliable timing in tests

[Chris Lamb]

Source: node-liftoff
Version: 2.3.0-2
Severity: serious
Justification: fails to build from source
User: reproducible-bui...@lists.alioth.debian.org
Usertags: ftbfs
X-Debbugs-Cc: reproducible-b...@lists.alioth.debian.org

Dear Maintainer,

node-liftoff's testsuite appears to use method timing/benchmarking
in such a way that it will non-deterministically FTBFS:

  […]

  
    1) Liftoff launch should skip respawning if process.argv has no values 
from v8flags in it:
   Error: timeout of 5000ms exceeded
at null. (/usr/lib/nodejs/mocha/lib/runnable.js:139:19)
at Timer.listOnTimeout (timers.js:92:15)
  
  
  
  debian/rules:13: recipe for target 'override_dh_auto_test' failed
  make[1]: *** [override_dh_auto_test] Error 1
  make[1]: Leaving directory '«BUILDDIR»'
  debian/rules:8: recipe for target 'build' failed
  make: *** [build] Error 2
  dpkg-buildpackage: error: debian/rules build gave error exit status 2

  […]

The full build log is attached.


Regards,

-- 
  ,''`.
 : :'  : Chris Lamb
 `. `'`  la...@debian.org / chris-lamb.co.uk
   `-


node-liftoff.2.3.0-2.unstable.amd64.log.txt.gz
Description: Binary data
-- 
Pkg-javascript-devel mailing list
Pkg-javascript-devel@lists.alioth.debian.org
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-javascript-devel