Bug#1025234: prometheus: flaky autopkgtest (on 32 bit archs?)

2023-01-06 Thread Daniel Swarbrick 

On 07.01.23 12:40, Adrian Bunk wrote:

Does running the autopkgtests on 32-bit bring more benefits than hassle,
or should they be run only on 64-bit architectures?


As troublesome as the tests are on 32-bit, and as much as it would 
probably be simpler to just blanket disable them in d/rules, I have seen 
other dubious code land occasionally, which would fail on 32-bit.


On several occasions in the past, I have had to patch tests which 
attempted to read numeric values into an untyped int / uint, which would 
overflow on 32-bit. For this reason, I think we still need to keep 
testing on 32-bit, to keep the upstream developers on their toes ;-)


OpenPGP_signature
Description: OpenPGP digital signature

Bug#1025234: prometheus: flaky autopkgtest (on 32 bit archs?)

2023-01-06 Thread Adrian Bunk 
On Fri, Dec 02, 2022 at 05:51:20PM +1300, Daniel Swarbrick wrote:
> Hi Paul,
> 
> I have also noticed the fairly frequent failures of the memory-intensive
> tests on 32-bit, and I am doing my best to keep on top of them with t.Skip()
> patches where appropriate. Several of the tests result in the 4 GiB memory
> footprint threshold being exceeded.

4 GiB is the maximum address space limit, and it is typically only 
available on some architectures with 64-bit kernel (not on real
32-bit hardware).

2 GiB or 3 GiB are typical limits for available address space with 
32-bit kernels (on MIPS it's 2 GiB even with a 64-bit kernel).

> Prometheus itself is still usable on 32-bit, but obviously only up to a
> certain size. The upstream developers don't seem to consider 32-bit when
> writing unit tests, thus the regular addition of new tests which fail.

Does running the autopkgtests on 32-bit bring more benefits than hassle,
or should they be run only on 64-bit architectures?

> Daniel.

cu
Adrian

Bug#1025234: prometheus: flaky autopkgtest (on 32 bit archs?)

2022-12-01 Thread Daniel Swarbrick 

Hi Paul,

I have also noticed the fairly frequent failures of the memory-intensive 
tests on 32-bit, and I am doing my best to keep on top of them with 
t.Skip() patches where appropriate. Several of the tests result in the 4 
GiB memory footprint threshold being exceeded.


Prometheus itself is still usable on 32-bit, but obviously only up to a 
certain size. The upstream developers don't seem to consider 32-bit when 
writing unit tests, thus the regular addition of new tests which fail.


Daniel.



OpenPGP_signature
Description: OpenPGP digital signature

Bug#1025234: prometheus: flaky autopkgtest (on 32 bit archs?)

2022-12-01 Thread Paul Gevers 

Source: prometheus
Version: 2.33.5+ds1-2
Severity: serious
User: debian...@lists.debian.org
Usertags: flaky

Dear maintainer(s),

I looked at the results of the autopkgtest of your package (because it 
blocked glibc). I noticed that it regularly fails. At least on 32 bits 
architectures, I think the test (and maybe the general use of the 
package?) is nearing memory limititations? Maybe the failures on 64 bit 
architectures were due to non-available test dependencies (I didn't 
check all logs).


Because the unstable-to-testing migration software now blocks on
regressions in testing, flaky tests, i.e. tests that flip between
passing and failing without changes to the list of installed packages,
are causing people unrelated to your package to spend time on these
tests.

Don't hesitate to reach out if you need help and some more information
from our infrastructure.

Paul

https://ci.debian.net/data/autopkgtest/testing/armhf/p/prometheus/28799146/log.gz

FAIL
FAILgithub.com/prometheus/prometheus/promql 9.743s


--- FAIL: TestQueryHistogramFromBlocksWithCompaction (11.14s)
--- PASS: 
TestQueryHistogramFromBlocksWithCompaction/serial_blocks_with_only_histograms 
(1.23s)
--- PASS: 
TestQueryHistogramFromBlocksWithCompaction/serial_blocks_with_either_histograms_or_floats_in_a_block_and_not_both 
(2.57s)
--- PASS: 
TestQueryHistogramFromBlocksWithCompaction/serial_blocks_with_mix_of_histograms_and_float64 
(2.59s)
--- PASS: 
TestQueryHistogramFromBlocksWithCompaction/overlapping_blocks_with_only_histograms 
(1.68s)
--- PASS: 
TestQueryHistogramFromBlocksWithCompaction/overlapping_blocks_with_only_histograms_and_only_float_in_a_series 
(2.63s)
--- FAIL: 
TestQueryHistogramFromBlocksWithCompaction/overlapping_blocks_with_mix_of_histograms_and_float64 
(0.44s)

panic: mmap, size 134217728: cannot allocate memory [recovered]
panic: mmap, size 134217728: cannot allocate memory

goroutine 11328 [running]:
testing.tRunner.func1.2({0x88a108, 0x4c04030})
/usr/lib/go-1.19/src/testing/testing.go:1396 +0x27c
testing.tRunner.func1()
/usr/lib/go-1.19/src/testing/testing.go:1399 +0x3f4
panic({0x88a108, 0x4c04030})
/usr/lib/go-1.19/src/runtime/panic.go:884 +0x23c
github.com/prometheus/prometheus/tsdb.handleChunkWriteError({0xbe5844, 
0x4c04030})


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/head_append.go:1225
 +0x88
github.com/prometheus/prometheus/tsdb/chunks.(*ChunkDiskMapper).WriteChunk(0x4bba000, 
0x1, 0x0, 0xafc80, {0xbe9cd8, 0x4b9a6b0}, 0x93e6b8)


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/chunks/head_chunks.go:418
 +0x130
github.com/prometheus/prometheus/tsdb.(*memSeries).mmapCurrentHeadChunk(0x4baa120, 
0x4bba000)


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/head_append.go:1214
 +0xa0
github.com/prometheus/prometheus/tsdb.(*memSeries).cutNewHeadChunk(0x4baa120, 
0xdbba0, 0x1, 0x4bba000, 0x6ddd00)


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/head_append.go:1149
 +0x24
github.com/prometheus/prometheus/tsdb.(*memSeries).appendPreprocessor(0x4baa120, 
0xdbba0, 0x1, 0x4bba000, 0x6ddd00)


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/head_append.go:1101
 +0x19c
github.com/prometheus/prometheus/tsdb.(*memSeries).append(0x4baa120, 
0xdbba0, 0x3fe1b0583e7c7c6a, 0x2, 0x4bba000, 0x6ddd00)


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/head_append.go:983
 +0x4c
github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0x4be6140)

/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/head_append.go:876
 +0x1410
github.com/prometheus/prometheus/tsdb.CreateBlock({0x3eea780, 0xa, 0xa}, 
{0x3d12000, 0x72}, 0x0, {0xbe57e0, 0x105b3f8})


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/tsdbblockutil.go:63
 +0x570
github.com/prometheus/prometheus/tsdb.createBlock({0xbeb950, 0x35e6960}, 
{0x3d12000, 0x72}, {0x3eea780, 0xa, 0xa})


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/block_test.go:474
 +0x5c
github.com/prometheus/prometheus/tsdb.TestQueryHistogramFromBlocksWithCompaction.func2(0x35e6960, 
{0x6142000, 0x4, 0x4})


/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/db_test.go:6035
 +0x95c
github.com/prometheus/prometheus/tsdb.TestQueryHistogramFromBlocksWithCompaction.func8(0x35e6960)

/tmp/autopkgtest-lxc.mtbe33dm/downtmp/autopkgtest_tmp/.build/src/github.com/prometheus/prometheus/tsdb/db_test.go:6133
 +0x43c