Package: victoria-metrics
Version: 1.79.5+ds1-2+b5
Severity: grave
victoria-metrics v1.79.5 randomly crashes on aligned reads file reads,
rendering it more or less unusable.
Sep 7 08:46:31 localhost victoria-metrics[17179]: unexpected fault address 0x0
Sep 7 08:46:31 localhost kernel: [233869.646841] Alignment trap: not handling
instruction ed9c0b00 at [<005af9bc>]
Sep 7 08:46:31 localhost kernel: [233869.646882] 8<--- cut here ---
Sep 7 08:46:31 localhost kernel: [233869.646887] Unhandled fault: alignment
exception (0x001) at 0x02153612
Sep 7 08:46:31 localhost kernel: [233869.646898] [02153612] *pgd=3ea5d831
Sep 7 08:46:31 localhost victoria-metrics[17179]: fatal error: fault
Sep 7 08:46:31 localhost victoria-metrics[17179]: [signal SIGBUS: bus error
code=0x1 addr=0x0 pc=0x5af9c0]
Sep 7 08:46:31 localhost victoria-metrics[17179]: goroutine 28 [running]:
Sep 7 08:46:31 localhost victoria-metrics[17179]: runtime.throw({0x6abcef,
0x5})
Sep 7 08:46:31 localhost victoria-metrics[17179]: #011runtime/panic.go:1047
+0x4c fp=0x24993b4 sp=0x24993a0 pc=0x50424
Sep 7 08:46:31 localhost victoria-metrics[17179]: runtime.sigpanic()
Sep 7 08:46:31 localhost victoria-metrics[17179]:
#011runtime/signal_unix.go:832 +0xac fp=0x24993cc sp=0x24993b4 pc=0x69fc8
Sep 7 08:46:31 localhost victoria-metrics[17179]: math.IsNaN(...)
Sep 7 08:46:31 localhost victoria-metrics[17179]: #011math/bits.go:39
Sep 7 08:46:31 localhost victoria-metrics[17179]:
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.removeEmptySeries(...)
Sep 7 08:46:31 localhost victoria-metrics[17179]:
#011github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql/exec.go:157
Sep 7 08:46:31 localhost victoria-metrics[17179]:
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.timeseriesToResult({0x21fe3b0,
0x1, 0x2}, 0x1)
Sep 7 08:46:31 localhost victoria-metrics[17179]:
#011github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql/exec.go:99
+0x610 fp=0x24994ac sp=0x24993d0 pc=0x5af9c0
Sep 7 08:46:31 localhost victoria-metrics[17179]:
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql.Exec(0x0,
0x28c6540, {0x201f325, 0x1a}, 0x0)
Sep 7 08:46:31 localhost victoria-metrics[17179]:
#011github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/promql/exec.go:58
+0x2ec fp=0x2499584 sp=0x24994ac pc=0x5aebbc
Sep 7 08:46:31 localhost victoria-metrics[17179]:
github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus.queryRangeHandler(0x0,
{0xc1368159f7200fc8, 0x299f41302, 0xb5cc08}, {0x81ffbc, 0x292a320},
{0x201f325, 0x1a}, 0x18a6b9c9ae0, 0x18a6bb93718, ...)
Sep 7 08:46:31 localhost victoria-metrics[17179]:
#011github.com/VictoriaMetrics/VictoriaMetrics/app/vmselect/prometheus/prometheus.go:840
+0x5f4 fp=0x2499748 sp=0x2499584 pc=0x5ea4cc
[...]
Seehttps://github.com/VictoriaMetrics/VictoriaMetrics/pull/3927 for more
details on the issue.
The bug has already been patched upstream and an LTS release (v1.79.11) has
been made in March, along with other important
fixes:https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.79.11
.
This is on an armhf box (hardkernel odroid-c1) running Debian 12.1, kernel
6.1.0-11-armmp, fully up to date.
Steps to reproduce:
- run victoria-metrics on a 32-bit arm system,
- ingest a small amount of data (my data directory is less than 10MB),
- query the data with e.g. grafana, by zooming in and moving the observed time
window.
Querying with some automated script should also work as all that needs to be
done is to cause an unaligned read on disk, which is a naturally occurring
condition due to how datapoints are stored and compressed.
Would it be possible to upgrade to that LTS release?
Thanks in advance,
-Simon
--
Simon Vetter
