In case anyone came across this, I resolve it by moving to Raspberry Pi OS 11 64-bit.
All other configs remains the same. Moral of the story - don't run Prometheus in 32-bit... On Thursday, 23 December 2021 at 10:42:21 am UTC+11 Kok-Yan Lo wrote: > Hey Everyone, my Prometheus instance died with the dreaded error: > "panic: mmap, size 134217728: cannot allocate memory" > > Just wondering if anyone managed to solve it? > > It is *exactly* the same issue (and log pattern too) as reported here - > I'm on 32-bit Raspbian 10, on a Pi 4 (4 GB model): > https://github.com/prometheus/prometheus/issues/8661 > > I tried removing all files in "wal" directory, which brought it back but > then failed again with the similar error. > ts=2021-12-22T17:57:22.806Z caller=compact.go:459 level=info > component=tsdb msg="compact blocks" count=3 mint=1639375206018 > maxt=1639958400000 ulid=01FQHMMJYJQK9B3NQD5P7PGA6Q > sources="[01FPZGK37J5MZ80SGREDQ6J87Q 01FQ533ZRMFYPNGG9MYZC36KHC > 01FQAWGKH8XE52XXFE6NSJAWXX]" duration=5.925005803s > ts=2021-12-22T17:57:23.063Z caller=db.go:830 level=error component=tsdb > msg="compaction failed" err="reloadBlocks blocks: corrupted block > 01FQHMMJYJQK9B3NQD5P7PGA6Q: mmap files: mmap, size 206139815: cannot > allocate memory" > ts=2021-12-22T17:59:24.390Z caller=stdlib.go:105 level=error component=web > caller="http: TLS handshake error from 172.19.0.1:60974" msg=EOF > panic: mmap, size 134217728: cannot allocate memory > > goroutine 428 [running]: > > github.com/prometheus/prometheus/tsdb.(*memSeries).mmapCurrentHeadChunk(0x6b38a50, > > 0x6710510) > /app/tsdb/head_append.go:565 +0x220 > > github.com/prometheus/prometheus/tsdb.(*memSeries).cutNewHeadChunk(0x6b38a50, > 0x17de34cdd97, 0x6710510) > /app/tsdb/head_append.go:536 +0x24 > github.com/prometheus/prometheus/tsdb.(*memSeries).append(0x6b38a50, > 0x17de34cdd97, 0x3ff0000000000000, 0x392, 0x6710510) > /app/tsdb/head_append.go:503 +0x320 > github.com/prometheus/prometheus/tsdb.(*headAppender).Commit(0x78fb620) > /app/tsdb/head_append.go:439 +0x3f0 > github.com/prometheus/prometheus/tsdb.dbAppender.Commit({{0x2bfce60 > <http://github.com/prometheus/prometheus/tsdb.dbAppender.Commit(%7B%7B0x2bfce60>, > > 0x78fb620}, 0x6710000}) > /app/tsdb/db.go:871 +0x24 > > github.com/prometheus/prometheus/storage.(*fanoutAppender).Commit(0x83ac920) > /app/storage/fanout.go:176 +0x28 > > github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport.func1(0x7505e18, > > 0x7505e20, 0x5fb4280) > /app/scrape/scrape.go:1250 +0x38 > > github.com/prometheus/prometheus/scrape.(*scrapeLoop).scrapeAndReport(0x5fb4280, > > 0xdf8475800, 0x2540be400, {0xc068f7ba5031d0cd, 0x57dc6cc6a0d, 0x3e8c588}, > {0x10293dea, 0xed9555ea5, 0x3e8c588}, 0x0) > /app/scrape/scrape.go:1317 +0xe58 > github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0x5fb4280, 0x0) > /app/scrape/scrape.go:1203 +0x324 > created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync > /app/scrape/scrape.go:584 +0x86c > > > > > > I don't have any special configs to my container - just this > docker-compose.yml: > version: '3.8' > services: > prometheus: > image: prom/prometheus:v2.31.1 > container_name: prometheus > volumes: > # Config > - /srv/prometheus/etc/prometheus/:/etc/prometheus/:ro > # Data > - /srv/prometheus/prometheus/:/prometheus/ > command: > # Default commands and paths from > https://github.com/prometheus/prometheus/blob/main/Dockerfile > - "--config.file=/etc/prometheus/prometheus.yml" > - "--storage.tsdb.path=/prometheus" > - "--web.console.libraries=/usr/share/prometheus/console_libraries" > - "--web.console.templates=/usr/share/prometheus/consoles" > # Keep data for 3 months > - --storage.tsdb.retention.time=90d > # Listen on port 9090 inside container > - --web.listen-address=:9090 > # Use TLS with Basic Auth > - --web.config.file=/etc/prometheus/web-config.yml > ports: > - 443:9090 > restart: on-failure:3 > > Any idea / suggestions welcome :-) > > Thanks! > > Kok-Yan > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/423b3d04-4a99-4bde-9bad-e50bb9c6b80en%40googlegroups.com.
