In fact, the good design choices made in 2.x were intentional to help embedded use cases. The updated version reduces IO by a factor of 100. Eliminating the damaging write patterns that existed in 1.x. It is no longer required to write to tmpfs in order to protect flash storage from poor write patterns that existed in Prometheus 1.x.
This change was well documented in the 2.0.0 release notes and announcements in 2017. https://github.com/prometheus/prometheus/blob/master/CHANGELOG.md#200--2017-11-08 On Mon, Jan 25, 2021 at 4:42 PM prommyonapi <[email protected]> wrote: > Let me get this straight, poor software design choices result in > destroying any local tmpfs IoT device use case for Prometheus and anyone > that points this out is banned and given some bullshit code of conduct > reason. Anyone using Prometheus on flash memory is writing hundreds of MB > of data an hour or a day, or worse to stop kilobytes of data thus > obliterating the majority of IoT storage in the process. > > Got it. > > Well done. Well done. > > I guess there's no need to use Prometheus at this point for local IoT. > > Good job. > > > > > On Monday, January 25, 2021 at 7:39:10 AM UTC-7 [email protected] > wrote: > >> Hi, >> >> The experimental flag to allow limiting disk usage was mostly meant for >> larger disk usages than a couple of hundred MB, but there's an opportunity >> to clarify that in the flag description. Prometheus will need to store at >> least a couple of hundred MB due to the way it segments and preallocates >> data (current TSDB devs would be able to provide more detail here). Indeed >> this behavior changed with 2.x, as the entire storage engine got swapped >> out for a much better one (except for tiny use cases like this). >> >> Please refrain from personal insults, as it's against our code of conduct >> (https://github.com/cncf/foundation/blob/master/code-of-conduct.md). >> >> Regards, >> Julius >> >> On Mon, Jan 25, 2021 at 2:59 PM prommyonapi <[email protected]> wrote: >> >>> https://github.com/prometheus/prometheus/issues/8406 >>> >>> pi@raspberrypi:~ $ cat /etc/fstab >>> proc /proc proc defaults 0 0 >>> PARTUUID=6c586e13-01 /boot vfat defaults,noatime 0 2 >>> PARTUUID=6c586e13-02 / ext4 >>> defaults,noatime,commit=600,errors=remount-ro 0 1 >>> tmpfs /tmp tmpfs defaults,noatime,nosuid 0 0 >>> tmpfs /var/log tmpfs defaults,noatime,nosuid,size=50M 0 0 >>> tmpfs /prometheus-tmpfs tmpfs defaults,nosuid,size=200M >>> >>> *What did you do?* >>> Prometheus + node_exporter on Buster >>> saving prometheus in TMPFS to save SD writes >>> >>> *What did you expect to see?* >>> It working >>> >>> *What did you see instead? Under which circumstances?* >>> faliures: >>> >>> Jan 25 03:12:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:12:23.027Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:13:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:13:23.069Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:14:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:14:23.150Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:15:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:15:23.186Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:16:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:16:23.217Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:17:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:17:23.254Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:18:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:18:23.316Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:19:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:19:23.391Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:20:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:20:23.414Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> Jan 25 03:21:23 raspberrypi prometheus[6999]: level=error >>> ts=2021-01-25T03:21:23.430Z caller=db.go:745 component=tsdb msg="compaction >>> failed" err="compact head: persist head block: populate block: write >>> chunks: preallocate: no space left on device" >>> >>> *Environment* >>> Raspbian Buster >>> >>> - System information: >>> >>> Linux 5.4.83-v7l+ armv7l >>> >>> - Prometheus version: >>> >>> prometheus, version 2.24.1 (branch: HEAD, revision: e448727 >>> <https://github.com/prometheus/prometheus/commit/e4487274853c587717006eeda8804e597d120340> >>> ) >>> build user: root@0b5231a0de0f >>> build date: 20210120-01:04:17 >>> go version: go1.15.6 >>> platform: linux/arm >>> >>> - Prometheus configuration file: >>> >>> - job_name: 'node' scrape_interval: 60s scrape_timeout: 10s >>> static_configs: - targets: ['localhost:9100'] - job_name: 'dump1090' >>> scrape_interval: 60s scrape_timeout: 10s static_configs: - targets: >>> ['localhost:9105'] =/usr/bin/prometheus >>> --config.file=/etc/prometheus/prometheus.yml \ >>> --storage.tsdb.path="/prometheus-tmpfs/prometheus" \ >>> --storage.tsdb.min-block-duration=30m --storage.tsdb.max-block-duration=1h >>> \ --storage.tsdb.wal-compression \ --storage.tsdb.retention.time=1d >>> --storage.tsdb.retention.size=50MB \ --storage.tsdb.no-lockfile \ >>> --web.enable-admin-api [image: Screenshot_2021-01-24_20-22-42.png] >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/510fc30b-c42f-4884-b9e7-b3257540db5dn%40googlegroups.com >>> <https://groups.google.com/d/msgid/prometheus-users/510fc30b-c42f-4884-b9e7-b3257540db5dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Julius Volz >> PromLabs - promlabs.com >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/a62b0b9c-60d3-4595-8d62-26c6180fe11cn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/a62b0b9c-60d3-4595-8d62-26c6180fe11cn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmoS6%2B9mx%2BNQLRng3xFFmdHcY-Zx0JWjUQZ%2BPBxCY8uGBQ%40mail.gmail.com.
