Hi,
On 12/9/20 3:48 PM, iono sphere wrote:
I am not sure who could I ask this, but I would like to try here.
Currently, I'm seeing something weird in my server. Thanks to Prometheus
and Node-Exporter, I have seen that node_filesystem_avail_bytes has been
fluctuating up and down for hundreds of Megabytes once a day or two. So
I am trying to hunt down the root cause of this. I have been asking
everyone already that at the time of
node_filesystem_avail_bytes increasing/decreasing for hundreds of
Megabytes nobody has access the server. I have already checked the
auth.log file as well to confirm this. Our server only has a PHP Larvel
project being used as a development server, MySQL, Vue.js Front-End
Server, and Docker to containerize these. But none of these should
create or remove hundreds of Megabytes...
My current guess right now is the swap file might be at work but then I
am not sure.
Swap often has its own partition. If it's really a file, it is usually
pre-allocated, I think. So this sounds unlikely to me.
Are there other possible systematic process in Ubuntu that
creates/removes hundreds of Megabytes once in a day or two?
Cron jobs or systemd timers come to mind. Log rotation might behave that
way for large logs. MySQL might perform some reorganization. Prometheus
itself runs a process called compaction which temporarily needs more space:
https://prometheus.io/docs/prometheus/latest/storage/#compaction
Checking the Prometheus logs and maybe increasing the log level can
help. There are also metrics around compaction (e.g.
prometheus_tsdb_compactions_total) which are worth checking.
In fact, is there a way to subtract all the values of expected
systematic processes from node_filesystem_avail_bytes?
I don't think so. node_exporter takes the value directly from the
relevant kernel interfaces. It's hard to decide what's systematic and
what not. ;)
If you cannot find the cause after checking the items from above, you
can try hunting it down using systematic tracing (perf trace, systemtap,
iosnoop [1]).
[1] http://www.brendangregg.com/blog/2014-07-16/iosnoop-for-linux.html
Kind regards,
Christian
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to prometheus-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/8c1b329e-7821-5413-f073-7b01ce453591%40hoffmann-christian.info.
