#29682: remove traces munin-node everywhere -------------------------------------------------+------------------------- Reporter: anarcat | Owner: anarcat Type: defect | Status: | assigned Priority: Medium | Milestone: Component: Internal Services/Tor Sysadmin Team | Version: Severity: Normal | Resolution: Keywords: | Actual Points: Parent ID: #29681 | Points: Reviewer: | Sponsor: -------------------------------------------------+-------------------------
Comment (by anarcat): fun facts found while ripping that stuff out: 1. there is a DSA-specific munin package, with the following plugins: * `spamassassin`: ham/spam/total counts, looks for `spamd: ((processing|checking) message|identified spam|clean message)` in `mail.log`, could be replaced with [https://github.com/google/mtail mtail] * `postgres-wal-traffic_`: output of `psql -p "$port" --no-align --command 'SELECT * FROM pg_current_xlog_insert_location()' --tuples-only --quiet | tr -d /`, probably covered by the [https://github.com/wrouesnel/postgres_exporter postgres exporter] * `ksm_scans`: output of `/sys/kernel/mm/ksm/full_scans`, see [https://www.kernel.org/doc/html/latest/admin-guide/mm/ksm.html KSM docs] * `ksm`: same, but with `pages_shared`, `_unshared`, `_volatile`, `_sharing`, possibly covered by the node exporter, but hardly seems critical in any case * `vsftpd`: logtail of `/var/log/ftp/vsftpd.log` looking for upload/download/login/delete/connexions, would require a custom mtail plugin as well * `bind`: logtail of `/var/log/daemon.log`, looking for queries etc, easy replacement with the [https://github.com/digitalocean/bind_exporter/ Prometheus exporter] * `apache_servers`: apache server-status, equivalent of the apache exporter, already deployed 2. there's a packet counting script in `ferm` which seem to count per-IP packet stats from iptables: {{{ $munin_ips = split(regsubst($v4ips, '([^,]+)', 'ip_\1', 'G'), ',') munin::check { $munin_ips: script => "ip_"; } if $v6ips { $munin6_ips = split(regsubst($v6ips, '([^,]+)', 'ip_\1', 'G'), ',') munin::check { $munin6_ips: script => 'ip_', } } }}} i have just removed those, without a replacement. 3. hiding in the haproxy puppet module is another munin plugin. there is also a [https://github.com/prometheus/haproxy_exporter prometheus exporter for haproxy] which we can eventually deploy to replace this. in the meantime, it was deleted 4. the VM image installer (`modules/roles/files/virt/tor-install-VM`) has noises about setting up VM-specific plugins: `echo ' for i in `/usr/local/sbin/vm_du_ suggest`; do ln -vsf /usr/local/sbin/vm_du_ /etc/munin/plugins/vm_du_$i; done'`. that file does not seem to be deployed through Puppet, and consists of a script checking the disk space of all VMs. It looks something like this on `kvm4` right now: {{{ #!/bin/bash # -*- sh -*- MUNIN_LIBDIR=${MUNIN_LIBDIR:-/usr/share/munin} . $MUNIN_LIBDIR/plugins/plugin.sh BASE=/srv/vmstore VM=${0##*vm_du_} #VM=${VM//_/.} case $1 in autoconf) if [[ -d "$BASE" ]]; then echo yes exit 0 else echo "no ($BASE not found)" exit 0 fi ;; suggest) if [[ -d "$BASE" ]]; then find "$BASE" -mindepth 1 -maxdepth 1 -type d -a ! -name lost+found -printf '%f\n' # | tr . _ fi exit 0 ;; config) echo "graph_title disk usage VM $VM" echo 'graph_args --base 1024 --lower-limit 0' echo 'graph_vlabel bytes' echo 'graph_category disk' echo 'graph_total Total' find "$BASE/$VM" -mindepth 1 -maxdepth 1 -type f | while read fn; do label="${fn##*/}" label=${label//./_} name=${label//-/_} echo "$name.label $label" echo "$name.cdef $name,1024,*" done exit 0 ;; esac find "$BASE/$VM" -mindepth 1 -maxdepth 1 -type f -printf '%f %k\n' | while read fn du; do fn=${fn//[.-]/_} echo "$fn.value $du" done }}} that is covered by #29816. 5. the munin-common package doesn't remove its own user/group by default so I did that by hand. there's a possibility that some files are leftover in /var or /etc, but I am ready to assume the consequence of a possible UID reuse there to remove an extra account from all servers 6. normally, the package removal process should have removed all of /etc/munin/plugins, but there are some leftovers sometimes, e.g. on `oo- hetzner-03`: {{{ diskstats fw_forwarded_local if_err_eth0 ip_38.229.72.27 ntp_kernel_err ntp_kernel_pll_off postfix_mailvolume threads fw_conntrack fw_packets if_eth0 netstat ntp_kernel_pll_freq postfix_mailqueue proc_pri users }}} Those are all symlinks to builtin plugins, so I think they can be safely removed and have done so. 7. nagios was watching that munin was running everywhere in its static configuration, I have removed that check as well All those changes will take some time to propagate everywhere, which will make Nagios noisy for a little while. Tomorrow, it will be possible to remove remaining Munin code from Puppet entirely, assuming all nodes will have run Puppet correctly. -- Ticket URL: <https://trac.torproject.org/projects/tor/ticket/29682#comment:1> Tor Bug Tracker & Wiki <https://trac.torproject.org/> The Tor Project: anonymity online
_______________________________________________ tor-bugs mailing list tor-bugs@lists.torproject.org https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs