Filippo Giunchedi has uploaded a new change for review. https://gerrit.wikimedia.org/r/183537
Change subject: graphite: archive received metrics on disk ...................................................................... graphite: archive received metrics on disk Right now if the carbon pipeline can't properly persist metrics we have data loss. To mitigate this we can store the received metrics in plaintext format on disk for some time. This can be used to replay the metric traffic and backfill in case of data loss and metrics are kept at full resolution with no downsampling. Note that the class isn't applied anywhere yet, the disk requirements are very different than serving graphite (raw storage space as opposed to many IOPS). Estimating the disk requirements is somewhat hard at the moment but taking a guesstimate of 1MB/s it would be 86G/day uncompressed. Bug: T85908 Change-Id: Ifbf0fa4f06c26ba142ebf3025d19be0f50b1d6ad --- A modules/graphite/files/carbon-archive A modules/graphite/manifests/archive.pp A modules/graphite/templates/archive.logrotate.conf.erb A modules/graphite/templates/archive.upstart.conf.erb 4 files changed, 71 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/37/183537/1 diff --git a/modules/graphite/files/carbon-archive b/modules/graphite/files/carbon-archive new file mode 100644 index 0000000..791d83d --- /dev/null +++ b/modules/graphite/files/carbon-archive @@ -0,0 +1,13 @@ +#!/bin/bash +# listen on LISTEN_PORT and log the received traffic to ARCHIVE_DIR according +# to TEMPLATE +set -e +set -u + +listen_port=${LISTEN_PORT:-1803} +archive_dir=${ARCHIVE_DIR:-/var/lib/carbon/archive} +template=${TEMPLATE:-%Y-%m-%dT%H.log} + +nc -k -l "${listen_port}" | + cronolog --symlink "${archive_dir}/current" \ + "${archive_dir}/${template}" diff --git a/modules/graphite/manifests/archive.pp b/modules/graphite/manifests/archive.pp new file mode 100644 index 0000000..0156522 --- /dev/null +++ b/modules/graphite/manifests/archive.pp @@ -0,0 +1,39 @@ +# == Class: graphite::archive +# +# Provide a local listener to accept plaintext metrics and archive them to +# disk. The metrics archive can be used to recover from data loss or backfill +# the metrics elsewhere for example. + +class graphite::archive( + $storage_dir = '/var/lib/carbon', +) { + require_package('cronolog', 'netcat-traditional', 'logrotate') + + $carbon_archive_path = '/usr/bin/carbon-archive' + + file { '/etc/init/carbon/archive.conf': + owner => 'root', + group => 'root', + mode => '0444', + content => template('graphite/archive.upstart.conf.erb'), + notify => Service['carbon/archive'], + } + + file { $carbon_archive_path: + source => 'puppet:///modules/graphite/carbon-archive', + mode => '0555', + before => Service['carbon/archive'], + } + + file { '/etc/logrotate.d/carbon-archive': + owner => 'root', + group => 'root', + mode => '0444', + content => template('graphite/archive.logrotate.conf.erb'), + } + + service { 'carbon/archive': + ensure => 'running', + provider => 'upstart', + } +} diff --git a/modules/graphite/templates/archive.logrotate.conf.erb b/modules/graphite/templates/archive.logrotate.conf.erb new file mode 100644 index 0000000..cce7a8c --- /dev/null +++ b/modules/graphite/templates/archive.logrotate.conf.erb @@ -0,0 +1,7 @@ +<%= "#{storage_dir}" -%>/archive/*.log { + daily + delaycompress + compress + nocreate + rotate 90 +} diff --git a/modules/graphite/templates/archive.upstart.conf.erb b/modules/graphite/templates/archive.upstart.conf.erb new file mode 100644 index 0000000..851e05f --- /dev/null +++ b/modules/graphite/templates/archive.upstart.conf.erb @@ -0,0 +1,12 @@ +description "archive local carbon metric traffic to disk" + +start on (local-filesystems and net-device-up IFACE!=lo) +stop on [!12345] + +limit nofile 32768 32768 + +setuid _graphite + +env ARCHIVE_DIR=<%= "#{storage_dir}/archive" %> + +exec <%= "#{carbon_archive_path}" %> -- To view, visit https://gerrit.wikimedia.org/r/183537 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ifbf0fa4f06c26ba142ebf3025d19be0f50b1d6ad Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Filippo Giunchedi <fgiunch...@wikimedia.org> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits