Filippo Giunchedi has uploaded a new change for review.
https://gerrit.wikimedia.org/r/183537
Change subject: graphite: archive received metrics on disk
......................................................................
graphite: archive received metrics on disk
Right now if the carbon pipeline can't properly persist metrics we have
data loss. To mitigate this we can store the received metrics in plaintext
format on disk for some time. This can be used to replay the metric traffic and
backfill in case of data loss and metrics are kept at full resolution with no
downsampling.
Note that the class isn't applied anywhere yet, the disk requirements are very
different than serving graphite (raw storage space as opposed to many IOPS).
Estimating the disk requirements is somewhat hard at the moment but taking a
guesstimate of 1MB/s it would be 86G/day uncompressed.
Bug: T85908
Change-Id: Ifbf0fa4f06c26ba142ebf3025d19be0f50b1d6ad
---
A modules/graphite/files/carbon-archive
A modules/graphite/manifests/archive.pp
A modules/graphite/templates/archive.logrotate.conf.erb
A modules/graphite/templates/archive.upstart.conf.erb
4 files changed, 71 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/37/183537/1
diff --git a/modules/graphite/files/carbon-archive
b/modules/graphite/files/carbon-archive
new file mode 100644
index 0000000..791d83d
--- /dev/null
+++ b/modules/graphite/files/carbon-archive
@@ -0,0 +1,13 @@
+#!/bin/bash
+# listen on LISTEN_PORT and log the received traffic to ARCHIVE_DIR according
+# to TEMPLATE
+set -e
+set -u
+
+listen_port=${LISTEN_PORT:-1803}
+archive_dir=${ARCHIVE_DIR:-/var/lib/carbon/archive}
+template=${TEMPLATE:-%Y-%m-%dT%H.log}
+
+nc -k -l "${listen_port}" |
+ cronolog --symlink "${archive_dir}/current" \
+ "${archive_dir}/${template}"
diff --git a/modules/graphite/manifests/archive.pp
b/modules/graphite/manifests/archive.pp
new file mode 100644
index 0000000..0156522
--- /dev/null
+++ b/modules/graphite/manifests/archive.pp
@@ -0,0 +1,39 @@
+# == Class: graphite::archive
+#
+# Provide a local listener to accept plaintext metrics and archive them to
+# disk. The metrics archive can be used to recover from data loss or backfill
+# the metrics elsewhere for example.
+
+class graphite::archive(
+ $storage_dir = '/var/lib/carbon',
+) {
+ require_package('cronolog', 'netcat-traditional', 'logrotate')
+
+ $carbon_archive_path = '/usr/bin/carbon-archive'
+
+ file { '/etc/init/carbon/archive.conf':
+ owner => 'root',
+ group => 'root',
+ mode => '0444',
+ content => template('graphite/archive.upstart.conf.erb'),
+ notify => Service['carbon/archive'],
+ }
+
+ file { $carbon_archive_path:
+ source => 'puppet:///modules/graphite/carbon-archive',
+ mode => '0555',
+ before => Service['carbon/archive'],
+ }
+
+ file { '/etc/logrotate.d/carbon-archive':
+ owner => 'root',
+ group => 'root',
+ mode => '0444',
+ content => template('graphite/archive.logrotate.conf.erb'),
+ }
+
+ service { 'carbon/archive':
+ ensure => 'running',
+ provider => 'upstart',
+ }
+}
diff --git a/modules/graphite/templates/archive.logrotate.conf.erb
b/modules/graphite/templates/archive.logrotate.conf.erb
new file mode 100644
index 0000000..cce7a8c
--- /dev/null
+++ b/modules/graphite/templates/archive.logrotate.conf.erb
@@ -0,0 +1,7 @@
+<%= "#{storage_dir}" -%>/archive/*.log {
+ daily
+ delaycompress
+ compress
+ nocreate
+ rotate 90
+}
diff --git a/modules/graphite/templates/archive.upstart.conf.erb
b/modules/graphite/templates/archive.upstart.conf.erb
new file mode 100644
index 0000000..851e05f
--- /dev/null
+++ b/modules/graphite/templates/archive.upstart.conf.erb
@@ -0,0 +1,12 @@
+description "archive local carbon metric traffic to disk"
+
+start on (local-filesystems and net-device-up IFACE!=lo)
+stop on [!12345]
+
+limit nofile 32768 32768
+
+setuid _graphite
+
+env ARCHIVE_DIR=<%= "#{storage_dir}/archive" %>
+
+exec <%= "#{carbon_archive_path}" %>
--
To view, visit https://gerrit.wikimedia.org/r/183537
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifbf0fa4f06c26ba142ebf3025d19be0f50b1d6ad
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Filippo Giunchedi <fgiunch...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
