Ottomata has uploaded a new change for review.
https://gerrit.wikimedia.org/r/258202
Change subject: Add kafka::server::monitoring for WMF specific monitoring,
remove deprecated kafka::client class
......................................................................
Add kafka::server::monitoring for WMF specific monitoring, remove deprecated
kafka::client class
class kafka now does everything it used to, and what kafka::client did.
class kafka::server::monitoring can be used to monitor a kafka broker
in Wikimedia context. You can only use this class if you are using
this Kafka puppet module with Wikmedia's puppet repository.
Also fixup some README stuff.
Change-Id: If223c250ef12e073f3e55b7c87d320dd3d782c5b
---
M README.md
D manifests/client.pp
M manifests/init.pp
A manifests/server/monitoring.pp
4 files changed, 88 insertions(+), 39 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/kafka
refs/changes/02/258202/1
diff --git a/README.md b/README.md
index d1c4d25..31e28e0 100644
--- a/README.md
+++ b/README.md
@@ -32,12 +32,13 @@
## Kafka (Clients)
```puppet
-# Install the kafka package.
+# Install the kafka libraries and client packages.
class { 'kafka': }
```
-This will install the Kafka package which includes /usr/sbin/kafka, useful for
-running client (console-consumer, console-producer, etc.) commands.
+This will install the kafka-common and kafka-cli which includes
+/usr/bin/kafka, useful for running client (console-consumer,
+console-producer, etc.) commands.
## Kafka Broker Server
@@ -54,21 +55,21 @@
}
```
-```log_dirs``` defaults to a single ```['/var/spool/kafka]```, but you may
+`log_dirs` defaults to a single `['/var/spool/kafka]`, but you may
specify multiple Kafka log data directories here. This is useful for spreading
your topic partitions across multiple disks.
-The ```brokers``` parameter is a Hash keyed by ```$::fqdn```. Each value is
another Hash
-that contains config settings for that kafka host. ```id``` is required and
must
-be unique for each Kafka Broker Server host. ```port``` is optional, and
defaults
+The `brokers` parameter is a Hash keyed by `$::fqdn`. Each value is another
Hash
+that contains config settings for that kafka host. `id` is required and must
+be unique for each Kafka Broker Server host. `port` is optional, and defaults
to 9092.
-Each Kafka Broker Server's ```broker_id``` and ```port``` properties in
server.properties
-will be set based by looking up the node's ```$::fqdn``` in the hosts
-Hash passed into the ```kafka``` base class.
+Each Kafka Broker Server's `broker_id` and `port` properties in
server.properties
+will be set based by looking up the node's `$::fqdn` in the hosts
+Hash passed into the `kafka` base class.
-```zookeeper_hosts``` is an array of Zookeeper host:port pairs.
-```zookeeper_chroot``` is optional, and allows you to specify a Znode under
+`zookeeper_hosts` is an array of Zookeeper host:port pairs.
+`zookeeper_chroot` is optional, and allows you to specify a Znode under
which Kafka will store its metadata in Zookeeper. This is useful if you
want to use a single Zookeeper cluster to manage multiple Kafka clusters.
See below for information on how to create this Znode in Zookeeper.
@@ -79,11 +80,12 @@
If Kafka will share a Zookeeper cluster with other users, you might want to
create a Znode in zookeeper in which to store your Kafka cluster's data.
-You can set the ```zookeeper_chroot``` parameter on the ```kafka``` class to
do this.
+You can set the `zookeeper_chroot` parameter on the `kafka` class to do this.
First, you'll need to create the znode manually yourself. You can use
-```zkCli.sh``` that ships with Zookeeper, or you can use the kafka built in
-```zookeeper-shell```:
+`zkCli.sh` that ships with Zookeeper, or you can use the kafka built in
+`zookeeper-shell`:
+
```
$ kafka zookeeper-shell <zookeeper_host>:2182
@@ -99,7 +101,7 @@
```
You can use whatever chroot znode path you like. The second argument
-(```data```) is arbitrary. I used 'kafka' here.
+(`data`) is arbitrary. I used 'kafka' here.
Then:
```puppet
@@ -144,7 +146,7 @@
## jmxtrans monitoring
-This module contains a class called ```kafka::server::jmxtrans```. It contains
+This module contains a class called `kafka::server::jmxtrans`. It contains
a useful jmxtrans JSON config object that can be used to tell jmxtrans to send
to any output writer (Ganglia, Graphite, etc.). To you use this, you will need
the [puppet-jmxtrans](https://github.com/wikimedia/puppet-jmxtrans) module.
diff --git a/manifests/client.pp b/manifests/client.pp
deleted file mode 100644
index 6b8dbf1..0000000
--- a/manifests/client.pp
+++ /dev/null
@@ -1,17 +0,0 @@
-# == Class kafka::client
-# Installs kafka-client package.
-#
-class kafka::client {
- # Kafka class must be included before kafka::client.
- # Using 'require' here rather than an explicit class dependency
- # so that this class can be used without having to manually
- # include the base kafka class. This is for elegance only.
- # You'd only need to manually include the base kafka class if
- # you need to explicitly set the version of the Kafka package
- # you want installed.
- require ::kafka
-
- package { 'kafka-client':
- ensure => $::kafka::version
- }
-}
diff --git a/manifests/init.pp b/manifests/init.pp
index 6a4de9e..e166ddb 100644
--- a/manifests/init.pp
+++ b/manifests/init.pp
@@ -1,8 +1,7 @@
# == Class kafka
-# Installs kafka-common package.
-# Unless you need to explicitly set the version of the Kafka package you
-# want, you probably don't need to include this class directly.
-# Including just kafka::client, kafka::server to set up a Kafka Broker, or just
+# Installs kafka-common and kafka-cli packages.
+# You might not need to include this class directly.
+# Use just kafka::server to set up a Kafka Broker, or just
# kafka::mirror to set upa Kafka MirrorMaker service should suffice.
#
# == Parameters:
@@ -15,7 +14,7 @@
$version = $kafka::defaults::version
)
{
- package { 'kafka-common':
+ package { ['kafka-common', 'kafka-cli']:
ensure => $version,
}
}
diff --git a/manifests/server/monitoring.pp b/manifests/server/monitoring.pp
new file mode 100644
index 0000000..bda535d
--- /dev/null
+++ b/manifests/server/monitoring.pp
@@ -0,0 +1,65 @@
+# == Class kafka::server::monitoring
+# Wikimedia Foundation specific monitoring class.
+# Only include this if you are using this kafka puppet module
+# with the Wikimedia puppet repository from
+# github.com/wikimedia/operations-puppet.
+#
+# == Parameters
+# $nagios_servicegroup - Nagios Service group to use for alerts. Default:
undef
+#
+class kafka::server::monitoring($nagios_servicegroup = undef) {
+ # Generate icinga alert if Kafka Server is not running.
+ nrpe::monitor_service { 'kafka':
+ description => 'Kafka Broker Server',
+ nrpe_command => '/usr/lib/nagios/plugins/check_procs -c 1:1 -C java -a
"kafka.Kafka /etc/kafka/server.properties"',
+ require => Class['::kafka::server'],
+ critical => true,
+ }
+
+ # Generate icinga alert if this jmxtrans instance is not running.
+ nrpe::monitor_service { 'jmxtrans':
+ description => 'jmxtrans',
+ nrpe_command => '/usr/lib/nagios/plugins/check_procs -c 1:1 -C java
--ereg-argument-array "-jar.+jmxtrans-all.jar"',
+ require => Class['::kafka::server::jmxtrans'],
+ }
+
+ # Set up icinga monitoring of Kafka broker per second.
+ # If this drops too low, trigger an alert.
+ $nagios_servicegroup = 'analytics_eqiad'
+
+ # jmxtrans statsd writer emits Kafka Broker fqdns in keys
+ # by substiting '.' with '_' and suffixing the Broker port.
+ $graphite_broker_key = regsubst("${::fqdn}_${jmx_port}", '\.', '_', 'G')
+
+ # Alert if any Kafka has under replicated partitions.
+ # If it does, this means a broker replica is falling behind
+ # and will be removed from the ISR.
+ monitoring::graphite_threshold { 'kafka-broker-UnderReplicatedPartitions':
+ description => 'Kafka Broker Under Replicated Partitions',
+ metric =>
"kafka.${graphite_broker_key}.kafka.server.ReplicaManager.UnderReplicatedPartitions.Value",
+ # UnderReplicated partitions for more than a minute
+ # or two shouldn't happen.
+ warning => '1',
+ critical => '10',
+ require => Class['::kafka::server::jmxtrans'],
+ group => $nagios_servicegroup,
+ }
+
+ # Alert if any Kafka Broker replica lag is too high
+ monitoring::graphite_threshold { 'kafka-broker-Replica-MaxLag':
+ description => 'Kafka Broker Replica Max Lag',
+ metric =>
"kafka.${graphite_broker_key}.kafka.server.ReplicaFetcherManager.MaxLag.Value",
+ # As of 2014-02 replag could catch up at more than 1000 msgs / sec,
+ # (probably more like 2 or 3 K / second). At that rate, 1M messages
+ # behind should catch back up in at least 30 minutes.
+ warning => '1000000',
+ critical => '5000000',
+ require => Class['::kafka::server::jmxtrans'],
+ group => $nagios_servicegroup,
+ }
+
+ # monitor disk statistics
+ if $::standard::has_ganglia {
+ ganglia::plugin::python { 'diskstat': }
+ }
+}
--
To view, visit https://gerrit.wikimedia.org/r/258202
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If223c250ef12e073f3e55b7c87d320dd3d782c5b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/kafka
Gerrit-Branch: master
Gerrit-Owner: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
