Ori.livneh has uploaded a new change for review. https://gerrit.wikimedia.org/r/85337
Change subject: Add EventLogging Kafka writer plug-in ...................................................................... Add EventLogging Kafka writer plug-in Depends on kafka-python, which still needs to be packaged. Tracked: <https://mingle.corp.wikimedia.org/projects/analytics/cards/1165> Change-Id: Ia58c4503b91bdeba6016432602159df2f735db7b --- M manifests/role/eventlogging.pp A modules/eventlogging/files/plugins/eventlogging-kafka.py 2 files changed, 36 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/37/85337/1 diff --git a/manifests/role/eventlogging.pp b/manifests/role/eventlogging.pp index 103c11c..4ec17d2 100644 --- a/manifests/role/eventlogging.pp +++ b/manifests/role/eventlogging.pp @@ -156,6 +156,18 @@ } + ## Kafka / Hadoop + + eventlogging::plugin { 'kafka': + source => 'puppet:///modules/eventlogging/plugins/eventlogging-kafka.py', + } + + # eventlogging::service::consumer { 'kafka': + # input => "tcp://${processor}:8600", + # output => "kafka://${kafka_host}:${kafka_port}", + # } + + ## Monitoring nrpe::monitor_service { 'eventlogging': diff --git a/modules/eventlogging/files/plugins/eventlogging-kafka.py b/modules/eventlogging/files/plugins/eventlogging-kafka.py new file mode 100644 index 0000000..0de07e0 --- /dev/null +++ b/modules/eventlogging/files/plugins/eventlogging-kafka.py @@ -0,0 +1,24 @@ +# -*- coding: utf-8 -*- +""" + Kafka writer plug-in for EventLogging + + Requires kafka-python: <https://github.com/mumrah/kafka-python>. + + Copyright (C) 2013, Ori Livneh <[email protected]> + Licensed under the terms of the GPL, version 2 or later. + +""" +from kafka.client import KafkaClient +from kafka.producer import KeyedProducer + + +@writes('kafka') +def kafka_writer(hostname, port, topic='eventlogging', **kwargs): + """Write events to Kafka, keyed by SCID.""" + kafka = KafkaClient(hostname, port) + producer = KeyedProducer(kafka, topic, **kwargs) + + while 1: + event = (yield) + key = '%(schema)s_%(revision)s' % event # e.g. 'EchoMail_5467650' + producer.send(key, json.dumps(event, sort_keys=True)) -- To view, visit https://gerrit.wikimedia.org/r/85337 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ia58c4503b91bdeba6016432602159df2f735db7b Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Ori.livneh <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
