Ottomata has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/358023 )
Change subject: Update camus config for eventbus topics ...................................................................... Update camus config for eventbus topics Change-Id: I12601afe5ec1d753715d28011533c0c48c30c684 --- M modules/camus/templates/eventbus.erb 1 file changed, 6 insertions(+), 2 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/23/358023/1 diff --git a/modules/camus/templates/eventbus.erb b/modules/camus/templates/eventbus.erb index a459d5c..6410b1b 100644 --- a/modules/camus/templates/eventbus.erb +++ b/modules/camus/templates/eventbus.erb @@ -10,7 +10,7 @@ # final top-level data output directory, sub-directory will be dynamically # created for each topic pulled -etl.destination.path=hdfs://analytics-hadoop/wmf/data/raw/eventbus +etl.destination.path=hdfs://analytics-hadoop/wmf/data/raw/event # Allow overwrites of previously imported files in etl.destination.path etl.destination.overwrite=true @@ -69,7 +69,11 @@ kafka.blacklist.topics= # These are the kafka topics camus brings to HDFS -kafka.whitelist.topics=mediawiki\.(page_delete|page_move|page_restore|revision_visibility_set|revision_create)|wmf\.resource_change +# These datacenter prefixed topics are going to be annoying, since they will show up as +# different datasets in HDFS. We'll either have to do some custom camus jobs for each +# logical topic to write it to a special directory, or write some special refinement +# code to load these up. Probably the latter. +kafka.whitelist.topics=(eqiad|codfw)\.mediawiki\.(page|revision|user).* # Name of the client as seen by kafka kafka.client.name=camus-eventbus-00 -- To view, visit https://gerrit.wikimedia.org/r/358023 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I12601afe5ec1d753715d28011533c0c48c30c684 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Ottomata <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
