Ottomata has uploaded a new change for review.
https://gerrit.wikimedia.org/r/159381
Change subject: Fix for mobile-100 kafkatee output filter
......................................................................
Fix for mobile-100 kafkatee output filter
Since August 23, kafkatee has been consuming more than just the
mobile webrequest topic. We didn't have any filtering in place
for the mobile output, so it was saving much more than mobile
requests.
Change-Id: I6339e98e199ce633d71f8007c59ebab617291b89
---
M manifests/role/analytics/kafkatee.pp
1 file changed, 10 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/81/159381/1
diff --git a/manifests/role/analytics/kafkatee.pp
b/manifests/role/analytics/kafkatee.pp
index a7eb4f3..4f4f718 100644
--- a/manifests/role/analytics/kafkatee.pp
+++ b/manifests/role/analytics/kafkatee.pp
@@ -75,11 +75,18 @@
class role::analytics::kafkatee::webrequest::mobile inherits
role::analytics::kafkatee::webrequest {
include role::analytics::kafkatee::input::webrequest::mobile
- # 1/100 sampling of traffic to mobile varnishes
+ # Include this to infer mobile varnish frontend hostnames on which to
filter.
+ include role::cache::configuration
+ $cache_configuration =
$role::cache::configuration::active_nodes['production']['mobile']
+ $mobile_hosts_regex = inline_template('(<%=
@cache_configuration.values.flatten.sort.join(\'|\') %>)')
+
+ # 1/100 sampling of traffic from mobile varnishes
::kafkatee::output { 'mobile-sampled-100':
- destination =>
"${webrequest_log_directory}/mobile-sampled-100.tsv.log",
+ destination => "/bin/grep -P '${mobile_hosts_regex}' >>
${webrequest_log_directory}/mobile-sampled-100.tsv.log",
sample => 100,
+ type => 'pipe',
}
+
# Capture all logs with 'zero=' set. The X-Analytics header is set with
this
# by mobile varnish frontends upon getting a Wikipedia Zero request.
::kafkatee::output { 'zero':
@@ -99,8 +106,8 @@
# by including their kafkatee::input::* roles.
#
class role::analytics::kafkatee::webrequest::webstatscollector {
- include role::analytics::kafkatee::input::webrequest::mobile
include role::analytics::kafkatee::input::webrequest::text
+ include role::analytics::kafkatee::input::webrequest::mobile
# webstatscollector package creates this directory.
# webstats-collector process writes dump files here.
--
To view, visit https://gerrit.wikimedia.org/r/159381
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I6339e98e199ce633d71f8007c59ebab617291b89
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
