[MediaWiki-commits] [Gerrit] operations...testing[master]: Add another test to run Varnishkafka with Valgrind
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/328381 )
Change subject: Add another test to run Varnishkafka with Valgrind
..
Add another test to run Varnishkafka with Valgrind
Bug: T147438
Change-Id: I745588655af75e636b921ba4fd62244e24a2cbfb
---
M Dockerfile
M README.md
A run_valgrind_test
3 files changed, 92 insertions(+), 2 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/operations/software/varnish/varnishkafka/testing
refs/changes/81/328381/1
diff --git a/Dockerfile b/Dockerfile
index 8a086c0..26c78ae 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -48,12 +48,14 @@
# Update the repository sources list
RUN apt-get update && apt-get -y dist-upgrade
-
# Install Varnish 4 and Apache 2.4
RUN apt-get install -y varnish apache2 php5-fpm
# Install Varnishkafka dev libs
RUN apt-get install -y librdkafka-dev libvarnishapi-dev zlib1g-dev libyajl-dev
git gcc make python
+
+# Install Valgrind
+RUN apt-get install -y valgrind
# Copy configuration files
RUN mkdir -p /var/www/test
@@ -71,9 +73,11 @@
COPY ${VARNISHKAFKA_SOURCE_PATH}/* /tmp/varnishkafka/
COPY ${SERVICES_CONFIG_PATH}/varnishkafka.conf
/tmp/varnishkafka/varnishkafka.conf
-# Copy the test script
+# Copy the test scripts
COPY run_test /run_test
+COPY run_valgrind_test /run_valgrind_test
RUN chmod +x /run_test
+RUN chmod +x /run_valgrind_test
# Copy validation script
COPY check_output.py /tmp/varnishkafka/check_vk_output.py
diff --git a/README.md b/README.md
index 3071021..a5a9383 100644
--- a/README.md
+++ b/README.md
@@ -36,6 +36,10 @@
* Logs periodically system metrics related to CPU/Memory utilization and load.
* Checks the Varnishkafka output and report any inconsistency.
+## run_valgrind_test
+This bash script is the same as the above one with the important exception
that Varnishkafka
+will run through Valgrind. The final report is the Valgrind output for memory
leak checks.
+
# Instructions
## Building
diff --git a/run_valgrind_test b/run_valgrind_test
new file mode 100644
index 000..d5a697d
--- /dev/null
+++ b/run_valgrind_test
@@ -0,0 +1,82 @@
+#!/bin/bash
+
+# Copyright © 2016 Luca Toscano
+# Copyright © 2016 Wikimedia Foundation, Inc.
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+##
+# Script that performs the following things:
+# 1) Configure Apache modules loaded.
+# 2) Spins up Apache, Varnish and PHP-FPM.
+# 3) Build and Install Varnishkafka.
+# 4) Runs Varnishkafka via Valgrind and nohup.
+# 5) Return the Valgrind output after certain amount
+#of HTTP requests. Since it is not meant to be a load test
+#it is sufficient only a simple bash loop with curl.
+#
+# GOAL: Memory leak checks for Varnishkafka.
+#
+# Assumptions:
+# 1) The OS is Debian Jessie (especially for the Apache config).
+# 2) Varnishkafka logs to stdout in JSON format (Kafka not tested).
+# 3) Apache, Varnish, PHP-FPM are already configured.
+# 4) Varnishkafka sources are located in /tmp/varnishkafka.
+# 5) Curl and Python are available.
+###
+
+set -e
+
+#Number of runs to perform
+RUNS=100
+
+if [ $1 -gt 0 ]
+then
+RUNS=$1
+fi
+
+# Compile and install Varnishkafka
+cd /tmp/varnishkafka
+make && make install
+
+# Enable Apache httpd modules
+a2enmod proxy
+a2enmod proxy_fcgi
+
+# Start the processing chain Varnish -> Apache -> php-fpm
+/usr/sbin/service varnish start
+/usr/sbin/service php5-fpm start
+/usr/sbin/service apache2 start
+
+VALGRIND_ARGS="--log-file=/tmp/varnishkafka/valgrind.log --leak-check=full
--show-leak-kinds=all"
+# Start Varnishkafka in the background with Valgrind, logging to a known file.
+# Note: stdout tuned to be line buffered
+nohup stdbuf -oL /usr/bin/valgrind $VALGRIND_ARGS /usr/local/bin/varnishkafka
-S /tmp/varnishkafka/varnishkafka.conf > /dev/null &
+
+echo "Running curl to generate some fake traffic (all MISSes thanks to the
variable query string)"
+for run in $(seq 1 $RUNS);
+do
+echo "Run: $run";
+curl -i -s http://localhost/test.php?$run --header "Host: test.com"
--header "Referer: hello.com?advice=readwikipedia" | wc -l | xargs echo -e
"\tHTML Response lines:";
+done
+
+# Kill Varnishkafka to force Valgrind to emit its output
+echo -e "\n\nKilling Varnishkafka"
+kill $(pgrep -f varnishkafka)
+
+echo -e "\n\nWait a couple of s
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Move mw116[89] to a single partition recipe
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/328345 )
Change subject: Move mw116[89] to a single partition recipe
..
Move mw116[89] to a single partition recipe
The mw116[89] hosts have only one disk and the current recipe
creates a very tiny /tmp partition that fills up very quickly
(they are videoscalers using the tmp dir for transcode temporary
files).
The new recipe is basically a single partition without the restriction
of the small tmp directory.
Bug: T153488
Change-Id: Id336e8addae60645026360b54eeab96bb4a6ff37
---
M modules/install_server/files/autoinstall/netboot.cfg
A modules/install_server/files/autoinstall/partman/mw-no-tmp.cfg
2 files changed, 31 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/45/328345/1
diff --git a/modules/install_server/files/autoinstall/netboot.cfg
b/modules/install_server/files/autoinstall/netboot.cfg
index 40e2c85..fc94bad 100755
--- a/modules/install_server/files/autoinstall/netboot.cfg
+++ b/modules/install_server/files/autoinstall/netboot.cfg
@@ -119,6 +119,7 @@
mw13*) echo partman/mw-raid1.cfg ;; \
mw22*) echo partman/mw-raid1.cfg ;; \
osmium) echo partman/mw-raid1.cfg ;; \
+mw116[89]) echo partman/mw-no-tmp.cfg ;; \
mw[0-9]*|rcs[0-9]*) echo partman/mw.cfg ;; \
mx1001|mx2001) echo partman/flat.cfg virtual.cfg ;; \
labservices1001|labservices1002|radium|scandium|silver|uranium) echo
partman/raid1.cfg ;; \
diff --git a/modules/install_server/files/autoinstall/partman/mw-no-tmp.cfg
b/modules/install_server/files/autoinstall/partman/mw-no-tmp.cfg
new file mode 100644
index 000..5b6079f
--- /dev/null
+++ b/modules/install_server/files/autoinstall/partman/mw-no-tmp.cfg
@@ -0,0 +1,30 @@
+# MediaWiki. Creates:
+# * no RAID (either single disk or hardware raid presenting a single device)
+# * no LVM
+# * no swap
+# * One ext4 partition for /
+
+d-ipartman-auto/method string regular
+d-ipartman-auto/disk string /dev/sda
+
+d-ipartman-auto/expert_recipe string \
+ apache :\
+ 64 10001000 ext4 \
+ $primary{ } \
+ $bootable{ }\
+ method{ format }\
+ format{ } \
+ use_filesystem{ } \
+ filesystem{ ext4 } \
+ mountpoint{ / } \
+ .
+
+d-ipartman-auto/choose_recipe apache
+
+# no swap
+d-ipartman-basicfilesystems/no_swapboolean false
+
+d-ipartman/choose_partitionselect finish
+d-ipartman/confirm boolean true
+d-ipartman/confirm_nooverwrite boolean true
+d-ipartman/confirm_write_new_label boolean true
\ No newline at end of file
--
To view, visit https://gerrit.wikimedia.org/r/328345
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id336e8addae60645026360b54eeab96bb4a6ff37
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add the new user fdans with basic Analytics group permissions
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/327730 ) Change subject: Add the new user fdans with basic Analytics group permissions .. Add the new user fdans with basic Analytics group permissions Francisco Dans is the new member of the Analytics team and he needs basic Analytics production access credentials to do his work. This code review needs a Ops meeting approval since it grants SUDO permissions. Bug: T153303 Change-Id: I4a58c9028943d908f551137a9c734e89f7f9ac2e --- M modules/admin/data/data.yaml 1 file changed, 17 insertions(+), 8 deletions(-) Approvals: Cmjohnson: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index 004c04e..be5fa37 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -112,7 +112,7 @@ jminor, asherman, etonkovidova, sbisson, addshore, matmarex, elukey, nikerabbit, nschaaf, dstrine, joewalsh, mpany, hjiang, jsamra, bcohn, jdittrich, chelsyx, ovasileva, mtizzoni, panisson, paolotti, ciro, debt, - samwalton9, zareen] + samwalton9, zareen, fdans] ldap-admins: gid: 715 description: ldap admins @@ -168,7 +168,7 @@ statistics-web-users: gid: 724 description: access for stats.wikimedia.org -members: [ezachte, joal, milimetric, tomasz] +members: [ezachte, joal, milimetric, tomasz, fdans] statistics-privatedata-users: gid: 725 description: Access to stat boxes that host private data, including @@ -180,7 +180,8 @@ members: [ezachte, milimetric, dartar, yurik, halfak, awight, dr0ptp4kt, nuria, leila, ellery, nettrom, mforns, bmansurov, tbayer, joal, madhuvishy, tjones, legoktm, dcausse, bearloga, dpatrick, atgomez, dstrine, - joewalsh, marktraceur, mtizzoni, panisson, paolotti, ciro, melodykramer, zareen] + joewalsh, marktraceur, mtizzoni, panisson, paolotti, ciro, melodykramer, zareen, + fdans] statistics-users: gid: 726 description: Access statistics number crunching hosts. NO PRIVS. @@ -195,7 +196,7 @@ statistics-admins: posix_name: stats description: access files created by stats user cron jobs -members: [ezachte, milimetric, joal, mforns, bearloga, nuria] +members: [ezachte, milimetric, joal, mforns, bearloga, nuria, fdans] privileges: ['ALL = (stats) NOPASSWD: ALL'] oit: gid: 727 @@ -232,20 +233,20 @@ bmansurov, atgomez, amire80, kartik, pcoombe, foks, mpany, hjiang, jsamra, bcohn, jdlrobson, chelsyx, ovasileva, mtizzoni, panisson, paolotti, ciro, psinger, flemmerich, mkroetzsch, akrausetud, filippo, nschaaf, pmiazga, faidon, - zareen, piccardi] + zareen, piccardi, fdans] analytics-admins: gid: 732 description: Admin access to analytics cluster. This will grant shell access on all Analytics Cluster nodes, as well as the ability to sudo to certain Analytics Cluster system users. -members: [joal, elukey, nuria, mforns, milimetric] +members: [joal, elukey, nuria, mforns, milimetric, fdans] privileges: ['ALL = (hdfs) NOPASSWD: ALL', 'ALL = (oozie) NOPASSWD: ALL', 'ALL = (hive) NOPASSWD: ALL'] eventlogging-admins: gid: 733 description: Login access for EventLogging deployment and investigation -members: [nuria, milimetric, mforns, joal, legoktm, otto, elukey] +members: [nuria, milimetric, mforns, joal, legoktm, otto, elukey, fdans] citoid-admin: gid: 736 description: group of citoid admins @@ -570,7 +571,7 @@ druid-admins: gid: 788 description: admins for the Analytics Druid cluster -members: [nuria, milimetric, joal, mforns] +members: [nuria, milimetric, joal, mforns, fdans] privileges: ['ALL = NOPASSWD: /bin/systemctl start druid-*.service', 'ALL = NOPASSWD: /bin/systemctl stop druid-*.service', 'ALL = NOPASSWD: /bin/systemctl restart druid-*.service', @@ -,3 +2223,11 @@ ssh_keys: - ssh-rsa B3NzaC1yc2EBJQAAAQEAozlVbJ2Cmn3TrZvHChyiHShcWIERWVtl0A+V/w2P5jrpq4WGe2qx/DMPkUDsS5AvhOwY2b9qtAGbx/M+26+b0+xkJihrnT9Fy/IwEDxBFRtcIZ0N1ZvRlK2epb12gfX3oTfQWiSJcmlZOhJCS/GUfeZAAvJiDifnyWeaJAYlkPxPaWVhbF0Xv/W+3zQTQYfPuQEp1S6IKecN3bvW0/8t8fANzbfIjRtwBTSfZiKtHNZwnphaVAPF+xF7wV36spPeRbrxaB6Xy/3pEU00uT7qvyoDpB511w6IVIQfhiAVROVfCbTwyrPtuP+BZHRGFGE7hyo8pFQDQUnEU0yM11NYdQ== uid: 15999 + fdans: +ensure: present +gid: 500 +name: fdans +realname: Francisco Dans +ssh_keys: + - ssh-rsa B3NzaC1yc2EDAQABAAACAQCusvafyr2JyFMUuWJ3N7NMIrKXVGk3dpEhdeqkJIZdRU5+Zyq7sA47bKBP13M06oF
[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable instance restbase1018-b.eqiad.wmnet
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/328192 ) Change subject: enable instance restbase1018-b.eqiad.wmnet .. enable instance restbase1018-b.eqiad.wmnet Bug: T151086 Change-Id: Icd245358ba63127f576f5c937569108dc90c838b --- M hieradata/hosts/restbase1018.yaml 1 file changed, 5 insertions(+), 5 deletions(-) Approvals: Eevans: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/hieradata/hosts/restbase1018.yaml b/hieradata/hosts/restbase1018.yaml index 50e849b..ebd02d8 100644 --- a/hieradata/hosts/restbase1018.yaml +++ b/hieradata/hosts/restbase1018.yaml @@ -6,11 +6,11 @@ listen_address: 10.64.48.98 rpc_address: 10.64.48.98 rpc_interface: eth0 -# b: -#jmx_port: 7190 -#listen_address: 10.64.48.99 -#rpc_address: 10.64.48.99 -#rpc_interface: eth0 + b: +jmx_port: 7190 +listen_address: 10.64.48.99 +rpc_address: 10.64.48.99 +rpc_interface: eth0 # c: #jmx_port: 7191 #listen_address: 10.64.48.100 -- To view, visit https://gerrit.wikimedia.org/r/328192 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Icd245358ba63127f576f5c937569108dc90c838b Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Eevans Gerrit-Reviewer: Dzahn Gerrit-Reviewer: Eevans Gerrit-Reviewer: Elukey Gerrit-Reviewer: Filippo Giunchedi Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add the apache/hhvm prometheus exporter to all the mw hosts
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/328184 )
Change subject: Add the apache/hhvm prometheus exporter to all the mw hosts
..
Add the apache/hhvm prometheus exporter to all the mw hosts
Bug: T147316
Bug: T147316
Change-Id: I007fbf2138804aaf633295eb788585849c5ae11b
---
M modules/role/manifests/mediawiki/appserver.pp
M modules/role/manifests/mediawiki/appserver/api.pp
M modules/role/manifests/mediawiki/jobrunner.pp
M modules/role/manifests/mediawiki/videoscaler.pp
4 files changed, 6 insertions(+), 3 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/84/328184/1
diff --git a/modules/role/manifests/mediawiki/appserver.pp
b/modules/role/manifests/mediawiki/appserver.pp
index 4ddb786..06a0fb4 100644
--- a/modules/role/manifests/mediawiki/appserver.pp
+++ b/modules/role/manifests/mediawiki/appserver.pp
@@ -2,9 +2,6 @@
system::role { 'role::mediawiki::appserver': }
include ::role::mediawiki::webserver
-
-# Test of the new HHVM and Apache Prometheus exporters
-# Bug: T147423, T147316
include ::role::prometheus::apache_exporter
include ::role::prometheus::hhvm_exporter
diff --git a/modules/role/manifests/mediawiki/appserver/api.pp
b/modules/role/manifests/mediawiki/appserver/api.pp
index 934ea08..645aae7 100644
--- a/modules/role/manifests/mediawiki/appserver/api.pp
+++ b/modules/role/manifests/mediawiki/appserver/api.pp
@@ -2,6 +2,8 @@
system::role { 'role::mediawiki::appserver::api': }
include ::role::mediawiki::webserver
+include ::role::prometheus::apache_exporter
+include ::role::prometheus::hhvm_exporter
# Using fastcgi we need more local ports
sysctl::parameters { 'raise_port_range':
diff --git a/modules/role/manifests/mediawiki/jobrunner.pp
b/modules/role/manifests/mediawiki/jobrunner.pp
index 74f6c44..e637bd6 100644
--- a/modules/role/manifests/mediawiki/jobrunner.pp
+++ b/modules/role/manifests/mediawiki/jobrunner.pp
@@ -2,6 +2,8 @@
system::role { 'role::mediawiki::jobrunner': }
include ::role::mediawiki::common
+include ::role::prometheus::apache_exporter
+include ::role::prometheus::hhvm_exporter
include ::mediawiki::jobrunner
monitoring::service { 'jobrunner_http_hhvm':
diff --git a/modules/role/manifests/mediawiki/videoscaler.pp
b/modules/role/manifests/mediawiki/videoscaler.pp
index a65f752..2ab6639 100644
--- a/modules/role/manifests/mediawiki/videoscaler.pp
+++ b/modules/role/manifests/mediawiki/videoscaler.pp
@@ -2,6 +2,8 @@
system::role { 'role::mediawiki::videoscaler': }
include ::role::mediawiki::scaler
+include ::role::prometheus::apache_exporter
+include ::role::prometheus::hhvm_exporter
include ::mediawiki::jobrunner
include ::base::firewall
--
To view, visit https://gerrit.wikimedia.org/r/328184
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I007fbf2138804aaf633295eb788585849c5ae11b
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Repurpose two jobrunners to videoscalers in eqiad
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/328144 )
Change subject: Repurpose two jobrunners to videoscalers in eqiad
..
Repurpose two jobrunners to videoscalers in eqiad
Bug: T153488
Change-Id: I11e8b95f52fa2fde0a03c78cdb0472e3da171ad3
---
M conftool-data/nodes/eqiad.yaml
M manifests/site.pp
M modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
3 files changed, 10 insertions(+), 5 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/conftool-data/nodes/eqiad.yaml b/conftool-data/nodes/eqiad.yaml
index d67c789..d7ebec7 100644
--- a/conftool-data/nodes/eqiad.yaml
+++ b/conftool-data/nodes/eqiad.yaml
@@ -4,6 +4,8 @@
mwdebug1001.eqiad.wmnet: [apache2]
mwdebug1002.eqiad.wmnet: [apache2]
videoscaler:
+ mw1168.eqiad.wmnet: [apache2]
+ mw1169.eqiad.wmnet: [apache2]
mw1259.eqiad.wmnet: [apache2]
mw1260.eqiad.wmnet: [apache2]
mw2152.codfw.wmnet: [apache2]
@@ -16,8 +18,6 @@
mw1165.eqiad.wmnet: [apache2]
mw1166.eqiad.wmnet: [apache2]
mw1167.eqiad.wmnet: [apache2]
- mw1168.eqiad.wmnet: [apache2]
- mw1169.eqiad.wmnet: [apache2]
mw1299.eqiad.wmnet: [apache2]
mw1300.eqiad.wmnet: [apache2]
mw1301.eqiad.wmnet: [apache2]
diff --git a/manifests/site.pp b/manifests/site.pp
index 8d526fb..d14a6cb 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2061,8 +2061,8 @@
include base::firewall
}
-# mw1161-1169 are job runners
-node /^mw116[1-9]\.eqiad\.wmnet$/ {
+# mw1161-1167 are job runners
+node /^mw116[1-7]\.eqiad\.wmnet$/ {
role(mediawiki::jobrunner)
include base::firewall
}
@@ -2098,7 +2098,8 @@
}
#mw1259-60 are videoscalers
-node /^mw12(59|60)\.eqiad\.wmnet/ {
+#mw1168-69 were previously jobrunners (T153488)
+node /^mw1(16[89]|259|260)\.eqiad\.wmnet/ {
role(mediawiki::videoscaler)
}
diff --git a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
index 540dbfe..3c98a6f 100644
--- a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
+++ b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
@@ -3976,11 +3976,15 @@
host mw1168 {
hardware ethernet 90:B1:1C:27:88:5B;
fixed-address mw1168.eqiad.wmnet;
+option pxelinux.pathprefix "trusty-installer/";
+filename "trusty-installer/ubuntu-installer/amd64/pxelinux.0";
}
host mw1169 {
hardware ethernet 90:B1:1C:28:CD:24;
fixed-address mw1169.eqiad.wmnet;
+option pxelinux.pathprefix "trusty-installer/";
+filename "trusty-installer/ubuntu-installer/amd64/pxelinux.0";
}
host mw1170 {
--
To view, visit https://gerrit.wikimedia.org/r/328144
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I11e8b95f52fa2fde0a03c78cdb0472e3da171ad3
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Repurpose two jobrunners to videoscalers in eqiad
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/328144 )
Change subject: Repurpose two jobrunners to videoscalers in eqiad
..
Repurpose two jobrunners to videoscalers in eqiad
Bug: T153488
Change-Id: I11e8b95f52fa2fde0a03c78cdb0472e3da171ad3
---
M conftool-data/nodes/eqiad.yaml
M modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
2 files changed, 6 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/44/328144/1
diff --git a/conftool-data/nodes/eqiad.yaml b/conftool-data/nodes/eqiad.yaml
index d67c789..d7ebec7 100644
--- a/conftool-data/nodes/eqiad.yaml
+++ b/conftool-data/nodes/eqiad.yaml
@@ -4,6 +4,8 @@
mwdebug1001.eqiad.wmnet: [apache2]
mwdebug1002.eqiad.wmnet: [apache2]
videoscaler:
+ mw1168.eqiad.wmnet: [apache2]
+ mw1169.eqiad.wmnet: [apache2]
mw1259.eqiad.wmnet: [apache2]
mw1260.eqiad.wmnet: [apache2]
mw2152.codfw.wmnet: [apache2]
@@ -16,8 +18,6 @@
mw1165.eqiad.wmnet: [apache2]
mw1166.eqiad.wmnet: [apache2]
mw1167.eqiad.wmnet: [apache2]
- mw1168.eqiad.wmnet: [apache2]
- mw1169.eqiad.wmnet: [apache2]
mw1299.eqiad.wmnet: [apache2]
mw1300.eqiad.wmnet: [apache2]
mw1301.eqiad.wmnet: [apache2]
diff --git a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
index 540dbfe..3c98a6f 100644
--- a/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
+++ b/modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200
@@ -3976,11 +3976,15 @@
host mw1168 {
hardware ethernet 90:B1:1C:27:88:5B;
fixed-address mw1168.eqiad.wmnet;
+option pxelinux.pathprefix "trusty-installer/";
+filename "trusty-installer/ubuntu-installer/amd64/pxelinux.0";
}
host mw1169 {
hardware ethernet 90:B1:1C:28:CD:24;
fixed-address mw1169.eqiad.wmnet;
+option pxelinux.pathprefix "trusty-installer/";
+filename "trusty-installer/ubuntu-installer/amd64/pxelinux.0";
}
host mw1170 {
--
To view, visit https://gerrit.wikimedia.org/r/328144
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I11e8b95f52fa2fde0a03c78cdb0472e3da171ad3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable instance restbase1017-c.eqiad.wmnet
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/327745 ) Change subject: enable instance restbase1017-c.eqiad.wmnet .. enable instance restbase1017-c.eqiad.wmnet Bug: T151086 Change-Id: I0e40f1779e43750383a528185abebc165212a56e --- M hieradata/hosts/restbase1017.yaml 1 file changed, 5 insertions(+), 5 deletions(-) Approvals: Eevans: Looks good to me, but someone else must approve Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/hieradata/hosts/restbase1017.yaml b/hieradata/hosts/restbase1017.yaml index dde73eb..afb63e4 100644 --- a/hieradata/hosts/restbase1017.yaml +++ b/hieradata/hosts/restbase1017.yaml @@ -11,8 +11,8 @@ listen_address: 10.64.32.131 rpc_address: 10.64.32.131 rpc_interface: eth0 -# c: -#jmx_port: 7191 -#listen_address: 10.64.32.132 -#rpc_address: 10.64.32.132 -#rpc_interface: eth0 + c: +jmx_port: 7191 +listen_address: 10.64.32.132 +rpc_address: 10.64.32.132 +rpc_interface: eth0 -- To view, visit https://gerrit.wikimedia.org/r/327745 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I0e40f1779e43750383a528185abebc165212a56e Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Eevans Gerrit-Reviewer: Dzahn Gerrit-Reviewer: Eevans Gerrit-Reviewer: Elukey Gerrit-Reviewer: Filippo Giunchedi Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Apply the hhvm/apache prometheus exporter roles to the MW ap...
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/327734 )
Change subject: Apply the hhvm/apache prometheus exporter roles to the MW
appservers
..
Apply the hhvm/apache prometheus exporter roles to the MW appservers
Bug: T147423
Bug: T147316
Change-Id: I8c8bf1d8fc96386b56fc455f7b3991638356d35a
---
M manifests/site.pp
M modules/role/manifests/mediawiki/appserver.pp
2 files changed, 6 insertions(+), 1 deletion(-)
Approvals:
Giuseppe Lavagetto: Looks good to me, but someone else must approve
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/site.pp b/manifests/site.pp
index a3f1bc9..8d526fb 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2050,7 +2050,7 @@
# mwdebug servers are for mediawiki testing/debugging
# They replace mw1017 and mw1099
node /^mwdebug100[12]\.eqiad\.wmnet$/ {
-role(mediawiki::canary_appserver, prometheus::hhvm_exporter,
prometheus::apache_exporter)
+role(mediawiki::canary_appserver)
include base::firewall
}
diff --git a/modules/role/manifests/mediawiki/appserver.pp
b/modules/role/manifests/mediawiki/appserver.pp
index 0dc2aa5..4ddb786 100644
--- a/modules/role/manifests/mediawiki/appserver.pp
+++ b/modules/role/manifests/mediawiki/appserver.pp
@@ -3,4 +3,9 @@
include ::role::mediawiki::webserver
+# Test of the new HHVM and Apache Prometheus exporters
+# Bug: T147423, T147316
+include ::role::prometheus::apache_exporter
+include ::role::prometheus::hhvm_exporter
+
}
--
To view, visit https://gerrit.wikimedia.org/r/327734
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I8c8bf1d8fc96386b56fc455f7b3991638356d35a
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Filippo Giunchedi
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Apply the hhvm/apache prometheus exporter roles to the MW ap...
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/327734 ) Change subject: Apply the hhvm/apache prometheus exporter roles to the MW appservers .. Apply the hhvm/apache prometheus exporter roles to the MW appservers Bug: T147423 Bug: T147316 Change-Id: I8c8bf1d8fc96386b56fc455f7b3991638356d35a --- M modules/role/manifests/mediawiki/appserver.pp 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/34/327734/1 diff --git a/modules/role/manifests/mediawiki/appserver.pp b/modules/role/manifests/mediawiki/appserver.pp index 0dc2aa5..4ddb786 100644 --- a/modules/role/manifests/mediawiki/appserver.pp +++ b/modules/role/manifests/mediawiki/appserver.pp @@ -3,4 +3,9 @@ include ::role::mediawiki::webserver +# Test of the new HHVM and Apache Prometheus exporters +# Bug: T147423, T147316 +include ::role::prometheus::apache_exporter +include ::role::prometheus::hhvm_exporter + } -- To view, visit https://gerrit.wikimedia.org/r/327734 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I8c8bf1d8fc96386b56fc455f7b3991638356d35a Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add the new user fdans with basic Analytics group permissions
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/327730 ) Change subject: Add the new user fdans with basic Analytics group permissions .. Add the new user fdans with basic Analytics group permissions Francisco Dans is the new member of the Analytics team and he needs basic Analytics production access credentials to do his work. This code review needs a Ops meeting approval since it grants SUDO permissions. Bug: T153303 Change-Id: I4a58c9028943d908f551137a9c734e89f7f9ac2e --- M modules/admin/data/data.yaml 1 file changed, 17 insertions(+), 8 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/30/327730/1 diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index 004c04e..be5fa37 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -112,7 +112,7 @@ jminor, asherman, etonkovidova, sbisson, addshore, matmarex, elukey, nikerabbit, nschaaf, dstrine, joewalsh, mpany, hjiang, jsamra, bcohn, jdittrich, chelsyx, ovasileva, mtizzoni, panisson, paolotti, ciro, debt, - samwalton9, zareen] + samwalton9, zareen, fdans] ldap-admins: gid: 715 description: ldap admins @@ -168,7 +168,7 @@ statistics-web-users: gid: 724 description: access for stats.wikimedia.org -members: [ezachte, joal, milimetric, tomasz] +members: [ezachte, joal, milimetric, tomasz, fdans] statistics-privatedata-users: gid: 725 description: Access to stat boxes that host private data, including @@ -180,7 +180,8 @@ members: [ezachte, milimetric, dartar, yurik, halfak, awight, dr0ptp4kt, nuria, leila, ellery, nettrom, mforns, bmansurov, tbayer, joal, madhuvishy, tjones, legoktm, dcausse, bearloga, dpatrick, atgomez, dstrine, - joewalsh, marktraceur, mtizzoni, panisson, paolotti, ciro, melodykramer, zareen] + joewalsh, marktraceur, mtizzoni, panisson, paolotti, ciro, melodykramer, zareen, + fdans] statistics-users: gid: 726 description: Access statistics number crunching hosts. NO PRIVS. @@ -195,7 +196,7 @@ statistics-admins: posix_name: stats description: access files created by stats user cron jobs -members: [ezachte, milimetric, joal, mforns, bearloga, nuria] +members: [ezachte, milimetric, joal, mforns, bearloga, nuria, fdans] privileges: ['ALL = (stats) NOPASSWD: ALL'] oit: gid: 727 @@ -232,20 +233,20 @@ bmansurov, atgomez, amire80, kartik, pcoombe, foks, mpany, hjiang, jsamra, bcohn, jdlrobson, chelsyx, ovasileva, mtizzoni, panisson, paolotti, ciro, psinger, flemmerich, mkroetzsch, akrausetud, filippo, nschaaf, pmiazga, faidon, - zareen, piccardi] + zareen, piccardi, fdans] analytics-admins: gid: 732 description: Admin access to analytics cluster. This will grant shell access on all Analytics Cluster nodes, as well as the ability to sudo to certain Analytics Cluster system users. -members: [joal, elukey, nuria, mforns, milimetric] +members: [joal, elukey, nuria, mforns, milimetric, fdans] privileges: ['ALL = (hdfs) NOPASSWD: ALL', 'ALL = (oozie) NOPASSWD: ALL', 'ALL = (hive) NOPASSWD: ALL'] eventlogging-admins: gid: 733 description: Login access for EventLogging deployment and investigation -members: [nuria, milimetric, mforns, joal, legoktm, otto, elukey] +members: [nuria, milimetric, mforns, joal, legoktm, otto, elukey, fdans] citoid-admin: gid: 736 description: group of citoid admins @@ -570,7 +571,7 @@ druid-admins: gid: 788 description: admins for the Analytics Druid cluster -members: [nuria, milimetric, joal, mforns] +members: [nuria, milimetric, joal, mforns, fdans] privileges: ['ALL = NOPASSWD: /bin/systemctl start druid-*.service', 'ALL = NOPASSWD: /bin/systemctl stop druid-*.service', 'ALL = NOPASSWD: /bin/systemctl restart druid-*.service', @@ -,3 +2223,11 @@ ssh_keys: - ssh-rsa B3NzaC1yc2EBJQAAAQEAozlVbJ2Cmn3TrZvHChyiHShcWIERWVtl0A+V/w2P5jrpq4WGe2qx/DMPkUDsS5AvhOwY2b9qtAGbx/M+26+b0+xkJihrnT9Fy/IwEDxBFRtcIZ0N1ZvRlK2epb12gfX3oTfQWiSJcmlZOhJCS/GUfeZAAvJiDifnyWeaJAYlkPxPaWVhbF0Xv/W+3zQTQYfPuQEp1S6IKecN3bvW0/8t8fANzbfIjRtwBTSfZiKtHNZwnphaVAPF+xF7wV36spPeRbrxaB6Xy/3pEU00uT7qvyoDpB511w6IVIQfhiAVROVfCbTwyrPtuP+BZHRGFGE7hyo8pFQDQUnEU0yM11NYdQ== uid: 15999 + fdans: +ensure: present +gid: 500 +name: fdans +realname: Francisco Dans +ssh_keys: + - ssh-rsa B3NzaC1yc2EDAQABAAACAQCusvafyr2JyFMUuWJ3N7NMIrKXVGk3dpEhdeqkJIZdRU5+Zyq7sA47bKBP13M06oFhtDQv/77rsqKSS1zy4CSFs9RPR/WDzeRcuHRxxWbtA0+6A0pfmvxCf
[MediaWiki-commits] [Gerrit] operations/puppet[production]: enable instance restbase1017-a.codfw.wmnet
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/327520 ) Change subject: enable instance restbase1017-a.codfw.wmnet .. enable instance restbase1017-a.codfw.wmnet Bug: T151086 Change-Id: Icf1d247b4d7c0b4f10751fac47efd455edebacb6 --- M hieradata/hosts/restbase1017.yaml 1 file changed, 6 insertions(+), 6 deletions(-) Approvals: Eevans: Looks good to me, but someone else must approve Elukey: Verified; Looks good to me, approved diff --git a/hieradata/hosts/restbase1017.yaml b/hieradata/hosts/restbase1017.yaml index be53b8a..8642d19 100644 --- a/hieradata/hosts/restbase1017.yaml +++ b/hieradata/hosts/restbase1017.yaml @@ -1,11 +1,11 @@ cassandra::rack: 'b' cassandra::num_tokens: 256 -#cassandra::instances: -# a: -#jmx_port: 7189 -#listen_address: 10.64.32.130 -#rpc_address: 10.64.32.130 -#rpc_interface: eth0 +cassandra::instances: + a: +jmx_port: 7189 +listen_address: 10.64.32.130 +rpc_address: 10.64.32.130 +rpc_interface: eth0 # b: #jmx_port: 7190 #listen_address: 10.64.32.131 -- To view, visit https://gerrit.wikimedia.org/r/327520 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Icf1d247b4d7c0b4f10751fac47efd455edebacb6 Gerrit-PatchSet: 2 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Eevans Gerrit-Reviewer: Dzahn Gerrit-Reviewer: Eevans Gerrit-Reviewer: Elukey Gerrit-Reviewer: Filippo Giunchedi Gerrit-Reviewer: Mobrovac Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add the prometheus-apache-exporter module and role
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/327240 )
Change subject: Add the prometheus-apache-exporter module and role
..
Add the prometheus-apache-exporter module and role
Bug: T147316
Change-Id: If66ba8e532f3d9ea7b925e44ae360dbffd3b330a
---
A modules/prometheus/manifests/apache_exporter.pp
A modules/role/manifests/prometheus/apache_exporter.pp
2 files changed, 46 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/40/327240/1
diff --git a/modules/prometheus/manifests/apache_exporter.pp
b/modules/prometheus/manifests/apache_exporter.pp
new file mode 100644
index 000..ad8ed2a
--- /dev/null
+++ b/modules/prometheus/manifests/apache_exporter.pp
@@ -0,0 +1,28 @@
+# == Define: prometheus::apache_exporter
+#
+# Prometheus exporter for Apache httpd server metrics.
+#
+# = Parameters
+#
+# [*arguments*]
+# Additional command line arguments for prometheus-apache-exporter.
+
+define prometheus::apache_exporter (
+$arguments = '-scrape_uri \'http://127.0.0.1/server-status/?auto\'',
+) {
+require_package('prometheus-apache-exporter')
+
+file { '/etc/default/prometheus-apache-exporter':
+ensure => present,
+mode=> '0444',
+owner => 'root',
+group => 'root',
+content => "ARGS=\"${arguments}\"",
+notify => Service['prometheus-apache-exporter'],
+}
+
+service { 'prometheus-apache-exporter':
+ensure => running,
+require => Package['prometheus-apache-exporter'],
+}
+}
diff --git a/modules/role/manifests/prometheus/apache_exporter.pp
b/modules/role/manifests/prometheus/apache_exporter.pp
new file mode 100644
index 000..81f6d32
--- /dev/null
+++ b/modules/role/manifests/prometheus/apache_exporter.pp
@@ -0,0 +1,18 @@
+class role::prometheus::apache_exporter {
+prometheus::apache_exporter { 'default': }
+
+if $::realm == 'labs' {
+$ferm_srange = '$LABS_NETWORKS'
+} else {
+$prometheus_nodes = hiera('prometheus_nodes')
+$prometheus_ferm_nodes = join($prometheus_nodes, ' ')
+$ferm_srange = "(@resolve((${prometheus_ferm_nodes}))
@resolve((${prometheus_ferm_nodes}), ))"
+}
+
+ferm::service { 'prometheus-apache_exporter':
+proto => 'tcp',
+port => '9117',
+srange => $ferm_srange,
+}
+}
+
--
To view, visit https://gerrit.wikimedia.org/r/327240
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If66ba8e532f3d9ea7b925e44ae360dbffd3b330a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...prometheus-apache-exporter[master]: Add upstream source files
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/327020 )
Change subject: Add upstream source files
..
Add upstream source files
Change-Id: Ided340c6d56a11b90d12c731dacda7c5b186b25d
---
A LICENSE
A README.md
A apache_exporter.go
A apache_exporter_test.go
4 files changed, 474 insertions(+), 0 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-apache-exporter
refs/changes/20/327020/1
diff --git a/LICENSE b/LICENSE
new file mode 100644
index 000..8792d9f
--- /dev/null
+++ b/LICENSE
@@ -0,0 +1,22 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 neezgee
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+
diff --git a/README.md b/README.md
new file mode 100644
index 000..90496a4
--- /dev/null
+++ b/README.md
@@ -0,0 +1,23 @@
+# Apache Exporter for Prometheus
+
+Exports apache mod_status statistics via HTTP for Prometheus consumption.
+
+With working golang environment it can be built with `go get`. There is a
[good
article](https://machineperson.github.io/monitoring/2016/01/04/exporting-apache-metrics-to-prometheus.html)
with build HOWTO and usage example.
+
+Help on flags:
+
+```
+ -insecure
+ Ignore server certificate if using https (default false)
+ -log.level value
+ Only log messages with the given severity or above. Valid levels:
[debug, info, warn, error, fatal, panic]. (default info)
+ -scrape_uri string
+ URI to apache stub status page (default
"http://localhost/server-status/?auto";)
+ -telemetry.address string
+ Address on which to expose metrics. (default ":9117")
+ -telemetry.endpoint string
+ Path under which to expose metrics. (default "/metrics")
+```
+
+Tested on Apache 2.2 and Apache 2.4.
+
diff --git a/apache_exporter.go b/apache_exporter.go
new file mode 100644
index 000..321c1f3
--- /dev/null
+++ b/apache_exporter.go
@@ -0,0 +1,286 @@
+package main
+
+import (
+ "crypto/tls"
+ "flag"
+ "fmt"
+ "io/ioutil"
+ "net/http"
+ "strconv"
+ "strings"
+ "sync"
+
+ "github.com/prometheus/client_golang/prometheus"
+ "github.com/prometheus/common/log"
+)
+
+const (
+ namespace = "apache" // For Prometheus metrics.
+)
+
+var (
+ listeningAddress = flag.String("telemetry.address", ":9117", "Address
on which to expose metrics.")
+ metricsEndpoint = flag.String("telemetry.endpoint", "/metrics", "Path
under which to expose metrics.")
+ scrapeURI= flag.String("scrape_uri",
"http://localhost/server-status/?auto";, "URI to apache stub status page.")
+ insecure = flag.Bool("insecure", false, "Ignore server
certificate if using https.")
+)
+
+type Exporter struct {
+ URIstring
+ mutex sync.Mutex
+ client *http.Client
+
+ up *prometheus.Desc
+ scrapeFailures prometheus.Counter
+ accessesTotal *prometheus.Desc
+ kBytesTotal*prometheus.Desc
+ uptime *prometheus.Desc
+ workers*prometheus.GaugeVec
+ scoreboard *prometheus.GaugeVec
+ connections*prometheus.GaugeVec
+}
+
+func NewExporter(uri string) *Exporter {
+ return &Exporter{
+ URI: uri,
+ up: prometheus.NewDesc(
+prometheus.BuildFQName(namespace, "", "up"),
+"Could the apache server be reached",
+nil,
+ nil),
+scrapeFailures: prometheus.NewCounter(prometheus.CounterOpts{
+Namespace: namespace,
+Name: "exporter_scrape_failures_total",
+Help: "Number of errors while scraping apache.",
+}),
+accessesTotal: prometheus.NewDesc(
+prometheus.BuildFQName(namespace, "",
"accesses_total"),
+"Cu
[MediaWiki-commits] [Gerrit] operations...prometheus-apache-exporter[master]: Initial debianization
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/325568 )
Change subject: Initial debianization
..
Initial debianization
Bug: T147316
Change-Id: I14d46a8998ca25f1d376065bb196b8286d3ac4cd
---
A debian/changelog
A debian/compat
A debian/control
A debian/copyright
A debian/default
A debian/dirs
A debian/docs
A debian/gdb.conf
A debian/init
A debian/postinst
A debian/postrm
A debian/rules
A debian/service
A debian/watch
14 files changed, 208 insertions(+), 0 deletions(-)
Approvals:
Elukey: Verified; Looks good to me, approved
Filippo Giunchedi: Looks good to me, but someone else must approve
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 000..a80c7ce
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+prometheus-apache-exporter (0.3-1) jessie-wikimedia; urgency=low
+
+ * Initial Release, packaged the 0.3 upstream version.
+
+ -- Luca Toscano Thu, 03 Nov 2016 15:13:55 +0100
diff --git a/debian/compat b/debian/compat
new file mode 100644
index 000..ec63514
--- /dev/null
+++ b/debian/compat
@@ -0,0 +1 @@
+9
diff --git a/debian/control b/debian/control
new file mode 100644
index 000..0b20b9c
--- /dev/null
+++ b/debian/control
@@ -0,0 +1,16 @@
+Source: prometheus-apache-exporter
+Section: net
+Priority: extra
+Maintainer: Luca Toscano
+Build-Depends: debhelper (>= 9),
+ dh-golang,
+ dh-systemd (>= 1.5),
+ golang-go,
+ golang-github-prometheus-client-golang-dev
+Standards-Version: 3.9.8
+Homepage: https://github.com/neezgee/apache_exporter
+
+Package: prometheus-apache-exporter
+Architecture: any
+Depends: ${shlibs:Depends}, ${misc:Depends}
+Description: Exports apache mod_status statistics via HTTP for Prometheus
consumption.
diff --git a/debian/copyright b/debian/copyright
new file mode 100644
index 000..9f4877a
--- /dev/null
+++ b/debian/copyright
@@ -0,0 +1,27 @@
+Files: *
+Copyright: 2016 neezgee
+License: MIT
+
+Files: debian/*
+Copyright: 2016 Luca Toscano
+License: MIT
+Comment: Debian packaging is licensed under the same terms as upstream
+
+License: MIT
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+ .
+ The above copyright notice and this permission notice shall be included in all
+ copies or substantial portions of the Software.
+ .
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ SOFTWARE.
diff --git a/debian/default b/debian/default
new file mode 100644
index 000..406a2eb
--- /dev/null
+++ b/debian/default
@@ -0,0 +1,14 @@
+ARGS=""
+
+# Usage of prometheus-apache-exporter:
+#
+# -insecure
+#Ignore server certificate if using https (default false)
+# -log.level value
+#Only log messages with the given severity or above. Valid levels:
[debug, info, warn, error, fatal, panic]. (default info)
+# -scrape_uri string
+#URI to apache stub status page (default
"http://localhost/server-status/?auto";)
+# -telemetry.address string
+#Address on which to expose metrics. (default ":9117")
+# -telemetry.endpoint string
+#Path under which to expose metrics. (default "/metrics")
diff --git a/debian/dirs b/debian/dirs
new file mode 100644
index 000..557ffa1
--- /dev/null
+++ b/debian/dirs
@@ -0,0 +1 @@
+var/log/prometheus
diff --git a/debian/docs b/debian/docs
new file mode 100644
index 000..b43bf86
--- /dev/null
+++ b/debian/docs
@@ -0,0 +1 @@
+README.md
diff --git a/debian/gdb.conf b/debian/gdb.conf
new file mode 100644
index 000..f692a5d
--- /dev/null
+++ b/debian/gdb.conf
@@ -0,0 +1,2 @@
+[buildpackage]
+upstream-tag=v%(version)s
diff --git a/debian/init b/debian/init
new file mode 100755
index 000..0cce939
--- /dev/null
+++ b/debian/init
@@ -0,0 +1,64 @@
+#!/bin/sh
+# kFreeBSD do not accept scripts as interpreters, using #!/bin/sh and sourcing.
+if [ true != "$INIT_D_SCRIPT_SOURCED" ] ; then
+set "$0" "$@"; INIT_D_SCRIPT_SOURCED=true . /lib/init/init-d-script
+fi
+### BEGIN INIT INFO
+# Provides: prometheus-apache-exporter
+# Required-Start:$remote_fs $syslog
+# Required-Stop: $remote_fs $syslog
+# Default-S
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove no longer needed statistics::migration role
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/326489 )
Change subject: Remove no longer needed statistics::migration role
..
Remove no longer needed statistics::migration role
Change-Id: I7b7fcb6b08ef7b62616124717c20835af393a941
---
M manifests/site.pp
D modules/role/manifests/statistics/migration.pp
2 files changed, 1 insertion(+), 43 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/site.pp b/manifests/site.pp
index b19df21..9b565dd 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -2734,8 +2734,7 @@
# stat1004 contains all the tools and libraries to access
# the Analytics Cluster services.
-role(analytics_cluster::client, analytics_cluster::refinery,
-statistics::migration)
+role(analytics_cluster::client, analytics_cluster::refinery)
include standard
}
diff --git a/modules/role/manifests/statistics/migration.pp
b/modules/role/manifests/statistics/migration.pp
deleted file mode 100644
index afd872b..000
--- a/modules/role/manifests/statistics/migration.pp
+++ /dev/null
@@ -1,41 +0,0 @@
-# setup rsync to copy home dirs for server upgrade
-class role::statistics::migration {
-
-$sourceip='10.64.21.101'
-
-ferm::service { 'stat-migration-rsync':
-proto => 'tcp',
-port => '873',
-srange => "${sourceip}/32",
-}
-
-include rsync::server
-
-file { [
-'/srv/stat1001',
-'/srv/stat1001/home',
-'/srv/stat1001/var',
-'/srv/stat1001/var/www',
-'/srv/stat1001/srv',
-]:
-ensure => 'directory',
-}
-
-rsync::server::module { 'home':
-path=> '/srv/stat1001/home',
-read_only => 'no',
-hosts_allow => $sourceip,
-}
-
-rsync::server::module { 'varwww':
-path=> '/srv/stat1001/var/www',
-read_only => 'no',
-hosts_allow => $sourceip,
-}
-
-rsync::server::module { 'srv':
-path=> '/srv/stat1001/srv',
-read_only => 'no',
-hosts_allow => $sourceip,
-}
-}
--
To view, visit https://gerrit.wikimedia.org/r/326489
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7b7fcb6b08ef7b62616124717c20835af393a941
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ottomata
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set systemd dependency correctly for vk statsv/el instances
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/326904 )
Change subject: Set systemd dependency correctly for vk statsv/el instances
..
Set systemd dependency correctly for vk statsv/el instances
While checking Varnishkafka graphs in Grafana I noticed that
the varnishkafka statsv and eventlogging instances were restarting
at the same time sporadically, meanwhile this didn't happen for
the webrequest instance. I checked their systemd units and
statsv/eventlogging are dependend on the varnish instance, not
the varnish-frontend one.
Add also some documentation to the role parameters.
Change-Id: I3ebf095c2d77dbad4541e6a2808baf345454e561
---
M modules/role/manifests/cache/kafka/eventlogging.pp
M modules/role/manifests/cache/kafka/statsv.pp
M modules/role/manifests/cache/kafka/webrequest.pp
M modules/role/manifests/cache/text.pp
4 files changed, 44 insertions(+), 19 deletions(-)
Approvals:
Ema: Looks good to me, but someone else must approve
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/role/manifests/cache/kafka/eventlogging.pp
b/modules/role/manifests/cache/kafka/eventlogging.pp
index c0e2b8b..80bcacd 100644
--- a/modules/role/manifests/cache/kafka/eventlogging.pp
+++ b/modules/role/manifests/cache/kafka/eventlogging.pp
@@ -1,6 +1,22 @@
+# === Define role::cache::kafka::eventlogging
+#
+# Sets up a varnishkafka logging endpoint for collecting
+# analytics events coming from external clients.
+#
+# More info: https://wikitech.wikimedia.org/wiki/Analytics/EventLogging
+#
+# === Parameters
+#
+# [*varnish_name*]
+# The name of the varnish instance to read shared logs from.
+# Default 'frontend'
+# [*varnish_svc_name*]
+# The name of the init unit for the above.
+# Default 'varnish-frontend'
+#
class role::cache::kafka::eventlogging(
-$varnish_name = $::hostname,
-$varnish_svc_name = 'varnish',
+$varnish_name = 'frontend',
+$varnish_svc_name = 'varnish-frontend'
) inherits role::cache::kafka
{
# Set varnish.arg.q or varnish.arg.m according to Varnish version
diff --git a/modules/role/manifests/cache/kafka/statsv.pp
b/modules/role/manifests/cache/kafka/statsv.pp
index 29f3e44..1ceb76d 100644
--- a/modules/role/manifests/cache/kafka/statsv.pp
+++ b/modules/role/manifests/cache/kafka/statsv.pp
@@ -1,16 +1,22 @@
-# == Class role::cache::kafka::statsv
+# === Define role::cache::kafka::statsv
+#
# Sets up a varnishkafka logging endpoint for collecting
# application level metrics. We are calling this system
# statsv, as it is similar to statsd, but uses varnish
# as its logging endpoint.
#
-# == Parameters
-# $varnish_name - the name of the varnish instance to read shared logs from.
Default $::hostname
-# $varnish_svc_name - the name of the varnish init service to read shared logs
from. Default 'varnish'
+# === Parameters
+#
+# [*varnish_name*]
+# The name of the varnish instance to read shared logs from.
+# Default 'frontend'
+# [*varnish_svc_name*]
+# The name of the init unit for the above.
+# Default 'varnish-frontend'
#
class role::cache::kafka::statsv(
-$varnish_name = $::hostname,
-$varnish_svc_name = 'varnish',
+$varnish_name = 'frontend',
+$varnish_svc_name = 'varnish-frontend'
) inherits role::cache::kafka
{
$format = "%{fake_tag0@hostname?${::fqdn}}x %{%FT%T@dt}t
%{X-Client-IP@ip}o %{@uri_path}U %{@uri_query}q %{User-Agent@user_agent}i"
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 4ea2ced..0d6841f 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -1,11 +1,18 @@
-# == Class role::cache::kafka::webrequest
+# === Define role::cache::kafka::webrequest
+#
# Sets up a varnishkafka instance producing varnish
# webrequest logs to the analytics Kafka brokers in eqiad.
#
-# == Parameters
-# $topic- the name of kafka topic to which to send messages
-# $varnish_name - the name of the varnish instance to read shared logs from.
Default 'frontend'
-# $varnish_svc_name - the name of the init unit for the above, default
'varnish-frontend'
+# === Parameters
+#
+# [*topic*]
+# the name of kafka topic to which to send messages.
+# [*varnish_name*]
+# The name of the varnish instance to read shared logs from.
+# Default 'frontend'
+# [*varnish_svc_name*]
+# The name of the init unit for the above.
+# Default 'varnish-frontend'
#
class role::cache::kafka::webrequest(
$topic,
diff --git a/modules/role/manifests/cache/text.pp
b/modules/role/manifests/cache/text.pp
index 249c76d..1492793 100644
--- a/modules/role/manifests/cache/text.pp
+++ b/modules/role/manifests/cache/text.pp
@@ -121,18 +121,14 @@
# and sends them to the 'statsv' topic in Kafka.
# A kafka consumer then consumes these and emits
# metrics.
-cla
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Set systemd dependency correctly for vk statsv/el instances
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/326904 )
Change subject: Set systemd dependency correctly for vk statsv/el instances
..
Set systemd dependency correctly for vk statsv/el instances
While checking Varnishkafka graphs in Grafana I noticed that
the varnishkafka statsv and eventlogging instances were restarting
at the same time sporadically, meanwhile this didn't happen for
the webrequest instance. I checked their systemd units and
statsv/eventlogging are dependend on the varnish instance, not
the varnish-frontend one.
Add also some documentation to the role parameters.
Change-Id: I3ebf095c2d77dbad4541e6a2808baf345454e561
---
M modules/role/manifests/cache/kafka/eventlogging.pp
M modules/role/manifests/cache/kafka/statsv.pp
M modules/role/manifests/cache/kafka/webrequest.pp
3 files changed, 42 insertions(+), 13 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/04/326904/1
diff --git a/modules/role/manifests/cache/kafka/eventlogging.pp
b/modules/role/manifests/cache/kafka/eventlogging.pp
index c0e2b8b..98bc818 100644
--- a/modules/role/manifests/cache/kafka/eventlogging.pp
+++ b/modules/role/manifests/cache/kafka/eventlogging.pp
@@ -1,6 +1,22 @@
+# == Class role::cache::kafka::eventlogging
+#
+# Sets up a varnishkafka logging endpoint for collecting
+# analytics events coming from external clients.
+#
+# More info: https://wikitech.wikimedia.org/wiki/Analytics/EventLogging
+#
+# === Parameters
+#
+# [*varnish_name*]
+# The name of the varnish instance to read shared logs from.
+# Default 'frontend'
+# [*varnish_svc_name*]
+# The name of the init unit for the above.
+# Default 'varnish-frontend'
+#
class role::cache::kafka::eventlogging(
-$varnish_name = $::hostname,
-$varnish_svc_name = 'varnish',
+$varnish_name = 'frontend',
+$varnish_svc_name = 'varnish-frontend'
) inherits role::cache::kafka
{
# Set varnish.arg.q or varnish.arg.m according to Varnish version
diff --git a/modules/role/manifests/cache/kafka/statsv.pp
b/modules/role/manifests/cache/kafka/statsv.pp
index 29f3e44..1ceb76d 100644
--- a/modules/role/manifests/cache/kafka/statsv.pp
+++ b/modules/role/manifests/cache/kafka/statsv.pp
@@ -1,16 +1,22 @@
-# == Class role::cache::kafka::statsv
+# === Define role::cache::kafka::statsv
+#
# Sets up a varnishkafka logging endpoint for collecting
# application level metrics. We are calling this system
# statsv, as it is similar to statsd, but uses varnish
# as its logging endpoint.
#
-# == Parameters
-# $varnish_name - the name of the varnish instance to read shared logs from.
Default $::hostname
-# $varnish_svc_name - the name of the varnish init service to read shared logs
from. Default 'varnish'
+# === Parameters
+#
+# [*varnish_name*]
+# The name of the varnish instance to read shared logs from.
+# Default 'frontend'
+# [*varnish_svc_name*]
+# The name of the init unit for the above.
+# Default 'varnish-frontend'
#
class role::cache::kafka::statsv(
-$varnish_name = $::hostname,
-$varnish_svc_name = 'varnish',
+$varnish_name = 'frontend',
+$varnish_svc_name = 'varnish-frontend'
) inherits role::cache::kafka
{
$format = "%{fake_tag0@hostname?${::fqdn}}x %{%FT%T@dt}t
%{X-Client-IP@ip}o %{@uri_path}U %{@uri_query}q %{User-Agent@user_agent}i"
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 4ea2ced..0d6841f 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -1,11 +1,18 @@
-# == Class role::cache::kafka::webrequest
+# === Define role::cache::kafka::webrequest
+#
# Sets up a varnishkafka instance producing varnish
# webrequest logs to the analytics Kafka brokers in eqiad.
#
-# == Parameters
-# $topic- the name of kafka topic to which to send messages
-# $varnish_name - the name of the varnish instance to read shared logs from.
Default 'frontend'
-# $varnish_svc_name - the name of the init unit for the above, default
'varnish-frontend'
+# === Parameters
+#
+# [*topic*]
+# the name of kafka topic to which to send messages.
+# [*varnish_name*]
+# The name of the varnish instance to read shared logs from.
+# Default 'frontend'
+# [*varnish_svc_name*]
+# The name of the init unit for the above.
+# Default 'varnish-frontend'
#
class role::cache::kafka::webrequest(
$topic,
--
To view, visit https://gerrit.wikimedia.org/r/326904
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3ebf095c2d77dbad4541e6a2808baf345454e561
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikim
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add the eventlogging admins back to eventlog1001
Elukey has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/326085 ) Change subject: Add the eventlogging admins back to eventlog1001 .. Add the eventlogging admins back to eventlog1001 They got removed as a consequence of https://gerrit.wikimedia.org/r/#/c/326083 (the eventlogging role in hiera defines the admins). Bug: T152621 Change-Id: I19a6c7e00f4e1bc0247f2df60342e6d1eeebe543 --- A hieradata/hosts/eventlog1001.yaml 1 file changed, 3 insertions(+), 0 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/hieradata/hosts/eventlog1001.yaml b/hieradata/hosts/eventlog1001.yaml new file mode 100644 index 000..d6909c8 --- /dev/null +++ b/hieradata/hosts/eventlog1001.yaml @@ -0,0 +1,3 @@ +admin::groups: + - eventlogging-admins + - eventlogging-roots \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/326085 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I19a6c7e00f4e1bc0247f2df60342e6d1eeebe543 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Ottomata Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add the eventlogging admins back to eventlog1001
Elukey has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/326085 ) Change subject: Add the eventlogging admins back to eventlog1001 .. Add the eventlogging admins back to eventlog1001 They got removed as a consequence of https://gerrit.wikimedia.org/r/#/c/326083 (the eventlogging role in hiera defines the admins). Bug: T152621 Change-Id: I19a6c7e00f4e1bc0247f2df60342e6d1eeebe543 --- A hieradata/hosts/eventlog1001.yaml 1 file changed, 3 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/85/326085/1 diff --git a/hieradata/hosts/eventlog1001.yaml b/hieradata/hosts/eventlog1001.yaml new file mode 100644 index 000..d6909c8 --- /dev/null +++ b/hieradata/hosts/eventlog1001.yaml @@ -0,0 +1,3 @@ +admin::groups: + - eventlogging-admins + - eventlogging-roots \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/326085 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I19a6c7e00f4e1bc0247f2df60342e6d1eeebe543 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove the role eventlogging from site.pp
Elukey has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/326083 )
Change subject: Remove the role eventlogging from site.pp
..
Remove the role eventlogging from site.pp
Bug: T152621
Change-Id: I50393ea4fcee458109cde00b395c61e1fd547f38
---
M manifests/site.pp
1 file changed, 1 insertion(+), 2 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/site.pp b/manifests/site.pp
index 15ca647..b19df21 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1099,8 +1099,7 @@
# Receives log data from Kafka processes it, and broadcasts
# to Kafka Schema based topics.
node 'eventlog1001.eqiad.wmnet' {
-role(eventlogging,
-eventlogging::analytics::zeromq,
+role(eventlogging::analytics::zeromq,
eventlogging::analytics::processor,
eventlogging::analytics::mysql,
eventlogging::analytics::files,
--
To view, visit https://gerrit.wikimedia.org/r/326083
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I50393ea4fcee458109cde00b395c61e1fd547f38
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove the role eventlogging from site.pp
Elukey has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/326083 )
Change subject: Remove the role eventlogging from site.pp
..
Remove the role eventlogging from site.pp
Bug: T152621
Change-Id: I50393ea4fcee458109cde00b395c61e1fd547f38
---
M manifests/site.pp
1 file changed, 1 insertion(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/83/326083/1
diff --git a/manifests/site.pp b/manifests/site.pp
index 15ca647..b19df21 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1099,8 +1099,7 @@
# Receives log data from Kafka processes it, and broadcasts
# to Kafka Schema based topics.
node 'eventlog1001.eqiad.wmnet' {
-role(eventlogging,
-eventlogging::analytics::zeromq,
+role(eventlogging::analytics::zeromq,
eventlogging::analytics::processor,
eventlogging::analytics::mysql,
eventlogging::analytics::files,
--
To view, visit https://gerrit.wikimedia.org/r/326083
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I50393ea4fcee458109cde00b395c61e1fd547f38
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: MW apache: remove bits.wm.o vhost
Elukey has submitted this change and it was merged. Change subject: MW apache: remove bits.wm.o vhost .. MW apache: remove bits.wm.o vhost Bug: T107430 Change-Id: I93af7b2874d2636c358a64b180dc0b78a2fd5837 --- M modules/mediawiki/files/apache/sites/wikimedia.conf 1 file changed, 0 insertions(+), 19 deletions(-) Approvals: Krinkle: Looks good to me, but someone else must approve Chad: Looks good to me, but someone else must approve Elukey: Verified; Looks good to me, approved Giuseppe Lavagetto: Looks good to me, but someone else must approve Alexandros Kosiaris: Looks good to me, but someone else must approve jenkins-bot: Verified BBlack: Looks good to me, but someone else must approve diff --git a/modules/mediawiki/files/apache/sites/wikimedia.conf b/modules/mediawiki/files/apache/sites/wikimedia.conf index 2df1f70..624b2d6 100644 --- a/modules/mediawiki/files/apache/sites/wikimedia.conf +++ b/modules/mediawiki/files/apache/sites/wikimedia.conf @@ -25,25 +25,6 @@ Include "sites-enabled/wikimedia-legacy.incl" -#bits - -ServerName bits.wikimedia.org -DocumentRoot "/srv/mediawiki/docroot/bits/" - -# Version-independent aliases for backwards-compatibility -Alias /w/extensions/ /srv/mediawiki/php/extensions/ -Alias /skins-1.5/ /srv/mediawiki/php/skins/ -Alias /skins/ /srv/mediawiki/php/skins/ - -Alias /images/ /srv/mediawiki/images/ - -# Handle error pages correctly if mod_php is absent -ProxyPass /w/404.php fcgi://127.0.0.1:9000/srv/mediawiki/docroot/wikimedia.org/w/404.php retry=0 - -# Used for Firefox OS web application manifest living on bits.wikimedia.org -AddType application/x-web-app-manifest+json .webapp - - # loginwiki ServerName login.wikimedia.org -- To view, visit https://gerrit.wikimedia.org/r/305536 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I93af7b2874d2636c358a64b180dc0b78a2fd5837 Gerrit-PatchSet: 5 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: BBlack Gerrit-Reviewer: Alex Monk Gerrit-Reviewer: Alexandros Kosiaris Gerrit-Reviewer: BBlack Gerrit-Reviewer: Chad Gerrit-Reviewer: Elukey Gerrit-Reviewer: Giuseppe Lavagetto Gerrit-Reviewer: Krinkle Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: [WIP] Yandex ClickHouse puppetization
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/325797
Change subject: [WIP] Yandex ClickHouse puppetization
..
[WIP] Yandex ClickHouse puppetization
Change-Id: Ib408eb152824ee119becc34d6346617847195071
---
A modules/clickhouse/manifests/init.pp
A modules/clickhouse/templates/clickhouse.systemd.erb
A modules/clickhouse/templates/config.xml.erb
3 files changed, 310 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/97/325797/1
diff --git a/modules/clickhouse/manifests/init.pp
b/modules/clickhouse/manifests/init.pp
new file mode 100644
index 000..62ee1c0
--- /dev/null
+++ b/modules/clickhouse/manifests/init.pp
@@ -0,0 +1,107 @@
+# == Class clickhouse
+#
+# Yandex Clickhouse (https://clickhouse.yandex)
+# More info: https://clickhouse.yandex/reference_en.html#What is ClickHouse?
+#
+# == Parameters
+#
+# [*clusters*]
+# Clusters to be defined in ClickHouse. Each list of replicas corresponds
+# to a shard configured for its related cluster.
+#
+# Ref: https://clickhouse.yandex/reference_en.html#Distributed
+# Example:
+# {
+# "cluster1" => [['replica1.hostname', replica1.hostname'],
+#['replica3.hostname', replica4.hostname']],
+# "cluster2" => [['replica3.hostname', replica4.hostname']],
+# ...
+#}
+# In the above example two clusters are defined: the first one contains
+# two shards, meanwhile the second only one. All the shards defined contains
+# two replicas.
+#
+
+class clickhouse (
+clusters= {},
+zookeeper_hosts = ['localhost'],
+zookeeper_port = 2181,
+interserver_http_port = 9009,
+tcp_port= 9000,
+http_server_port= 8123,
+http_server_accept_from = "::",
+http_server_max_conns = 4096,
+http_server_ka_timeout = 3,
+max_concurrent_queries = 100,
+uncompressed_cache_size = 8589934592,
+mark_cache_size = 5368709120,
+builtin_dict_reload_int = 3600,
+data_dir= "/srv/clickhouse",
+tmp_processing_dir = "/tmp/",
+log_dir = "/var/log/clickhouse",
+log_level = "info",
+log_rotation_max_size = "100M",
+log_rotation_max_files = 10,
+user_config_file= "users.xml",
+graphite_host = undef,
+graphite_port = undef,
+graphite_metric_prefix = "clickhouse",
+graphite_timeout= 0.1,
+contact_group = "admins"
+)
+{
+
+requires_os('debian >= jessie')
+require_package('clickhouse')
+
+group { 'clickhouse':
+ensure => present,
+system => true,
+}
+
+user { 'clickhouse':
+gid => 'clickhouse',
+shell => '/bin/bash',
+system => true,
+require => Group['clickhouse'],
+}
+
+file { '/etc/clickhouse':
+ensure => directory,
+owner => 'root',
+group => 'root',
+mode => '0755',
+}
+
+file { '/etc/clickhouse/config.xml':
+ensure => present,
+owner => 'root',
+group => 'root',
+mode=> '0444',
+content => template('clickhouse/config.xml.erb'),
+require => File['/etc/clickhouse'],
+}
+
+systemd::syslog { 'clickhouse':
+readable_by => 'all',
+base_dir=> '/var/log',
+group => 'root',
+}
+
+base::service_unit { 'clickhouse':
+ensure => present,
+systemd => true,
+require => [
+File['/etc/clickhouse/config.xml'],
+User['clickhouse'],
+Systemd::Syslog['clickhouse'],
+],
+}
+
+monitoring::service { 'clickhouse':
+description => 'clickhouse',
+check_command => "check_tcp!${tcp_port}",
+contact_group => $contact_group,
+require => Base::Service_unit['clickhouse'],
+}
+}
\ No newline at end of file
diff --git a/modules/clickhouse/templates/clickhouse.systemd.erb
b/modules/clickhouse/templates/clickhouse.systemd.erb
new file mode 100644
index 000..54d5be3
--- /dev/null
+++ b/modules/clickhouse/templates/clickhouse.systemd.erb
@@ -0,0 +1,18 @@
+# NOTE: This file is managed by Puppet
+# Systemd unit for Yandex Clickhouse
+[Unit]
+Description="clickhouse service"
+After=network.target
+
+[Service]
+User=clickhouse
+Group=clickhouse
+Restart=always
+RestartSec=2s
+# wait 60 seconds for a graceful restart before killing the master
+TimeoutStopSec=60
+ExecStart=/usr/bin/clickhouse-server
--config-file=/etc/clickhouse-server/config.xml
+SyslogIdentifier=clickhouse
+
+[Install]
+WantedBy=multi-user.target
\ No newline at end of file
diff --git a/modules/clickhouse/templates/config.xml.erb
b/modules/clickhouse/templates/config.xml.erb
new file mode 100644
index 000..937a348
--- /dev/null
+++ b/modules/clickhouse/templa
[MediaWiki-commits] [Gerrit] operations...prometheus-apache-exporter[master]: Initial debianization
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/325568
Change subject: Initial debianization
..
Initial debianization
Bug: T147316
Change-Id: I14d46a8998ca25f1d376065bb196b8286d3ac4cd
---
A debian/changelog
A debian/compat
A debian/control
A debian/copyright
A debian/default
A debian/dirs
A debian/docs
A debian/gdb.conf
A debian/init
A debian/postinst
A debian/postrm
A debian/rules
A debian/service
A debian/watch
14 files changed, 208 insertions(+), 0 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/operations/debs/prometheus-apache-exporter
refs/changes/68/325568/1
diff --git a/debian/changelog b/debian/changelog
new file mode 100644
index 000..a80c7ce
--- /dev/null
+++ b/debian/changelog
@@ -0,0 +1,5 @@
+prometheus-apache-exporter (0.3-1) jessie-wikimedia; urgency=low
+
+ * Initial Release, packaged the 0.3 upstream version.
+
+ -- Luca Toscano Thu, 03 Nov 2016 15:13:55 +0100
diff --git a/debian/compat b/debian/compat
new file mode 100644
index 000..ec63514
--- /dev/null
+++ b/debian/compat
@@ -0,0 +1 @@
+9
diff --git a/debian/control b/debian/control
new file mode 100644
index 000..0b20b9c
--- /dev/null
+++ b/debian/control
@@ -0,0 +1,16 @@
+Source: prometheus-apache-exporter
+Section: net
+Priority: extra
+Maintainer: Luca Toscano
+Build-Depends: debhelper (>= 9),
+ dh-golang,
+ dh-systemd (>= 1.5),
+ golang-go,
+ golang-github-prometheus-client-golang-dev
+Standards-Version: 3.9.8
+Homepage: https://github.com/neezgee/apache_exporter
+
+Package: prometheus-apache-exporter
+Architecture: any
+Depends: ${shlibs:Depends}, ${misc:Depends}
+Description: Exports apache mod_status statistics via HTTP for Prometheus
consumption.
diff --git a/debian/copyright b/debian/copyright
new file mode 100644
index 000..9f4877a
--- /dev/null
+++ b/debian/copyright
@@ -0,0 +1,27 @@
+Files: *
+Copyright: 2016 neezgee
+License: MIT
+
+Files: debian/*
+Copyright: 2016 Luca Toscano
+License: MIT
+Comment: Debian packaging is licensed under the same terms as upstream
+
+License: MIT
+ Permission is hereby granted, free of charge, to any person obtaining a copy
+ of this software and associated documentation files (the "Software"), to deal
+ in the Software without restriction, including without limitation the rights
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ copies of the Software, and to permit persons to whom the Software is
+ furnished to do so, subject to the following conditions:
+ .
+ The above copyright notice and this permission notice shall be included in all
+ copies or substantial portions of the Software.
+ .
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ SOFTWARE.
diff --git a/debian/default b/debian/default
new file mode 100644
index 000..8bf8fdb
--- /dev/null
+++ b/debian/default
@@ -0,0 +1,14 @@
+ARGS="-raw"
+
+# Usage of prometheus-apache-exporter:
+#
+# -insecure
+#Ignore server certificate if using https (default false)
+# -log.level value
+#Only log messages with the given severity or above. Valid levels:
[debug, info, warn, error, fatal, panic]. (default info)
+# -scrape_uri string
+#URI to apache stub status page (default
"http://localhost/server-status/?auto";)
+# -telemetry.address string
+#Address on which to expose metrics. (default ":9117")
+# -telemetry.endpoint string
+#Path under which to expose metrics. (default "/metrics")
diff --git a/debian/dirs b/debian/dirs
new file mode 100644
index 000..557ffa1
--- /dev/null
+++ b/debian/dirs
@@ -0,0 +1 @@
+var/log/prometheus
diff --git a/debian/docs b/debian/docs
new file mode 100644
index 000..b43bf86
--- /dev/null
+++ b/debian/docs
@@ -0,0 +1 @@
+README.md
diff --git a/debian/gdb.conf b/debian/gdb.conf
new file mode 100644
index 000..f692a5d
--- /dev/null
+++ b/debian/gdb.conf
@@ -0,0 +1,2 @@
+[buildpackage]
+upstream-tag=v%(version)s
diff --git a/debian/init b/debian/init
new file mode 100755
index 000..0cce939
--- /dev/null
+++ b/debian/init
@@ -0,0 +1,64 @@
+#!/bin/sh
+# kFreeBSD do not accept scripts as interpreters, using #!/bin/sh and sourcing.
+if [ true != "$INIT_D_SCRIPT_SOURCED" ] ; then
+set "$0" "$@"; INIT_D_SCRIPT_SOURCED=true . /lib/init/init-d-script
+fi
+### BEGIN INIT INFO
+# Provides: prometheus-apache-exporter
+# Required-Start:$remote_fs $syslog
+# Required-Stop: $remote_fs $syslog
+# Default-Start: 2 3 4 5
+#
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add Nagios process alarms for statsv and EL varnishkafka ins...
Elukey has submitted this change and it was merged.
Change subject: Add Nagios process alarms for statsv and EL varnishkafka
instance
..
Add Nagios process alarms for statsv and EL varnishkafka instance
On cache::text there are two more varnishkafka instances running,
collecting data for statsv and eventlogging.
Change-Id: I1da5280b2040f63a560775c1d9bde34e461897ab
---
M modules/role/manifests/cache/kafka/eventlogging.pp
M modules/role/manifests/cache/kafka/statsv.pp
M modules/role/manifests/cache/kafka/webrequest.pp
3 files changed, 21 insertions(+), 4 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Ottomata: Looks good to me, but someone else must approve
Ema: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/role/manifests/cache/kafka/eventlogging.pp
b/modules/role/manifests/cache/kafka/eventlogging.pp
index 57802a5..c0e2b8b 100644
--- a/modules/role/manifests/cache/kafka/eventlogging.pp
+++ b/modules/role/manifests/cache/kafka/eventlogging.pp
@@ -27,6 +27,14 @@
include ::standard
+# Generate icinga alert if varnishkafka is not running.
+nrpe::monitor_service { 'varnishkafka-eventlogging':
+description => 'Varnishkafka log producer',
+nrpe_command => "/usr/lib/nagios/plugins/check_procs -c 1 -a
'/usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf'",
+contact_group => 'admins,analytics',
+require => Class['::varnishkafka'],
+}
+
$cache_type = hiera('cache::cluster')
$graphite_metric_prefix =
"varnishkafka.${::hostname}.eventlogging.${cache_type}"
diff --git a/modules/role/manifests/cache/kafka/statsv.pp
b/modules/role/manifests/cache/kafka/statsv.pp
index 15115b4..29f3e44 100644
--- a/modules/role/manifests/cache/kafka/statsv.pp
+++ b/modules/role/manifests/cache/kafka/statsv.pp
@@ -37,6 +37,14 @@
include ::standard
+# Generate icinga alert if varnishkafka is not running.
+nrpe::monitor_service { 'varnishkafka-statsv':
+description => 'Varnishkafka log producer',
+nrpe_command => "/usr/lib/nagios/plugins/check_procs -c 1 -a
'/usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf'",
+contact_group => 'admins,analytics',
+require => Class['::varnishkafka'],
+}
+
$cache_type = hiera('cache::cluster')
$graphite_metric_prefix = "varnishkafka.${::hostname}.statsv.${cache_type}"
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 447cfa9..4ea2ced 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -111,10 +111,11 @@
include ::standard
# Generate icinga alert if varnishkafka is not running.
-nrpe::monitor_service { 'varnishkafka':
-description => 'Varnishkafka log producer',
-nrpe_command => '/usr/lib/nagios/plugins/check_procs -c 1: -C
varnishkafka',
-require => Class['::varnishkafka'],
+nrpe::monitor_service { 'varnishkafka-webrequest':
+description => 'Varnishkafka log producer',
+nrpe_command => "/usr/lib/nagios/plugins/check_procs -c 1 -a
'/usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf'",
+contact_group => 'admins,analytics',
+require => Class['::varnishkafka'],
}
$cache_type = hiera('cache::cluster')
--
To view, visit https://gerrit.wikimedia.org/r/325317
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I1da5280b2040f63a560775c1d9bde34e461897ab
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: BBlack
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ema
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add Nagios process alarms for statsv and EL varnishkafka ins...
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/325317
Change subject: Add Nagios process alarms for statsv and EL varnishkafka
instance
..
Add Nagios process alarms for statsv and EL varnishkafka instance
On cache::text there are two more varnishkafka instances running,
collecting data for statsv and eventlogging.
Change-Id: I1da5280b2040f63a560775c1d9bde34e461897ab
---
M modules/role/manifests/cache/kafka/eventlogging.pp
M modules/role/manifests/cache/kafka/statsv.pp
M modules/role/manifests/cache/kafka/webrequest.pp
3 files changed, 21 insertions(+), 4 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/17/325317/1
diff --git a/modules/role/manifests/cache/kafka/eventlogging.pp
b/modules/role/manifests/cache/kafka/eventlogging.pp
index 57802a5..6d7a886 100644
--- a/modules/role/manifests/cache/kafka/eventlogging.pp
+++ b/modules/role/manifests/cache/kafka/eventlogging.pp
@@ -27,6 +27,14 @@
include ::standard
+# Generate icinga alert if varnishkafka is not running.
+nrpe::monitor_service { 'varnishkafka-eventlogging':
+description => 'Varnishkafka log producer',
+nrpe_command => "/usr/lib/nagios/plugins/check_procs -c 1: -a
/usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf",
+contact_group => 'admins,analytics',
+require => Class['::varnishkafka'],
+}
+
$cache_type = hiera('cache::cluster')
$graphite_metric_prefix =
"varnishkafka.${::hostname}.eventlogging.${cache_type}"
diff --git a/modules/role/manifests/cache/kafka/statsv.pp
b/modules/role/manifests/cache/kafka/statsv.pp
index 15115b4..5186996 100644
--- a/modules/role/manifests/cache/kafka/statsv.pp
+++ b/modules/role/manifests/cache/kafka/statsv.pp
@@ -37,6 +37,14 @@
include ::standard
+# Generate icinga alert if varnishkafka is not running.
+nrpe::monitor_service { 'varnishkafka-statsv':
+description => 'Varnishkafka log producer',
+nrpe_command => "/usr/lib/nagios/plugins/check_procs -c 1: -a
/usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf",
+contact_group => 'admins,analytics',
+require => Class['::varnishkafka'],
+}
+
$cache_type = hiera('cache::cluster')
$graphite_metric_prefix = "varnishkafka.${::hostname}.statsv.${cache_type}"
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 447cfa9..95c0c5f 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -111,10 +111,11 @@
include ::standard
# Generate icinga alert if varnishkafka is not running.
-nrpe::monitor_service { 'varnishkafka':
-description => 'Varnishkafka log producer',
-nrpe_command => '/usr/lib/nagios/plugins/check_procs -c 1: -C
varnishkafka',
-require => Class['::varnishkafka'],
+nrpe::monitor_service { 'varnishkafka-webrequest':
+description => 'Varnishkafka log producer',
+nrpe_command => "/usr/lib/nagios/plugins/check_procs -c 1: -a
/usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf",
+contact_group => 'admins,analytics',
+require => Class['::varnishkafka'],
}
$cache_type = hiera('cache::cluster')
--
To view, visit https://gerrit.wikimedia.org/r/325317
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1da5280b2040f63a560775c1d9bde34e461897ab
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add openjdk-8-jdk to the list of statistics packages
Elukey has submitted this change and it was merged. Change subject: Add openjdk-8-jdk to the list of statistics packages .. Add openjdk-8-jdk to the list of statistics packages Bug: T151896 Change-Id: I9906286ce934de61e620f64229355d23e3f320af --- M modules/statistics/manifests/packages.pp 1 file changed, 1 insertion(+), 0 deletions(-) Approvals: Elukey: Looks good to me, approved Filippo Giunchedi: Looks good to me, but someone else must approve Ottomata: Looks good to me, but someone else must approve jenkins-bot: Verified diff --git a/modules/statistics/manifests/packages.pp b/modules/statistics/manifests/packages.pp index 7e1ccc8..df5ea96 100644 --- a/modules/statistics/manifests/packages.pp +++ b/modules/statistics/manifests/packages.pp @@ -35,6 +35,7 @@ 'build-essential', # Requested by halfak to install SciPy 'nodejs', 'openjdk-7-jdk', +'openjdk-8-jdk', 'g++-4.8', # Requested by bearloga (Mikhail) to ensure that there is a compiler with C++11 support that can compile R package 'Boom'; see T147682 and http://stackoverflow.com/a/36034866/1091835 for more info # GNU Scientific Library (GSL) requested by bearloga (Mikhail) 'libgsl0ldbl', # GSL part 1/3 -- To view, visit https://gerrit.wikimedia.org/r/324679 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I9906286ce934de61e620f64229355d23e3f320af Gerrit-PatchSet: 2 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Filippo Giunchedi Gerrit-Reviewer: Ottomata Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Switch Varnishkafka monitoring from Ganglia to statsd
Elukey has submitted this change and it was merged.
Change subject: Switch Varnishkafka monitoring from Ganglia to statsd
..
Switch Varnishkafka monitoring from Ganglia to statsd
This change includes:
1) the varnishkafka submodule SHA upgrade
ref: https://gerrit.wikimedia.org/r/#/c/324877
https://gerrit.wikimedia.org/r/#/c/324887
https://gerrit.wikimedia.org/r/#/c/324890
https://gerrit.wikimedia.org/r/#/c/324891
2) removal of the last Ganglia configuration for statsv;
3) removal of not used graphite monitors for webrequest.pp;
4) statsd configuration for statsv and eventlogging instances.
Bug: T152093
Change-Id: I3c14fa90baba8063bf946dd7feb3733d090ee1bb
---
M modules/role/manifests/cache/kafka/eventlogging.pp
M modules/role/manifests/cache/kafka/statsv.pp
M modules/role/manifests/cache/kafka/webrequest.pp
M modules/varnishkafka
4 files changed, 36 insertions(+), 42 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Ottomata: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/role/manifests/cache/kafka/eventlogging.pp
b/modules/role/manifests/cache/kafka/eventlogging.pp
index fc5515d..57802a5 100644
--- a/modules/role/manifests/cache/kafka/eventlogging.pp
+++ b/modules/role/manifests/cache/kafka/eventlogging.pp
@@ -24,4 +24,16 @@
topic_request_required_acks => '1',
conf_template => $conf_template,
}
+
+include ::standard
+
+$cache_type = hiera('cache::cluster')
+$graphite_metric_prefix =
"varnishkafka.${::hostname}.eventlogging.${cache_type}"
+
+# Sets up Logster to read from the Varnishkafka instance stats JSON file
+# and report metrics to statsd.
+varnishkafka::monitor::statsd { 'eventlogging':
+graphite_metric_prefix => $graphite_metric_prefix,
+statsd_host_port => hiera('statsd'),
+}
}
diff --git a/modules/role/manifests/cache/kafka/statsv.pp
b/modules/role/manifests/cache/kafka/statsv.pp
index a31b1a9..15115b4 100644
--- a/modules/role/manifests/cache/kafka/statsv.pp
+++ b/modules/role/manifests/cache/kafka/statsv.pp
@@ -36,7 +36,14 @@
}
include ::standard
-if $::standard::has_ganglia {
-varnishkafka::monitor { 'statsv': }
+
+$cache_type = hiera('cache::cluster')
+$graphite_metric_prefix = "varnishkafka.${::hostname}.statsv.${cache_type}"
+
+# Sets up Logster to read from the Varnishkafka instance stats JSON file
+# and report metrics to statsd.
+varnishkafka::monitor::statsd { 'statsv':
+graphite_metric_prefix => $graphite_metric_prefix,
+statsd_host_port => hiera('statsd'),
}
}
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 01926d9..447cfa9 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -117,55 +117,30 @@
require => Class['::varnishkafka'],
}
-# Extract cache type name from topic for use in statsd prefix.
-# There is probably a better way to do this.
-$cache_type = regsubst($topic, '^webrequest_(.+)$', '\1')
+$cache_type = hiera('cache::cluster')
$graphite_metric_prefix =
"varnishkafka.${::hostname}.webrequest.${cache_type}"
-# Test using logster to send varnishkafka stats to statsd -> graphite.
-# This may be moved into the varnishkafka module.
-logster::job { 'varnishkafka-webrequest':
-minute => '*/1',
-parser => 'JsonLogster',
-logfile => '/var/cache/varnishkafka/webrequest.stats.json',
-logster_options => "-o statsd --statsd-host=statsd.eqiad.wmnet:8125
--metric-prefix=${graphite_metric_prefix}",
+# Sets up Logster to read from the Varnishkafka instance stats JSON file
+# and report metrics to statsd.
+varnishkafka::monitor::statsd { 'webrequest':
+graphite_metric_prefix => $graphite_metric_prefix,
+statsd_host_port => hiera('statsd'),
}
-
-
-# TEMPORARY test --until on all vk drerr alerts
-$until = '0min'
# Generate an alert if too many delivery report errors per minute
# (logster only reports once a minute)
monitoring::graphite_threshold { 'varnishkafka-kafka_drerr':
-ensure => 'present',
-description => 'Varnishkafka Delivery Errors per minute',
-metric =>
"derivative(transformNull(${graphite_metric_prefix}.varnishkafka.kafka_drerr,
0))",
+ensure => 'present',
+description => 'Varnishkafka Delivery Errors per minute',
+metric =>
"derivative(transformNull(${graphite_metric_prefix}.varnishkafka.kafka_drerr,
0))",
# More than 0 errors is warning threshold.
-warning => 0,
+warning => 0,
# More than 2 errors is critical thresh
[MediaWiki-commits] [Gerrit] operations/puppet[production]: RESTBase: Fix the Citoid URI for BetaCluster
Elukey has submitted this change and it was merged. Change subject: RESTBase: Fix the Citoid URI for BetaCluster .. RESTBase: Fix the Citoid URI for BetaCluster Bug: T108646 Change-Id: I7787289e0591db7e8e9983a19fd190529c891c01 --- M hieradata/labs/deployment-prep/common.yaml 1 file changed, 1 insertion(+), 0 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/hieradata/labs/deployment-prep/common.yaml b/hieradata/labs/deployment-prep/common.yaml index 7c3506c..1bafd71 100644 --- a/hieradata/labs/deployment-prep/common.yaml +++ b/hieradata/labs/deployment-prep/common.yaml @@ -115,6 +115,7 @@ restbase::aqs_uri: https://wikimedia.org/api/rest_v1/metrics restbase::eventlogging_service_uri: http://deployment-eventlogging04.deployment-prep.eqiad.wmflabs:8085/v1/events restbase::pdfrender_uri: http://deployment-pdfrender02.deployment-prep.eqiad.wmflabs:5252 +restbase::citoid_uri: http://deployment-sca02.deployment-prep.eqiad.wmflabs:1970 "mediawiki::log_aggregator": deployment-fluorine02.eqiad.wmflabs:8420 "mediawiki::forward_syslog": deployment-logstash2.deployment-prep.eqiad.wmflabs:10514 mediawiki_memcached_servers: -- To view, visit https://gerrit.wikimedia.org/r/324926 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I7787289e0591db7e8e9983a19fd190529c891c01 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Mobrovac Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Fix statsd logster job name
Elukey has submitted this change and it was merged.
Change subject: Fix statsd logster job name
..
Fix statsd logster job name
Bug: T152093
Change-Id: Ib4106781b4266f2b5dc3dc351bb918f1144d1f9a
---
M manifests/monitor/statsd.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
index f7f2d07..e5b740d 100644
--- a/manifests/monitor/statsd.pp
+++ b/manifests/monitor/statsd.pp
@@ -14,7 +14,7 @@
# Send varnishkafka stats to statsd -> graphite using Logster.
# Logster runs every minute using a cronjob.
-logster::job { 'varnishkafka-${name}':
+logster::job { "varnishkafka-${name}":
minute => '*/1',
parser => 'JsonLogster',
logfile => $log_statistics_file,
--
To view, visit https://gerrit.wikimedia.org/r/324891
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib4106781b4266f2b5dc3dc351bb918f1144d1f9a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Fix statsd logster job name
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/324891
Change subject: Fix statsd logster job name
..
Fix statsd logster job name
Bug: T152093
Change-Id: Ib4106781b4266f2b5dc3dc351bb918f1144d1f9a
---
M manifests/monitor/statsd.pp
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/varnishkafka
refs/changes/91/324891/1
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
index f7f2d07..e5b740d 100644
--- a/manifests/monitor/statsd.pp
+++ b/manifests/monitor/statsd.pp
@@ -14,7 +14,7 @@
# Send varnishkafka stats to statsd -> graphite using Logster.
# Logster runs every minute using a cronjob.
-logster::job { 'varnishkafka-${name}':
+logster::job { "varnishkafka-${name}":
minute => '*/1',
parser => 'JsonLogster',
logfile => $log_statistics_file,
--
To view, visit https://gerrit.wikimedia.org/r/324891
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib4106781b4266f2b5dc3dc351bb918f1144d1f9a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Fix class dependency for ganglia/statsd monitoring
Elukey has submitted this change and it was merged.
Change subject: Fix class dependency for ganglia/statsd monitoring
..
Fix class dependency for ganglia/statsd monitoring
Bug: T152093
Change-Id: Ibde002bd88f12e3a7a034d292aeea7eafb235991
---
M manifests/monitor/ganglia.pp
M manifests/monitor/statsd.pp
2 files changed, 2 insertions(+), 2 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/monitor/ganglia.pp b/manifests/monitor/ganglia.pp
index 526365a..dcb213f 100644
--- a/manifests/monitor/ganglia.pp
+++ b/manifests/monitor/ganglia.pp
@@ -10,7 +10,7 @@
) {
require ::varnishkafka
-Varnishkafka::Instance[$name] -> Varnishkafka::Monitor[$name]
+Varnishkafka::Instance[$name] -> Varnishkafka::Monitor::Ganglia[$name]
$varnishkafka_py = '/usr/lib/ganglia/python_modules/varnishkafka.py'
$generate_pyconf_command = "/usr/bin/python ${varnishkafka_py}
--generate-pyconf ${pyconf_file} --key-prefix=${key_prefix}
--tmax=${log_statistics_interval} ${log_statistics_file}"
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
index 380c895..f7f2d07 100644
--- a/manifests/monitor/statsd.pp
+++ b/manifests/monitor/statsd.pp
@@ -10,7 +10,7 @@
) {
require ::varnishkafka
-Varnishkafka::Instance[$name] -> Varnishkafka::Monitor[$name]
+Varnishkafka::Instance[$name] -> Varnishkafka::Monitor::Statsd[$name]
# Send varnishkafka stats to statsd -> graphite using Logster.
# Logster runs every minute using a cronjob.
--
To view, visit https://gerrit.wikimedia.org/r/324890
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibde002bd88f12e3a7a034d292aeea7eafb235991
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Fix class dependency for ganglia/statsd monitoring
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/324890
Change subject: Fix class dependency for ganglia/statsd monitoring
..
Fix class dependency for ganglia/statsd monitoring
Bug: T152093
Change-Id: Ibde002bd88f12e3a7a034d292aeea7eafb235991
---
M manifests/monitor/ganglia.pp
M manifests/monitor/statsd.pp
2 files changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/varnishkafka
refs/changes/90/324890/1
diff --git a/manifests/monitor/ganglia.pp b/manifests/monitor/ganglia.pp
index 526365a..dcb213f 100644
--- a/manifests/monitor/ganglia.pp
+++ b/manifests/monitor/ganglia.pp
@@ -10,7 +10,7 @@
) {
require ::varnishkafka
-Varnishkafka::Instance[$name] -> Varnishkafka::Monitor[$name]
+Varnishkafka::Instance[$name] -> Varnishkafka::Monitor::Ganglia[$name]
$varnishkafka_py = '/usr/lib/ganglia/python_modules/varnishkafka.py'
$generate_pyconf_command = "/usr/bin/python ${varnishkafka_py}
--generate-pyconf ${pyconf_file} --key-prefix=${key_prefix}
--tmax=${log_statistics_interval} ${log_statistics_file}"
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
index 380c895..f7f2d07 100644
--- a/manifests/monitor/statsd.pp
+++ b/manifests/monitor/statsd.pp
@@ -10,7 +10,7 @@
) {
require ::varnishkafka
-Varnishkafka::Instance[$name] -> Varnishkafka::Monitor[$name]
+Varnishkafka::Instance[$name] -> Varnishkafka::Monitor::Statsd[$name]
# Send varnishkafka stats to statsd -> graphite using Logster.
# Logster runs every minute using a cronjob.
--
To view, visit https://gerrit.wikimedia.org/r/324890
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibde002bd88f12e3a7a034d292aeea7eafb235991
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Fix ganglia/statsd class namespace
Elukey has submitted this change and it was merged.
Change subject: Fix ganglia/statsd class namespace
..
Fix ganglia/statsd class namespace
Bug: T152093
Change-Id: I67252b2ec3eaeb94b54599ee5fa3bd64da477235
---
M manifests/monitor/ganglia.pp
M manifests/monitor/statsd.pp
2 files changed, 2 insertions(+), 2 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/monitor/ganglia.pp b/manifests/monitor/ganglia.pp
index 78d40f1..526365a 100644
--- a/manifests/monitor/ganglia.pp
+++ b/manifests/monitor/ganglia.pp
@@ -2,7 +2,7 @@
#
# Installs varnishkafka python ganglia module.
#
-define varnishkafka::monitor(
+define varnishkafka::monitor::ganglia(
$log_statistics_file = "/var/cache/varnishkafka/${name}.stats.json",
$pyconf_file =
"/etc/ganglia/conf.d/varnishkafka-${name}.pyconf",
$log_statistics_interval = 60,
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
index 0b59c7e..380c895 100644
--- a/manifests/monitor/statsd.pp
+++ b/manifests/monitor/statsd.pp
@@ -3,7 +3,7 @@
# Configures Logster to parse Varnishkafka stats JSON files, extract metrics
# and push them to statsd.
#
-define varnishkafka::monitor(
+define varnishkafka::monitor::statsd(
$log_statistics_file = "/var/cache/varnishkafka/${name}.stats.json",
$graphite_metric_prefix = "varnishkafka.stats",
$statsd_host_port= "localhost:8125",
--
To view, visit https://gerrit.wikimedia.org/r/324887
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I67252b2ec3eaeb94b54599ee5fa3bd64da477235
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Fix ganglia/statsd class namespace
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/324887
Change subject: Fix ganglia/statsd class namespace
..
Fix ganglia/statsd class namespace
Bug: T152093
Change-Id: I67252b2ec3eaeb94b54599ee5fa3bd64da477235
---
M manifests/monitor/ganglia.pp
M manifests/monitor/statsd.pp
2 files changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/varnishkafka
refs/changes/87/324887/1
diff --git a/manifests/monitor/ganglia.pp b/manifests/monitor/ganglia.pp
index 78d40f1..526365a 100644
--- a/manifests/monitor/ganglia.pp
+++ b/manifests/monitor/ganglia.pp
@@ -2,7 +2,7 @@
#
# Installs varnishkafka python ganglia module.
#
-define varnishkafka::monitor(
+define varnishkafka::monitor::ganglia(
$log_statistics_file = "/var/cache/varnishkafka/${name}.stats.json",
$pyconf_file =
"/etc/ganglia/conf.d/varnishkafka-${name}.pyconf",
$log_statistics_interval = 60,
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
index 0b59c7e..380c895 100644
--- a/manifests/monitor/statsd.pp
+++ b/manifests/monitor/statsd.pp
@@ -3,7 +3,7 @@
# Configures Logster to parse Varnishkafka stats JSON files, extract metrics
# and push them to statsd.
#
-define varnishkafka::monitor(
+define varnishkafka::monitor::statsd(
$log_statistics_file = "/var/cache/varnishkafka/${name}.stats.json",
$graphite_metric_prefix = "varnishkafka.stats",
$statsd_host_port= "localhost:8125",
--
To view, visit https://gerrit.wikimedia.org/r/324887
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I67252b2ec3eaeb94b54599ee5fa3bd64da477235
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Switch Varnishkafka monitoring from Ganglia to statsd
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/324883
Change subject: Switch Varnishkafka monitoring from Ganglia to statsd
..
Switch Varnishkafka monitoring from Ganglia to statsd
This change includes:
1) the varnishkafka submodule SHA upgrade
(ref: https://gerrit.wikimedia.org/r/#/c/324877);
2) removal of the last Ganglia configuration for statsv;
3) removal of not used graphite monitors for webrequest.pp;
4) statsd configuration for statsv and eventlogging instances.
Bug: T152093
Change-Id: I3c14fa90baba8063bf946dd7feb3733d090ee1bb
---
M modules/role/manifests/cache/kafka/eventlogging.pp
M modules/role/manifests/cache/kafka/statsv.pp
M modules/role/manifests/cache/kafka/webrequest.pp
M modules/varnishkafka
4 files changed, 29 insertions(+), 26 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/83/324883/1
diff --git a/modules/role/manifests/cache/kafka/eventlogging.pp
b/modules/role/manifests/cache/kafka/eventlogging.pp
index fc5515d..7bf8b2c 100644
--- a/modules/role/manifests/cache/kafka/eventlogging.pp
+++ b/modules/role/manifests/cache/kafka/eventlogging.pp
@@ -24,4 +24,16 @@
topic_request_required_acks => '1',
conf_template => $conf_template,
}
+
+include ::standard
+
+$cache_type = hiera('cache::cluster')
+$graphite_metric_prefix =
"varnishkafka.${::hostname}.eventlogging.${cache_type}"
+
+# Sets up Logster to read from the Varnishkafka instance stats JSON file
+# and report metrics to statsd.
+varnishkafka::monitor { 'eventlogging':
+$graphite_metric_prefix = $graphite_metric_prefix,
+$statsd_host_port= hiera('statsd'),
+}
}
diff --git a/modules/role/manifests/cache/kafka/statsv.pp
b/modules/role/manifests/cache/kafka/statsv.pp
index a31b1a9..cea33d4 100644
--- a/modules/role/manifests/cache/kafka/statsv.pp
+++ b/modules/role/manifests/cache/kafka/statsv.pp
@@ -36,7 +36,14 @@
}
include ::standard
-if $::standard::has_ganglia {
-varnishkafka::monitor { 'statsv': }
+
+$cache_type = hiera('cache::cluster')
+$graphite_metric_prefix = "varnishkafka.${::hostname}.statsv.${cache_type}"
+
+# Sets up Logster to read from the Varnishkafka instance stats JSON file
+# and report metrics to statsd.
+varnishkafka::monitor { 'statsv':
+$graphite_metric_prefix = $graphite_metric_prefix,
+$statsd_host_port= hiera('statsd'),
}
}
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 01926d9..57ebd55 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -117,9 +117,7 @@
require => Class['::varnishkafka'],
}
-# Extract cache type name from topic for use in statsd prefix.
-# There is probably a better way to do this.
-$cache_type = regsubst($topic, '^webrequest_(.+)$', '\1')
+$cache_type = hiera('cache::cluster')
$graphite_metric_prefix =
"varnishkafka.${::hostname}.webrequest.${cache_type}"
# Test using logster to send varnishkafka stats to statsd -> graphite.
@@ -131,9 +129,12 @@
logster_options => "-o statsd --statsd-host=statsd.eqiad.wmnet:8125
--metric-prefix=${graphite_metric_prefix}",
}
-
-# TEMPORARY test --until on all vk drerr alerts
-$until = '0min'
+# Sets up Logster to read from the Varnishkafka instance stats JSON file
+# and report metrics to statsd.
+varnishkafka::monitor { 'webrequest':
+$graphite_metric_prefix = $graphite_metric_prefix,
+$statsd_host_port= hiera('statsd'),
+}
# Generate an alert if too many delivery report errors per minute
# (logster only reports once a minute)
@@ -149,23 +150,6 @@
# are over the threshold.
percentage => 80,
from=> '10min',
-until => $until,
-nagios_critical => false,
-require => Logster::Job['varnishkafka-webrequest'],
-}
-
-# Use graphite_anomaly to alert about anomolous deliver errors.
-monitoring::graphite_anomaly { 'varnishkafka-anomaly-kafka_drerr':
-# Disabling this. It doesn't work like I wanted it to.
-ensure => 'absent',
-description => 'Varnishkafka Delivery Errors per minute anomaly',
-metric =>
"nonNegativeDerivative(transformNull(${graphite_metric_prefix}.varnishkafka.kafka_drerr,
0))",
-over=> true,
-# warn if more than 10 anomylous datapoints (last 10 minutes)
-warning => 5,
-# critical if more than 45 anomylous datapoints (last 45 minutes)
-critical=> 45,
-nagios_critical => false,
require => Logster::Job['varnishkafka-webrequest'],
}
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Add a separate parameter for the statsd port
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/324880
Change subject: Add a separate parameter for the statsd port
..
Add a separate parameter for the statsd port
Bug: T152093
Change-Id: If2b5a42d2c0ed40777fabf9fad53c6e332881298
---
M manifests/monitor/statsd.pp
1 file changed, 5 insertions(+), 4 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/varnishkafka
refs/changes/80/324880/1
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
index 0b59c7e..f6b6c9c 100644
--- a/manifests/monitor/statsd.pp
+++ b/manifests/monitor/statsd.pp
@@ -4,9 +4,10 @@
# and push them to statsd.
#
define varnishkafka::monitor(
-$log_statistics_file = "/var/cache/varnishkafka/${name}.stats.json",
-$graphite_metric_prefix = "varnishkafka.stats",
-$statsd_host_port= "localhost:8125",
+$log_statistics_file= "/var/cache/varnishkafka/${name}.stats.json",
+$graphite_metric_prefix = "varnishkafka.stats",
+$statsd_host= "localhost",
+$statsd_port= 8125
) {
require ::varnishkafka
@@ -18,6 +19,6 @@
minute => '*/1',
parser => 'JsonLogster',
logfile => $log_statistics_file,
-logster_options => "-o statsd --statsd-host=${statsd_host_port}
--metric-prefix=${graphite_metric_prefix}",
+logster_options => "-o statsd
--statsd-host=${statsd_host}:${statsd_port}
--metric-prefix=${graphite_metric_prefix}",
}
}
--
To view, visit https://gerrit.wikimedia.org/r/324880
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If2b5a42d2c0ed40777fabf9fad53c6e332881298
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Refactor the monitor namespace to include Statsd
Elukey has submitted this change and it was merged.
Change subject: Refactor the monitor namespace to include Statsd
..
Refactor the monitor namespace to include Statsd
Bug: T152093
Change-Id: I7488293a17c72312caa22e64c2e29bbf449a7a44
---
M README.md
R manifests/monitor/ganglia.pp
A manifests/monitor/statsd.pp
3 files changed, 31 insertions(+), 6 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Ema: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/README.md b/README.md
index 0d85db3..cd6c61a 100644
--- a/README.md
+++ b/README.md
@@ -24,13 +24,14 @@
See the ```varnishkafka``` class docs in manifests/init.pp for more parameter
documentation.
-## Ganglia Monitoring
+## Monitoring
```puppet
-# This will install logster and a custom VarnishkafkaLogster
-# logster parser to send JSON stats from the log.statistics.file
-# to Ganglia.
-class { 'varnishkafka::monitoring': }
+# The following classes will install logster and a custom VarnishkafkaLogster
+# parser to send JSON stats from the log.statistics.file
+# to Ganglia or Statsd.
+class { 'varnishkafka::monitoring::ganglia': }
+class { 'varnishkafka::monitoring::statsd': }
```
## Testing
diff --git a/manifests/monitor.pp b/manifests/monitor/ganglia.pp
similarity index 96%
rename from manifests/monitor.pp
rename to manifests/monitor/ganglia.pp
index 17214da..78d40f1 100644
--- a/manifests/monitor.pp
+++ b/manifests/monitor/ganglia.pp
@@ -1,4 +1,5 @@
-# == Define varnishkafka::monitor
+# == Define varnishkafka::monitor::ganglia
+#
# Installs varnishkafka python ganglia module.
#
define varnishkafka::monitor(
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
new file mode 100644
index 000..0b59c7e
--- /dev/null
+++ b/manifests/monitor/statsd.pp
@@ -0,0 +1,23 @@
+# === Define varnishkafka::monitor::statsd
+#
+# Configures Logster to parse Varnishkafka stats JSON files, extract metrics
+# and push them to statsd.
+#
+define varnishkafka::monitor(
+$log_statistics_file = "/var/cache/varnishkafka/${name}.stats.json",
+$graphite_metric_prefix = "varnishkafka.stats",
+$statsd_host_port= "localhost:8125",
+) {
+require ::varnishkafka
+
+Varnishkafka::Instance[$name] -> Varnishkafka::Monitor[$name]
+
+# Send varnishkafka stats to statsd -> graphite using Logster.
+# Logster runs every minute using a cronjob.
+logster::job { 'varnishkafka-${name}':
+minute => '*/1',
+parser => 'JsonLogster',
+logfile => $log_statistics_file,
+logster_options => "-o statsd --statsd-host=${statsd_host_port}
--metric-prefix=${graphite_metric_prefix}",
+}
+}
--
To view, visit https://gerrit.wikimedia.org/r/324877
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7488293a17c72312caa22e64c2e29bbf449a7a44
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ema
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Refactor the monitor namespace to include Statsd
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/324877
Change subject: Refactor the monitor namespace to include Statsd
..
Refactor the monitor namespace to include Statsd
Bug: T152093
Change-Id: I7488293a17c72312caa22e64c2e29bbf449a7a44
---
M README.md
R manifests/monitor/ganglia.pp
A manifests/monitor/statsd.pp
3 files changed, 31 insertions(+), 6 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/varnishkafka
refs/changes/77/324877/1
diff --git a/README.md b/README.md
index 0d85db3..cd6c61a 100644
--- a/README.md
+++ b/README.md
@@ -24,13 +24,14 @@
See the ```varnishkafka``` class docs in manifests/init.pp for more parameter
documentation.
-## Ganglia Monitoring
+## Monitoring
```puppet
-# This will install logster and a custom VarnishkafkaLogster
-# logster parser to send JSON stats from the log.statistics.file
-# to Ganglia.
-class { 'varnishkafka::monitoring': }
+# The following classes will install logster and a custom VarnishkafkaLogster
+# parser to send JSON stats from the log.statistics.file
+# to Ganglia or Statsd.
+class { 'varnishkafka::monitoring::ganglia': }
+class { 'varnishkafka::monitoring::statsd': }
```
## Testing
diff --git a/manifests/monitor.pp b/manifests/monitor/ganglia.pp
similarity index 95%
rename from manifests/monitor.pp
rename to manifests/monitor/ganglia.pp
index 17214da..c2a56c9 100644
--- a/manifests/monitor.pp
+++ b/manifests/monitor/ganglia.pp
@@ -1,4 +1,5 @@
-# == Define varnishkafka::monitor
+# == Define varnishkafka::monitor::statsd::ganglia
+#
# Installs varnishkafka python ganglia module.
#
define varnishkafka::monitor(
diff --git a/manifests/monitor/statsd.pp b/manifests/monitor/statsd.pp
new file mode 100644
index 000..0b59c7e
--- /dev/null
+++ b/manifests/monitor/statsd.pp
@@ -0,0 +1,23 @@
+# === Define varnishkafka::monitor::statsd
+#
+# Configures Logster to parse Varnishkafka stats JSON files, extract metrics
+# and push them to statsd.
+#
+define varnishkafka::monitor(
+$log_statistics_file = "/var/cache/varnishkafka/${name}.stats.json",
+$graphite_metric_prefix = "varnishkafka.stats",
+$statsd_host_port= "localhost:8125",
+) {
+require ::varnishkafka
+
+Varnishkafka::Instance[$name] -> Varnishkafka::Monitor[$name]
+
+# Send varnishkafka stats to statsd -> graphite using Logster.
+# Logster runs every minute using a cronjob.
+logster::job { 'varnishkafka-${name}':
+minute => '*/1',
+parser => 'JsonLogster',
+logfile => $log_statistics_file,
+logster_options => "-o statsd --statsd-host=${statsd_host_port}
--metric-prefix=${graphite_metric_prefix}",
+}
+}
--
To view, visit https://gerrit.wikimedia.org/r/324877
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7488293a17c72312caa22e64c2e29bbf449a7a44
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove Ganglia monitoring for Varnishkafka
Elukey has submitted this change and it was merged.
Change subject: Remove Ganglia monitoring for Varnishkafka
..
Remove Ganglia monitoring for Varnishkafka
Bug: T152093
Change-Id: I95deeb9a0d14b9b415268233ae22bd68c156d6f9
---
M modules/role/manifests/cache/kafka/webrequest.pp
1 file changed, 0 insertions(+), 7 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Ottomata: Looks good to me, but someone else must approve
Ema: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 61db695..01926d9 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -109,13 +109,6 @@
}
include ::standard
-if $::standard::has_ganglia {
-varnishkafka::monitor { 'webrequest':
-# The primary webrequest varnishkafka instance was formerly the
-# only one running, so we don't prefix its Ganglia metric keys.
-key_prefix => '',
-}
-}
# Generate icinga alert if varnishkafka is not running.
nrpe::monitor_service { 'varnishkafka':
--
To view, visit https://gerrit.wikimedia.org/r/324708
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I95deeb9a0d14b9b415268233ae22bd68c156d6f9
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: BBlack
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ema
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update varnishkafka submodule SHA
Elukey has submitted this change and it was merged. Change subject: Update varnishkafka submodule SHA .. Update varnishkafka submodule SHA Related gerrit patch: https://gerrit.wikimedia.org/r/#/c/324706/ Change-Id: I0c5004a09487de0881db54ecde4f8f7469ea8158 --- M modules/varnishkafka 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Verified; Looks good to me, approved diff --git a/modules/varnishkafka b/modules/varnishkafka index 238a72d..63949e1 16 --- a/modules/varnishkafka +++ b/modules/varnishkafka @@ -1 +1 @@ -Subproject commit 238a72d304e8df8f1cfb787a6ab077269b86390c +Subproject commit 63949e1cb21e0a66fb974a135102b7795d1e2051 -- To view, visit https://gerrit.wikimedia.org/r/324716 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I0c5004a09487de0881db54ecde4f8f7469ea8158 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Update varnishkafka submodule SHA
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/324716 Change subject: Update varnishkafka submodule SHA .. Update varnishkafka submodule SHA Related gerrit patch: https://gerrit.wikimedia.org/r/#/c/324706/ Change-Id: I0c5004a09487de0881db54ecde4f8f7469ea8158 --- M modules/varnishkafka 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/16/324716/1 diff --git a/modules/varnishkafka b/modules/varnishkafka index 238a72d..63949e1 16 --- a/modules/varnishkafka +++ b/modules/varnishkafka @@ -1 +1 @@ -Subproject commit 238a72d304e8df8f1cfb787a6ab077269b86390c +Subproject commit 63949e1cb21e0a66fb974a135102b7795d1e2051 -- To view, visit https://gerrit.wikimedia.org/r/324716 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I0c5004a09487de0881db54ecde4f8f7469ea8158 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Set daily logrotation for stats JSON files
Elukey has submitted this change and it was merged.
Change subject: Set daily logrotation for stats JSON files
..
Set daily logrotation for stats JSON files
Varnishkafka might produce big JSON stat files during a whole week,
especially when the traffic is high, and the compression happens
only after two rotations (first one to .log.1, then log.1.gz).
This means that logrotate is able to compress a stat json file
after two days rather than two weeks.
Change-Id: I31b3139cf6c18f2410ed5e97e75f74a175f4a95a
---
M templates/varnishkafka-stats.logrotate.erb
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
Ema: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/templates/varnishkafka-stats.logrotate.erb
b/templates/varnishkafka-stats.logrotate.erb
index f99b228..596f47d 100644
--- a/templates/varnishkafka-stats.logrotate.erb
+++ b/templates/varnishkafka-stats.logrotate.erb
@@ -1,5 +1,5 @@
<%= @log_statistics_file %> {
- weekly
+ daily
rotate 4
create 0644 varnishlog root
missingok
--
To view, visit https://gerrit.wikimedia.org/r/324706
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I31b3139cf6c18f2410ed5e97e75f74a175f4a95a
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
Gerrit-Reviewer: BBlack
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ema
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Fix varnishkafka_ganglia.py unit test
Elukey has submitted this change and it was merged. Change subject: Fix varnishkafka_ganglia.py unit test .. Fix varnishkafka_ganglia.py unit test Duplicate map entry caused python warnings when for jenkins tox checks. Change-Id: I9ec5b43c89912ce2b5088f873bf246ec4edc3303 --- M files/varnishkafka_ganglia.py 1 file changed, 0 insertions(+), 1 deletion(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/files/varnishkafka_ganglia.py b/files/varnishkafka_ganglia.py index 680ec32..47aab0f 100755 --- a/files/varnishkafka_ganglia.py +++ b/files/varnishkafka_ganglia.py @@ -458,7 +458,6 @@ } }, '2.1': ['a', 'b'], -'2.1': ['a', 'b'], # '/' should be replaced with key_separator '3/1': 'nonya', 'notme': 'nope', -- To view, visit https://gerrit.wikimedia.org/r/324713 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I9ec5b43c89912ce2b5088f873bf246ec4edc3303 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet/varnishkafka Gerrit-Branch: master Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Ottomata Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Fix varnishkafka_ganglia.py unit test
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/324713 Change subject: Fix varnishkafka_ganglia.py unit test .. Fix varnishkafka_ganglia.py unit test Duplicate map entry caused python warnings when for jenkins tox checks. Change-Id: I9ec5b43c89912ce2b5088f873bf246ec4edc3303 --- M files/varnishkafka_ganglia.py 1 file changed, 0 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/varnishkafka refs/changes/13/324713/1 diff --git a/files/varnishkafka_ganglia.py b/files/varnishkafka_ganglia.py index 680ec32..47aab0f 100755 --- a/files/varnishkafka_ganglia.py +++ b/files/varnishkafka_ganglia.py @@ -458,7 +458,6 @@ } }, '2.1': ['a', 'b'], -'2.1': ['a', 'b'], # '/' should be replaced with key_separator '3/1': 'nonya', 'notme': 'nope', -- To view, visit https://gerrit.wikimedia.org/r/324713 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9ec5b43c89912ce2b5088f873bf246ec4edc3303 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet/varnishkafka Gerrit-Branch: master Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove Ganglia monitoring for Varnishkafka
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/324708
Change subject: Remove Ganglia monitoring for Varnishkafka
..
Remove Ganglia monitoring for Varnishkafka
Bug: T152093
Change-Id: I95deeb9a0d14b9b415268233ae22bd68c156d6f9
---
M modules/role/manifests/cache/kafka/webrequest.pp
1 file changed, 0 insertions(+), 7 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/08/324708/1
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 61db695..01926d9 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -109,13 +109,6 @@
}
include ::standard
-if $::standard::has_ganglia {
-varnishkafka::monitor { 'webrequest':
-# The primary webrequest varnishkafka instance was formerly the
-# only one running, so we don't prefix its Ganglia metric keys.
-key_prefix => '',
-}
-}
# Generate icinga alert if varnishkafka is not running.
nrpe::monitor_service { 'varnishkafka':
--
To view, visit https://gerrit.wikimedia.org/r/324708
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I95deeb9a0d14b9b415268233ae22bd68c156d6f9
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Set daily logrotation for stats JSON files
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/324706
Change subject: Set daily logrotation for stats JSON files
..
Set daily logrotation for stats JSON files
Varnishkafka might produce big JSON stat files during a whole week,
especially when the traffic is high, and the compression happens
only after two rotations (first one to .log.1, then log.1.gz).
This means that logrotate is able to compress a stat json file
after two days rather than two weeks.
Change-Id: I31b3139cf6c18f2410ed5e97e75f74a175f4a95a
---
M templates/varnishkafka-stats.logrotate.erb
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet/varnishkafka
refs/changes/06/324706/1
diff --git a/templates/varnishkafka-stats.logrotate.erb
b/templates/varnishkafka-stats.logrotate.erb
index f99b228..596f47d 100644
--- a/templates/varnishkafka-stats.logrotate.erb
+++ b/templates/varnishkafka-stats.logrotate.erb
@@ -1,5 +1,5 @@
<%= @log_statistics_file %> {
- weekly
+ daily
rotate 4
create 0644 varnishlog root
missingok
--
To view, visit https://gerrit.wikimedia.org/r/324706
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I31b3139cf6c18f2410ed5e97e75f74a175f4a95a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet/varnishkafka
Gerrit-Branch: master
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add openjdk-8-jdk to the list of statistics packages
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/324679 Change subject: Add openjdk-8-jdk to the list of statistics packages .. Add openjdk-8-jdk to the list of statistics packages Bug: T151896 Change-Id: I9906286ce934de61e620f64229355d23e3f320af --- M modules/statistics/manifests/packages.pp 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/79/324679/1 diff --git a/modules/statistics/manifests/packages.pp b/modules/statistics/manifests/packages.pp index 7e1ccc8..df5ea96 100644 --- a/modules/statistics/manifests/packages.pp +++ b/modules/statistics/manifests/packages.pp @@ -35,6 +35,7 @@ 'build-essential', # Requested by halfak to install SciPy 'nodejs', 'openjdk-7-jdk', +'openjdk-8-jdk', 'g++-4.8', # Requested by bearloga (Mikhail) to ensure that there is a compiler with C++11 support that can compile R package 'Boom'; see T147682 and http://stackoverflow.com/a/36034866/1091835 for more info # GNU Scientific Library (GSL) requested by bearloga (Mikhail) 'libgsl0ldbl', # GSL part 1/3 -- To view, visit https://gerrit.wikimedia.org/r/324679 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I9906286ce934de61e620f64229355d23e3f320af Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/refinery[master]: Add fi.wikivoyage to the pageview whitelist
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/324671 Change subject: Add fi.wikivoyage to the pageview whitelist .. Add fi.wikivoyage to the pageview whitelist Change-Id: Ib034f063c05616e7aa99c45eef46bbba8cf9ab69 --- M static_data/pageview/whitelist/whitelist.tsv 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/refinery refs/changes/71/324671/1 diff --git a/static_data/pageview/whitelist/whitelist.tsv b/static_data/pageview/whitelist/whitelist.tsv index 9d59f51..d8d3576 100644 --- a/static_data/pageview/whitelist/whitelist.tsv +++ b/static_data/pageview/whitelist/whitelist.tsv @@ -234,6 +234,7 @@ projectfi.wikiquote2015-10-12 00:00:00 projectfi.wikisource 2015-10-12 00:00:00 projectfi.wikiversity 2015-10-12 00:00:00 +projectfi.wikivoyage 2015-12-01 00:00:00 projectfi.wiktionary 2015-10-12 00:00:00 projectfj.wikipedia2015-10-12 00:00:00 projectfj.wiktionary 2015-10-12 00:00:00 -- To view, visit https://gerrit.wikimedia.org/r/324671 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ib034f063c05616e7aa99c45eef46bbba8cf9ab69 Gerrit-PatchSet: 1 Gerrit-Project: analytics/refinery Gerrit-Branch: master Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove HHVM X-Powered-By header from static Apache responses
Elukey has submitted this change and it was merged.
Change subject: Remove HHVM X-Powered-By header from static Apache responses
..
Remove HHVM X-Powered-By header from static Apache responses
Static pages, served directly by httpd, had a special X-Powered-By
header with the HHVM version on that server. For example:
curl -i -s -k https://www.wikiversity.org | grep -i "X-Powered-By"
X-Powered-By: HHVM/3.3.0-static
This was there historically for analytics, but now it is not
needed anymore.
Change-Id: I7272a1a231580d82e3bd1519c64a14e572d223fb
---
D modules/mediawiki/files/apache/configs/hhvm_mark_engine.conf
M modules/mediawiki/manifests/web/php_engine.pp
2 files changed, 0 insertions(+), 9 deletions(-)
Approvals:
Krinkle: Looks good to me, but someone else must approve
Ori.livneh: Looks good to me, but someone else must approve
Elukey: Looks good to me, approved
Giuseppe Lavagetto: Looks good to me, but someone else must approve
jenkins-bot: Verified
BBlack: Looks good to me, but someone else must approve
diff --git a/modules/mediawiki/files/apache/configs/hhvm_mark_engine.conf
b/modules/mediawiki/files/apache/configs/hhvm_mark_engine.conf
deleted file mode 100644
index 0125824..000
--- a/modules/mediawiki/files/apache/configs/hhvm_mark_engine.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# Sets an X-Powered-by header for static assets
-# This is then used in varnish.
-Header always setifempty X-Powered-By "HHVM/3.3.0-static"
diff --git a/modules/mediawiki/manifests/web/php_engine.pp
b/modules/mediawiki/manifests/web/php_engine.pp
index ac037ba..2ca276e 100644
--- a/modules/mediawiki/manifests/web/php_engine.pp
+++ b/modules/mediawiki/manifests/web/php_engine.pp
@@ -14,12 +14,6 @@
priority => 50,
}
-# Mark static assets as coming from an appserver as well. Needed for
Varnish
-apache::conf { 'mark_engine':
-source =>
'puppet:///modules/mediawiki/apache/configs/hhvm_mark_engine.conf',
-priority => 49,
-}
-
# Add headers lost by mod_proxy_fastcgi
apache::conf { 'fcgi_headers':
source =>
'puppet:///modules/mediawiki/apache/configs/fcgi_headers.conf',
--
To view, visit https://gerrit.wikimedia.org/r/314519
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7272a1a231580d82e3bd1519c64a14e572d223fb
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Alex Monk
Gerrit-Reviewer: BBlack
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ema
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: Krinkle
Gerrit-Reviewer: Muehlenhoff
Gerrit-Reviewer: Ori.livneh
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add temporary override to mc1019 hiera config to allow Redis...
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/323807 Change subject: Add temporary override to mc1019 hiera config to allow Redis config .. Add temporary override to mc1019 hiera config to allow Redis config The mediawiki::redis_servers::eqiad hiera map is used in the memcached role to establish if Redis needs to be configured/installed on the host. Since mc1019 is not part of the official shards (testing phase), a temporary override is needed to trick puppet. Bug: T137345 Change-Id: I13e2baf458ae391e3cbfb1b1028f1cdb81495725 --- A hieradata/hosts/mc1019.yaml 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/07/323807/1 diff --git a/hieradata/hosts/mc1019.yaml b/hieradata/hosts/mc1019.yaml new file mode 100644 index 000..acccf13 --- /dev/null +++ b/hieradata/hosts/mc1019.yaml @@ -0,0 +1,8 @@ +# Temporary override to test mc1019 before adding it to the official +# nutcracker's shards (Redis will not get installed/configured without this +# config). +# Bug: T137345 +mediawiki::redis_servers::eqiad: + shard01: +host: 10.64.0.80 +port: 6379 \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/323807 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I13e2baf458ae391e3cbfb1b1028f1cdb81495725 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add mc1019 to site.pp
Elukey has submitted this change and it was merged.
Change subject: Add mc1019 to site.pp
..
Add mc1019 to site.pp
Bug: T137345
Change-Id: I7168e979640e2fcb60e7fe96e0faf7c23722c787
---
M manifests/site.pp
1 file changed, 7 insertions(+), 0 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/site.pp b/manifests/site.pp
index ac01922..3562103 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1924,6 +1924,13 @@
role(memcached)
}
+# First of the new memcached/redis hosts
+# that will replace mc1001->mc1018.
+# Bug: T137345
+node 'mc1019.eqiad.wmnet' {
+role(memcached)
+}
+
node /^mc20[01][0-9]\.codfw\.wmnet/ {
role(memcached)
}
--
To view, visit https://gerrit.wikimedia.org/r/323805
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7168e979640e2fcb60e7fe96e0faf7c23722c787
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add mc1019 to site.pp
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/323805
Change subject: Add mc1019 to site.pp
..
Add mc1019 to site.pp
Bug: T137345
Change-Id: I7168e979640e2fcb60e7fe96e0faf7c23722c787
---
M manifests/site.pp
1 file changed, 7 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/05/323805/1
diff --git a/manifests/site.pp b/manifests/site.pp
index ac01922..3562103 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1924,6 +1924,13 @@
role(memcached)
}
+# First of the new memcached/redis hosts
+# that will replace mc1001->mc1018.
+# Bug: T137345
+node 'mc1019.eqiad.wmnet' {
+role(memcached)
+}
+
node /^mc20[01][0-9]\.codfw\.wmnet/ {
role(memcached)
}
--
To view, visit https://gerrit.wikimedia.org/r/323805
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7168e979640e2fcb60e7fe96e0faf7c23722c787
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Avoid Redis IPsec replication if the host doesn't need it.
Elukey has submitted this change and it was merged.
Change subject: Avoid Redis IPsec replication if the host doesn't need it.
..
Avoid Redis IPsec replication if the host doesn't need it.
The redis::multidc::ipsec is currently used by JobQueues
and memcached/redis hosts to set up a secure transport
between eqiad and codfw. A new host still not configured
in hiera to have a replica should not get automatically
configured for IPsec to avoid false alarms.
Bug: T137345
Change-Id: Ic64af79619119c9727bc4b7b94ba3084e73db003
---
M modules/redis/manifests/multidc/ipsec.pp
1 file changed, 5 insertions(+), 2 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Giuseppe Lavagetto: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/redis/manifests/multidc/ipsec.pp
b/modules/redis/manifests/multidc/ipsec.pp
index d16df28..ff89c4d 100644
--- a/modules/redis/manifests/multidc/ipsec.pp
+++ b/modules/redis/manifests/multidc/ipsec.pp
@@ -8,7 +8,10 @@
$my_ip = ipresolve($::fqdn, 4)
$ipsec_host_list = redis_shard_hosts($my_ip, $shards)
-class { 'role::ipsec':
-hosts => $ipsec_host_list
+# No reason to define IPsec if the host doesn't need replication.
+if size($ipsec_host_list) > 0 {
+class { 'role::ipsec':
+hosts => $ipsec_host_list
+}
}
}
--
To view, visit https://gerrit.wikimedia.org/r/323517
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ic64af79619119c9727bc4b7b94ba3084e73db003
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove cron notifications to root@ for jobchron/runner servi...
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/323528
Change subject: Remove cron notifications to root@ for jobchron/runner service
status
..
Remove cron notifications to root@ for jobchron/runner service status
Bug: T132324
Change-Id: I325403be1d0b786ace3518bdb574cf711322c3d4
---
M modules/mediawiki/templates/jobrunner/logrotate-jobchron.conf.erb
M modules/mediawiki/templates/jobrunner/logrotate.conf.erb
2 files changed, 2 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/28/323528/1
diff --git a/modules/mediawiki/templates/jobrunner/logrotate-jobchron.conf.erb
b/modules/mediawiki/templates/jobrunner/logrotate-jobchron.conf.erb
index 43982fb..d3b6970 100644
--- a/modules/mediawiki/templates/jobrunner/logrotate-jobchron.conf.erb
+++ b/modules/mediawiki/templates/jobrunner/logrotate-jobchron.conf.erb
@@ -11,7 +11,7 @@
nocreate
postrotate
<%- if scope.lookupvar('::initsystem') == 'systemd' -%>
-if /bin/systemctl is-active jobchron.service; then
+if /bin/systemctl -q is-active jobchron.service; then
/bin/systemctl restart jobchron.service
fi
<%- else -%>
diff --git a/modules/mediawiki/templates/jobrunner/logrotate.conf.erb
b/modules/mediawiki/templates/jobrunner/logrotate.conf.erb
index 4c99e75..201cbc3 100644
--- a/modules/mediawiki/templates/jobrunner/logrotate.conf.erb
+++ b/modules/mediawiki/templates/jobrunner/logrotate.conf.erb
@@ -10,7 +10,7 @@
nocreate
postrotate
<%- if scope.lookupvar('::initsystem') == 'systemd' -%>
-if /bin/systemctl is-active jobrunner.service; then
+if /bin/systemctl -q is-active jobrunner.service; then
/bin/systemctl restart jobrunner.service
fi
<%- else -%>
--
To view, visit https://gerrit.wikimedia.org/r/323528
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I325403be1d0b786ace3518bdb574cf711322c3d4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Avoid Redis IPsec replication if the host doesn't need it.
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/323517
Change subject: Avoid Redis IPsec replication if the host doesn't need it.
..
Avoid Redis IPsec replication if the host doesn't need it.
The redis::multidc::ipsec is currently used by JobQueues
and memcached/redis hosts to set up a secure transport
between eqiad and codfw. A new host still not configured
in hiera to have a replica should not get automatically
configured for IPsec to avoid false alarms.
Bug: T137345
Change-Id: Ic64af79619119c9727bc4b7b94ba3084e73db003
---
M modules/redis/manifests/multidc/ipsec.pp
1 file changed, 5 insertions(+), 2 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/17/323517/1
diff --git a/modules/redis/manifests/multidc/ipsec.pp
b/modules/redis/manifests/multidc/ipsec.pp
index d16df28..e6c9c8d 100644
--- a/modules/redis/manifests/multidc/ipsec.pp
+++ b/modules/redis/manifests/multidc/ipsec.pp
@@ -8,7 +8,10 @@
$my_ip = ipresolve($::fqdn, 4)
$ipsec_host_list = redis_shard_hosts($my_ip, $shards)
-class { 'role::ipsec':
-hosts => $ipsec_host_list
+# No reason to define IPsec if the host doesn't need replication.
+if $ipsec_host_list != [] {
+class { 'role::ipsec':
+hosts => $ipsec_host_list
+}
}
}
--
To view, visit https://gerrit.wikimedia.org/r/323517
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic64af79619119c9727bc4b7b94ba3084e73db003
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Force a 404 on each HTTP request landing to a non configured...
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/322268 Change subject: Force a 404 on each HTTP request landing to a non configured domain .. Force a 404 on each HTTP request landing to a non configured domain Bug: T137176 Change-Id: Icbe122a4aec0400adc049f846b0ad355a33b8a4c --- M modules/mediawiki/files/apache/sites/nonexistent.conf 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/68/322268/1 diff --git a/modules/mediawiki/files/apache/sites/nonexistent.conf b/modules/mediawiki/files/apache/sites/nonexistent.conf index 494561e..e61299e 100644 --- a/modules/mediawiki/files/apache/sites/nonexistent.conf +++ b/modules/mediawiki/files/apache/sites/nonexistent.conf @@ -3,5 +3,6 @@ DocumentRoot "/srv/mediawiki/docroot/default" ServerName nonexistent ErrorDocument 404 /index.html +Redirect 404 / -- To view, visit https://gerrit.wikimedia.org/r/322268 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icbe122a4aec0400adc049f846b0ad355a33b8a4c Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: [WIP] Refactor the parsing functions out of the main C file
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/322257
Change subject: [WIP] Refactor the parsing functions out of the main C file
..
[WIP] Refactor the parsing functions out of the main C file
This change should be a no-op, the plan is to move parsing
related functions on separate files and expose only the
functions needed via header files.
The next step will be to remove as much as possible
the use or global status from the parsing functions,
allowing them to be idempotent (and unit-testable).
Bug: T147440
Change-Id: Icaf31dd569fd87a6c19a5062832ccdd647955e77
---
M Makefile
A parsers.c
A parsers.h
A string_utils.c
A string_utils.h
M varnishkafka.c
6 files changed, 597 insertions(+), 397 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/operations/software/varnish/varnishkafka
refs/changes/57/322257/1
diff --git a/Makefile b/Makefile
index 1370bc8..5c00253 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,6 @@
PROG= varnishkafka
-SRCS= varnishkafka.c config.c base64.c
+SRCS= varnishkafka.c config.c base64.c string_utils.c parsers.c
DESTDIR?=/usr/local
diff --git a/parsers.c b/parsers.c
new file mode 100644
index 000..ab2fa8c
--- /dev/null
+++ b/parsers.c
@@ -0,0 +1,359 @@
+/*
+ * varnishkafka
+ *
+ * Copyright (c) 2016 Wikimedia Foundation
+ *
+ * All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * 1. Redistributions of source code must retain the above copyright notice,
+ *this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright notice,
+ *this list of conditions and the following disclaimer in the documentation
+ *and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+
+#define _GNU_SOURCE /* for strndupa() */
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+#include
+
+#include "varnishkafka.h"
+#include "string_utils.h"
+#include "base64.h"
+
+/**
+ * Allocate persistent memory space ('len' bytes) in lp scratch
+ */
+static char *scratch_alloc (struct logline *lp, size_t len) {
+char *ptr;
+
+if (unlikely(len > conf.scratch_size || (conf.scratch_size - len) <
lp->sof)) {
+vk_log("SCRATCH", LOG_CRIT, "Ran out of scratch_size, limit is %zu",
conf.scratch_size);
+exit(99);
+}
+
+ptr = lp->scratch + lp->sof;
+lp->sof += len;
+return ptr;
+}
+
+
+/**
+ * essentially, scratch_alloc + memcpy
+ */
+static char *scratch_cpy (struct logline *lp, const char *src, size_t len) {
+char *dst;
+
+dst = scratch_alloc(lp, len);
+memcpy(dst, src, len);
+return dst;
+}
+
+
+/**
+ * like scratch_cpy, but escapes all unprintable characters as well as the
+ * ones defined in 'map' below. sets *len to the escaped length of the result
+ */
+static char* scratch_cpy_esc (struct logline *lp, const char *src, size_t*
len) {
+static const char *map[256] = {
+['\t'] = "\\t",
+['\n'] = "\\n",
+['\r'] = "\\r",
+['\v'] = "\\v",
+['\f'] = "\\f",
+['"'] = "\\\"",
+[' '] = "\\ ",
+};
+
+
+/* Allocate initial space for escaped string.
+ * The maximum expansion size per character is 5 (octal coding).
+ */
+const size_t in_len = *len;
+char dst[in_len * 5];
+char *d = dst;
+const char *s = src;
+const char *srcend = src + in_len;
+
+while (s < srcend) {
+size_t outlen = 1;
+const char *out;
+char tmp[6];
+
+if (unlikely((out = map[(int)*s]) != NULL)) {
+/* Escape from 'map' */
+outlen = 2;
+} else if (unlikely(!isprint(*s))) {
+/* Escape non-printables as \ */
+sprintf(tmp, "\%04o", (int)*s);
+out = tmp;
+outlen = 5;
+} else {
+/* No escaping */
+out = s;
+}
+
+assert(out
[MediaWiki-commits] [Gerrit] operations...varnishkafka[master]: Move definitions to header files for a better code readability
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/322256
Change subject: Move definitions to header files for a better code readability
..
Move definitions to header files for a better code readability
Split the map struct definition from its declaration and moved some
header inclusions in varnishkafka.h
Change-Id: I1f7799b2ae6e06ad37eedc287e983a8edeb2a663
---
M varnishkafka.c
M varnishkafka.h
2 files changed, 65 insertions(+), 63 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/operations/software/varnish/varnishkafka
refs/changes/56/322256/1
diff --git a/varnishkafka.c b/varnishkafka.c
index b6d7a62..c395290 100644
--- a/varnishkafka.c
+++ b/varnishkafka.c
@@ -48,17 +48,6 @@
#include
#include
-#include
-#include
-#include
-#include
-
-#include
-
-#include
-#include
-#include
-
#include "varnishkafka.h"
#include "base64.h"
@@ -747,59 +736,8 @@
*/
static int format_parse (const char *format_orig,
char *errstr, size_t errstr_size) {
- /**
-* Maps a formatter %X to a VSL tag and column id, or parser, or both
-*/
- struct {
- /* A formatter may be backed by multiple tags.
-* The first matching tag observed in the log will be used.
-*
-* Example:
-* A formatter declared as following in 'map':
-* ['x'] = { {
-* { VSL_CLIENTMARKER, SLT_Timestamp,
-*.var = "Process",
-*.fmtvar = "Varnish:time_firstbyte", .col = 2
},
-* { VSL_CLIENTMARKER, SLT_Begin,
-*.fmtvar = "Varnish:xvid", .col = 2 }
-* } },
-*
-* In this case, the struct below is replicated two
-* times to match the 'x' formatter to multiple
-* Varnish tags (establishing also a priority).
-* Therefore, supposing that 'x' corresponds to 'map[42]',
-* the following variables will be accessible:
-* - map[42].f[0].var==> "Process"
-* - map[42].f[1].fmtvar ==> "Varnish:xvid"
-*/
- struct {
- /* VSL_CLIENTMARKER or VSL_BACKENDMARKER, or both */
- int spec;
- /* The SLT_.. tag id */
- int tag;
- /* For "Name: Value" tags (such as SLT_RespHeader),
-* this is the "Name" part. */
- const char *var;
- /* Special handling for non-name-value vars such as
-* %{Varnish:handling}x. fmtvar is "Varnish:handling" */
- const char *fmtvar;
- /* Column to extract:
-* 0 for entire string, else 1, 2, .. */
- int col;
- /* Parser to manually extract and/or convert a
-* tag's content. */
- size_t (*parser) (const struct tag *tag,
- struct logline *lp,
- const char *ptr, size_t len);
- /* Optional tag->flags */
- int tag_flags;
- } f[5+1]; /* increase size when necessary (max used size + 1) */
- /* Default string if no matching tag was found or all
-* parsers failed, defaults to "-". */
- const char *def;
-
- } map[256] = {
+ struct VSL_Tag_Formatter map[256] = {
/* Indexed by formatter character as
* specified by varnishncsa(1).
* Each formatter is associated with
diff --git a/varnishkafka.h b/varnishkafka.h
index 681f875..9d5b68e 100644
--- a/varnishkafka.h
+++ b/varnishkafka.h
@@ -32,6 +32,17 @@
#include
+#include
+#include
+#include
+#include
+
+#include
+
+#include
+#include
+#include
+
#ifndef likely
#define likely(x) __builtin_expect((x),1)
#endif
@@ -229,3 +240,56 @@
void out_stdout (struct logline *lp, const char *buf, size_t len);
void out_null (struct logline *lp, const char *buf, size_t len);
extern void (*outfunc) (struct logline *lp, const char *buf, size_t len);
+
+/**
+* Maps a formatter %X to a VSL tag and column id, or parser, or both
+*/
+struct VSL_Tag_Formatter {
+/* A formatter may be backed by multiple tags.
+ * The first matching tag observed in the log will be used.
+ *
+ * Example:
+ * A formatter declared as following in 'map':
+ * ['x'] = { {
+ * { VSL_CLIENTMARKER, SLT_Timestamp,
+ *.var = "Process",
+ *.fmtvar = "Varnish:time_firstbyte", .col = 2 },
+ * { VSL_CLIENTMARKER,
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Disable connection tracking for kafka broker
Elukey has submitted this change and it was merged.
Change subject: Disable connection tracking for kafka broker
..
Disable connection tracking for kafka broker
During several traffic peaks we've run into exhausting
the connection tracking table in the past and rather
than bumping the size further, let's disable connection
tracking as we already do for other high volume services.
Change-Id: If26b300e0cae4d8adf26b1516d18a80d08b4f3de
---
M modules/role/manifests/kafka/analytics/broker.pp
1 file changed, 4 insertions(+), 3 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/role/manifests/kafka/analytics/broker.pp
b/modules/role/manifests/kafka/analytics/broker.pp
index 45e6cd5..a68e3b7 100644
--- a/modules/role/manifests/kafka/analytics/broker.pp
+++ b/modules/role/manifests/kafka/analytics/broker.pp
@@ -115,11 +115,12 @@
}
# firewall Kafka Broker
ferm::service { 'kafka-broker':
-proto => 'tcp',
+proto => 'tcp',
# TODO: $::confluent::kafka::broker::port doesn't
# seem to work as expected. Hardcoding this for now.
-port => 9092,
-srange => '($PRODUCTION_NETWORKS $FRACK_NETWORKS)',
+port=> 9092,
+notrack => true,
+srange => '($PRODUCTION_NETWORKS $FRACK_NETWORKS)',
}
include ::ferm::ipsec_allow
--
To view, visit https://gerrit.wikimedia.org/r/320758
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: If26b300e0cae4d8adf26b1516d18a80d08b4f3de
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Muehlenhoff
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...testing[master]: First Docker prototype
Elukey has submitted this change and it was merged.
Change subject: First Docker prototype
..
First Docker prototype
This repository will be used to as tool to test Varnishkafka
before releasing it to Production. I have basically automated
my testing environment that I have been using during the past
months. It is not meant to be super generic but only to satisfy
the Wikimedia use case.
Bug: T147442
Change-Id: I3c3fd19a62ec67b60ccf7f7f728104a84c62a005
---
A Dockerfile
A README.md
A check_output.py
A run_test
A services_config/apache_ports.conf
A services_config/apache_vhost.conf
A services_config/apt_preferences
A services_config/php_fpm.conf
A services_config/test.php
A services_config/varnish_default
A services_config/varnish_default.vcl
A services_config/varnishkafka.conf
12 files changed, 730 insertions(+), 0 deletions(-)
Approvals:
Elukey: Verified; Looks good to me, approved
diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 000..8a086c0
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,79 @@
+# Copyright © 2016 Luca Toscano
+# Copyright © 2016 Wikimedia Foundation, Inc.
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+##
+# Dockerfile to create a simple testing environment:
+# - Varnish 4
+# - Apache 2.4
+# - PHP FPM 5
+# - Varnishkafka (and related VSL dev libs)
+# - Curl and Python
+#
+# GOAL: Set up a basic Web server scenario to test the local
+# Varnishkafka repository.
+#
+# ASSUMPTIONS:
+# 1) The OS is Debian Jessie (especially for the Apache config).
+# 2) The varnishakafka repo is inside the docker build path.
+# 3) The run_test script is run by the user.
+###
+
+# Limiting the tests to Debian stable
+FROM debian:latest
+
+# Path to the varnishkafka source files
+ARG VARNISHKAFKA_SOURCE_PATH=varnishkafka
+
+ARG SERVICES_CONFIG_PATH=services_config
+
+# Adding Jessie Wikimedia APT repository
+RUN apt-get update
+RUN apt-get install -y apt-transport-https curl
+RUN curl -s
"https://wikitech.wikimedia.org/w/index.php?title=APT_repository/Key&action=raw";
| apt-key add -
+RUN echo "deb http://apt.wikimedia.org/wikimedia jessie-wikimedia main
experimental backports" >> /etc/apt/sources.list
+COPY ${SERVICES_CONFIG_PATH}/apt_preferences /etc/apt/preferences
+
+# Update the repository sources list
+RUN apt-get update && apt-get -y dist-upgrade
+
+
+# Install Varnish 4 and Apache 2.4
+RUN apt-get install -y varnish apache2 php5-fpm
+
+# Install Varnishkafka dev libs
+RUN apt-get install -y librdkafka-dev libvarnishapi-dev zlib1g-dev libyajl-dev
git gcc make python
+
+# Copy configuration files
+RUN mkdir -p /var/www/test
+COPY ${SERVICES_CONFIG_PATH}/test.php /var/www/test/test.php
+COPY ${SERVICES_CONFIG_PATH}/varnish_default.vcl /etc/varnish/default.vcl
+COPY ${SERVICES_CONFIG_PATH}/apache_vhost.conf
/etc/apache2/sites-available/001-test.conf
+RUN rm /etc/apache2/sites-enabled/*
+RUN ln -s /etc/apache2/sites-available/001-test.conf
/etc/apache2/sites-enabled/001-test.conf
+COPY ${SERVICES_CONFIG_PATH}/apache_ports.conf /etc/apache2/ports.conf
+COPY ${SERVICES_CONFIG_PATH}/php_fpm.conf /etc/php5/fpm/pool.d/www.conf
+COPY ${SERVICES_CONFIG_PATH}/varnish_default /etc/default/varnish
+
+RUN mkdir /tmp/varnishkafka
+RUN mkdir -p /var/cache/varnishkafka
+COPY ${VARNISHKAFKA_SOURCE_PATH}/* /tmp/varnishkafka/
+COPY ${SERVICES_CONFIG_PATH}/varnishkafka.conf
/tmp/varnishkafka/varnishkafka.conf
+
+# Copy the test script
+COPY run_test /run_test
+RUN chmod +x /run_test
+
+# Copy validation script
+COPY check_output.py /tmp/varnishkafka/check_vk_output.py
diff --git a/README.md b/README.md
new file mode 100644
index 000..3071021
--- /dev/null
+++ b/README.md
@@ -0,0 +1,62 @@
+Basic integration testing framework with Docker for Varnishkafka
+==
+
+Copyright (c) 2016 [Wikimedia Foundation](http://www.wikimedia.org)
+
+Copyright (c) 2016 Luca Toscano
+
+
+# Description
+
+A simple Docker image to set up a basic Web Server scenario
+(Varnish 4 -> Apache httpd 2.4 -> PHP-FPM) to test Varnishkafka.
+The goal is to provide a simple integration testing environments
+to test new code before releasing it.
+
+Varnishkafka is tested only in stdout mode (JSON), no Kafka broker(s)
+will be created. Future versions of this tool will probabl
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Raise nagios retry_interval to avoid false alarms for HHVM r...
Elukey has submitted this change and it was merged.
Change subject: Raise nagios retry_interval to avoid false alarms for HHVM
restarts
..
Raise nagios retry_interval to avoid false alarms for HHVM restarts
In T147773 a regular HHVM restart workflow was introduced. It may
happen that the two retries for check_http_wikipedia_main are quicker
to complete than the restart, so we get false alarms that can confuse
people watching icinga/IRC.
Bug: T147773
Change-Id: I69168e1e9babc3f4805b715195ed065c052ed7de
---
M modules/role/manifests/mediawiki/webserver.pp
1 file changed, 10 insertions(+), 6 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Giuseppe Lavagetto: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/role/manifests/mediawiki/webserver.pp
b/modules/role/manifests/mediawiki/webserver.pp
index 78898b8..857d4e6 100644
--- a/modules/role/manifests/mediawiki/webserver.pp
+++ b/modules/role/manifests/mediawiki/webserver.pp
@@ -47,16 +47,20 @@
# If a service check happens to run while we are performing a
# graceful restart of Apache, we want to try again before declaring
# defeat. See T103008.
+# We want to avoid false alarms during scheduled HHVM restarts (T147773),
+# so a higher retry_interval is needed.
monitoring::service { 'appserver http':
-description => 'Apache HTTP',
-check_command => 'check_http_wikipedia',
-retries => 2,
+description=> 'Apache HTTP',
+check_command => 'check_http_wikipedia',
+retries=> 2,
+retry_interval => 2,
}
monitoring::service { 'appserver_http_hhvm':
-description => 'HHVM rendering',
-check_command => 'check_http_wikipedia_main',
-retries => 2,
+description=> 'HHVM rendering',
+check_command => 'check_http_wikipedia_main',
+retries=> 2,
+retry_interval => 2,
}
nrpe::monitor_service { 'hhvm':
--
To view, visit https://gerrit.wikimedia.org/r/320361
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I69168e1e9babc3f4805b715195ed065c052ed7de
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Alexandros Kosiaris
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Raise nagios retry_interval to avoid false alarms for HHVM r...
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/320361
Change subject: Raise nagios retry_interval to avoid false alarms for HHVM
restarts
..
Raise nagios retry_interval to avoid false alarms for HHVM restarts
In T147773 a regular HHVM restart workflow was introduced. It may
happen that the two retries for check_http_wikipedia_main are quicker
to complete than the restart, so we get false alarms that can confuse
people watching icinga/IRC.
Bug: T147773
Change-Id: I69168e1e9babc3f4805b715195ed065c052ed7de
---
M modules/mariadb
M modules/role/manifests/mediawiki/webserver.pp
2 files changed, 11 insertions(+), 7 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/61/320361/1
diff --git a/modules/mariadb b/modules/mariadb
index 4d96e95..a639204 16
--- a/modules/mariadb
+++ b/modules/mariadb
@@ -1 +1 @@
-Subproject commit 4d96e95500cfbceb8fa552e503644d387e56e334
+Subproject commit a6392048b98875ae19a51a1551877569bf1f9e3b
diff --git a/modules/role/manifests/mediawiki/webserver.pp
b/modules/role/manifests/mediawiki/webserver.pp
index 78898b8..857d4e6 100644
--- a/modules/role/manifests/mediawiki/webserver.pp
+++ b/modules/role/manifests/mediawiki/webserver.pp
@@ -47,16 +47,20 @@
# If a service check happens to run while we are performing a
# graceful restart of Apache, we want to try again before declaring
# defeat. See T103008.
+# We want to avoid false alarms during scheduled HHVM restarts (T147773),
+# so a higher retry_interval is needed.
monitoring::service { 'appserver http':
-description => 'Apache HTTP',
-check_command => 'check_http_wikipedia',
-retries => 2,
+description=> 'Apache HTTP',
+check_command => 'check_http_wikipedia',
+retries=> 2,
+retry_interval => 2,
}
monitoring::service { 'appserver_http_hhvm':
-description => 'HHVM rendering',
-check_command => 'check_http_wikipedia_main',
-retries => 2,
+description=> 'HHVM rendering',
+check_command => 'check_http_wikipedia_main',
+retries=> 2,
+retry_interval => 2,
}
nrpe::monitor_service { 'hhvm':
--
To view, visit https://gerrit.wikimedia.org/r/320361
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I69168e1e9babc3f4805b715195ed065c052ed7de
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/refinery[master]: Add ec.wikimedia.org to the pageview whitelist
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/320351 Change subject: Add ec.wikimedia.org to the pageview whitelist .. Add ec.wikimedia.org to the pageview whitelist Change-Id: Ie1787a4fb1b20cfdfd9a74de62b4572c76edf7e5 --- M static_data/pageview/whitelist/whitelist.tsv 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/analytics/refinery refs/changes/51/320351/1 diff --git a/static_data/pageview/whitelist/whitelist.tsv b/static_data/pageview/whitelist/whitelist.tsv index 31d11b2..9d59f51 100644 --- a/static_data/pageview/whitelist/whitelist.tsv +++ b/static_data/pageview/whitelist/whitelist.tsv @@ -174,6 +174,7 @@ projectdv.wiktionary 2015-10-12 00:00:00 projectdz.wikipedia2015-10-12 00:00:00 projectdz.wiktionary 2015-10-12 00:00:00 +projectec.wikimedia2016-11-08 00:00:00 projectee.wikimedia2016-11-01 00:00:00 projectee.wikipedia2015-10-12 00:00:00 projectel.wikibooks2015-10-12 00:00:00 -- To view, visit https://gerrit.wikimedia.org/r/320351 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ie1787a4fb1b20cfdfd9a74de62b4572c76edf7e5 Gerrit-PatchSet: 1 Gerrit-Project: analytics/refinery Gerrit-Branch: master Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations...testing[master]: First Docker prototype
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/319548
Change subject: First Docker prototype
..
First Docker prototype
This repository will be used to as tool to test Varnishkafka
before releasing it to Production. I have basically automated
my testing environment that I have been using during the past
months. It is not meant to be super generic but only to satisfy
the Wikimedia use case.
Bug: T147442
Change-Id: I3c3fd19a62ec67b60ccf7f7f728104a84c62a005
---
A Dockerfile
A check_output.py
A run_test
A services_config/apache_ports.conf
A services_config/apache_vhost.conf
A services_config/php_fpm.conf
A services_config/test.php
A services_config/varnish_default
A services_config/varnish_default.vcl
A services_config/varnishkafka.conf
10 files changed, 565 insertions(+), 0 deletions(-)
git pull
ssh://gerrit.wikimedia.org:29418/operations/software/varnish/varnishkafka/testing
refs/changes/48/319548/1
diff --git a/Dockerfile b/Dockerfile
new file mode 100644
index 000..deb33d8
--- /dev/null
+++ b/Dockerfile
@@ -0,0 +1,62 @@
+##
+# Dockerfile to create a simple testing environment:
+# - Varnish 4
+# - Apache 2.4
+# - PHP FPM 5
+# - Varnishkafka (and related VSL dev libs)
+# - Curl and Python
+#
+# GOAL: Set up a basic Web server scenario to test the local
+# Varnishkafka repository.
+#
+# ASSUMPTIONS:
+# 1) The OS is Debian Jessie (especially for the Apache config).
+# 2) The varnishakafka repo is inside the docker build path.
+# 3) The run_test script is run by the user.
+###
+
+# Limiting the tests to Debian stable
+FROM debian:latest
+
+# Update the repository sources list
+RUN apt-get update && apt-get -y upgrade
+
+# Varnish specific repositories
+RUN apt-get install -y apt-transport-https curl
+RUN curl https://repo.varnish-cache.org/GPG-key.txt | apt-key add -
+RUN echo "deb https://repo.varnish-cache.org/debian/ jessie varnish-4.1" >>
/etc/apt/sources.list.d/varnish-cache.list
+RUN apt-get update
+
+# Install Varnish 4 and Apache 2.4
+RUN apt-get install -y varnish apache2 php5-fpm
+
+# Install Varnishkafka dev libs
+RUN apt-get install -y librdkafka-dev libvarnishapi-dev zlib1g-dev libyajl-dev
git gcc make python
+
+# Path to the varnishkafka source files
+ARG VARNISHKAFKA_SOURCE_PATH=varnishkafka
+
+ARG SERVICES_CONFIG_PATH=services_config
+
+# Copy configuration files
+RUN mkdir -p /var/www/test
+COPY ${SERVICES_CONFIG_PATH}/test.php /var/www/test/test.php
+COPY ${SERVICES_CONFIG_PATH}/varnish_default.vcl /etc/varnish/default.vcl
+COPY ${SERVICES_CONFIG_PATH}/apache_vhost.conf
/etc/apache2/sites-available/001-test.conf
+RUN rm /etc/apache2/sites-enabled/*
+RUN ln -s /etc/apache2/sites-available/001-test.conf
/etc/apache2/sites-enabled/001-test.conf
+COPY ${SERVICES_CONFIG_PATH}/apache_ports.conf /etc/apache2/ports.conf
+COPY ${SERVICES_CONFIG_PATH}/php_fpm.conf /etc/php5/fpm/pool.d/www.conf
+COPY ${SERVICES_CONFIG_PATH}/varnish_default /etc/default/varnish
+
+RUN mkdir /tmp/varnishkafka
+RUN mkdir -p /var/cache/varnishkafka
+COPY ${VARNISHKAFKA_SOURCE_PATH}/* /tmp/varnishkafka/
+COPY ${SERVICES_CONFIG_PATH}/varnishkafka.conf
/tmp/varnishkafka/varnishkafka.conf
+
+# Copy the test script
+COPY run_test /run_test
+RUN chmod +x /run_test
+
+# Copy validation script
+COPY check_output.py /tmp/varnishkafka/check_vk_output.py
diff --git a/check_output.py b/check_output.py
new file mode 100644
index 000..9c39050
--- /dev/null
+++ b/check_output.py
@@ -0,0 +1,108 @@
+# Copyright © 2016 Luca Toscano
+# Copyright © 2016 Wikimedia Foundation, Inc.
+
+'''
+This Script checks a file containing Varnishkafka stdout
+(formatted in JSON) and performs basic sanity checks.
+It is not meant to be generic or applicable to a wide variety
+of use cases, conversely it is meant to test only
+the Wikimedia use case.
+'''
+
+
+
+import argparse
+import json
+import logging
+
+# Required fields to check for each line. These fields needs to be in sync
+# with what specified in varnishkafka.conf
+SEQUENCE_NUMBER = "sequence"
+DATE_TIMESTAMP = "dt"
+HTTP_STATUS = "http_status"
+HTTP_METHOD = "http_method"
+RESPONSE_SIZE = "response_size"
+
+rf = ["sequence", "dt",
+ "user_agent", "response_size",
+ "http_status", "uri_path",
+ "uri_query", "content_type",
+ "http_method"]
+
+logging.basicConfig()
+logger = logging.getLogger(__name__)
+
+def main(filepath, start_seqno):
+current_seq_no = 0
+if start_seqno:
+current_seq_no = start_seqno
+validation_error = 0
+
+with open(filepath, 'r') as fd:
+for line in fd:
+# Ignore empty lines
+if not line.strip():
+continue
+try:
+# Check that it is valid JSON
+json_line = json.loads(line)
+
+# Check
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add new mc* servers to site.pp with role:spare
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/319278
Change subject: Add new mc* servers to site.pp with role:spare
..
Add new mc* servers to site.pp with role:spare
Bug: T137345
Change-Id: I5f031156ddeba9dbddf8c79e19a4a2a9b19b0afa
---
M manifests/site.pp
1 file changed, 6 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/78/319278/1
diff --git a/manifests/site.pp b/manifests/site.pp
index 9b34095..9958644 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1934,6 +1934,12 @@
role(memcached)
}
+# New memcached/redis servers
+# T137345
+node /^mc10(19|2[0-9]|3[0-6])\.eqiad\.wmnet/ {
+role(spare::system)
+}
+
node /^mc20[01][0-9]\.codfw\.wmnet/ {
role(memcached)
}
--
To view, visit https://gerrit.wikimedia.org/r/319278
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5f031156ddeba9dbddf8c79e19a4a2a9b19b0afa
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: oozie: Restrict to analytics networks
Elukey has submitted this change and it was merged.
Change subject: oozie: Restrict to analytics networks
..
oozie: Restrict to analytics networks
We're getting rid of $INTERNAL, since it's needlessly broad. oozie
needs to be accessed from stat100[24], hadoop and hue, so restrict
access to the analytics networks.
Change-Id: If240f6d68d91410ded6f2d59cbd1e8e9a0d193ac
---
M modules/role/manifests/analytics_cluster/oozie/server.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/role/manifests/analytics_cluster/oozie/server.pp
b/modules/role/manifests/analytics_cluster/oozie/server.pp
index fa070c6..ff08e07 100644
--- a/modules/role/manifests/analytics_cluster/oozie/server.pp
+++ b/modules/role/manifests/analytics_cluster/oozie/server.pp
@@ -42,7 +42,7 @@
ferm::service{ 'oozie_server':
proto => 'tcp',
port => '11000',
-srange => '$INTERNAL',
+srange => '$ANALYTICS_NETWORKS',
}
# Include icinga alerts if production realm.
--
To view, visit https://gerrit.wikimedia.org/r/316359
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: If240f6d68d91410ded6f2d59cbd1e8e9a0d193ac
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Muehlenhoff
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Ottomata
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Force Content-type for files without extensions (noc.w.o)
Elukey has submitted this change and it was merged. Change subject: Force Content-type for files without extensions (noc.w.o) .. Force Content-type for files without extensions (noc.w.o) Bug: T146421 Change-Id: I29820d95b0bf77af591352a1e9fbae5e3bf2731f --- M modules/noc/templates/noc.wikimedia.org.erb 1 file changed, 7 insertions(+), 0 deletions(-) Approvals: Elukey: Looks good to me, approved Alexandros Kosiaris: Looks good to me, but someone else must approve jenkins-bot: Verified diff --git a/modules/noc/templates/noc.wikimedia.org.erb b/modules/noc/templates/noc.wikimedia.org.erb index 8192ca6..d61ccd8 100644 --- a/modules/noc/templates/noc.wikimedia.org.erb +++ b/modules/noc/templates/noc.wikimedia.org.erb @@ -35,5 +35,12 @@ AddDefaultCharset utf-8 +# Force the Content-Type to text/plain +# for files without an extension, since +# AddDefaultCharset works only +# for text/plain or text/html. + +ForceType text/plain + -- To view, visit https://gerrit.wikimedia.org/r/318074 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I29820d95b0bf77af591352a1e9fbae5e3bf2731f Gerrit-PatchSet: 2 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Alexandros Kosiaris Gerrit-Reviewer: Elukey Gerrit-Reviewer: Faidon Liambotis Gerrit-Reviewer: Giuseppe Lavagetto Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] analytics/wikistats[master]: Replace Bugzilla links by Phabricator links
Elukey has submitted this change and it was merged. Change subject: Replace Bugzilla links by Phabricator links .. Replace Bugzilla links by Phabricator links Change-Id: I5ca3091d899670084dfc2a3dfa115485d438f111 --- M squids/perl/SquidReportArchive.pl 1 file changed, 2 insertions(+), 22 deletions(-) Approvals: Elukey: Looks good to me, approved Paladox: Looks good to me, but someone else must approve Dzahn: Looks good to me, but someone else must approve jenkins-bot: Verified diff --git a/squids/perl/SquidReportArchive.pl b/squids/perl/SquidReportArchive.pl index 23fb268..349a454 100755 --- a/squids/perl/SquidReportArchive.pl +++ b/squids/perl/SquidReportArchive.pl @@ -4,7 +4,7 @@ # https://bugzilla.wikimedia.org/show_bug.cgi?id=46205 fixed # https://bugzilla.wikimedia.org/show_bug.cgi?id=46267 # https://bugzilla.wikimedia.org/show_bug.cgi?id=46277 -# https://bugzilla.wikimedia.org/show_bug.cgi?id=46289 +# https://bugzilla.wikimedia.org/show_bug.cgi?id=46289 fixed # -v -q 2012q3 -c -i "w:/# out stat1/squid/csv" -o "w:/# out test/squid/reports" -l . -a "w:/# out stat1/squid/csv/meta" @@ -667,10 +667,6 @@ # $notice = "" . # " Unresolved Bugzilla bugs: " . # "55443" . - # "" . - #"Recently resolved bugs: " . - # "46205 (Aug 2013)" . - # "46289 (Nov 2013)" . # "" ; } else @@ -678,23 +674,7 @@ $notice = "" . # "Warning: all recent Wikimedia traffic analysis reports have been generated from old scripts." . # "The scripts are orphaned, and have not been maintained for at least 6 months. Many bugs are considerably older." . - " Unresolved Bugzilla bugs: " . -"46190, " . -"46191, " . -"46195, " . -"46201, " . -"46265, " . -"(46267), " . -"46268, " . -"46269, " . -"46271, " . -"46273, " . -"46274, " . -"46275, " . -"46277, " . -"46278, " . -"46279, " . -"57376" . + " Unresolved bugs " . "" ; } $header =~ s/NOTICE/$notice/ ; -- To view, visit https://gerrit.wikimedia.org/r/315417 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5ca3091d899670084dfc2a3dfa115485d438f111 Gerrit-PatchSet: 1 Gerrit-Project: analytics/wikistats Gerrit-Branch: master Gerrit-Owner: Aklapper Gerrit-Reviewer: Dzahn Gerrit-Reviewer: Elukey Gerrit-Reviewer: Paladox Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Force Content-type for files without extensions (noc.w.o)
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/318074 Change subject: Force Content-type for files without extensions (noc.w.o) .. Force Content-type for files without extensions (noc.w.o) Bug: T146421 Change-Id: I29820d95b0bf77af591352a1e9fbae5e3bf2731f --- M modules/noc/templates/noc.wikimedia.org.erb 1 file changed, 7 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/74/318074/1 diff --git a/modules/noc/templates/noc.wikimedia.org.erb b/modules/noc/templates/noc.wikimedia.org.erb index 8192ca6..d61ccd8 100644 --- a/modules/noc/templates/noc.wikimedia.org.erb +++ b/modules/noc/templates/noc.wikimedia.org.erb @@ -35,5 +35,12 @@ AddDefaultCharset utf-8 +# Force the Content-Type to text/plain +# for files without an extension, since +# AddDefaultCharset works only +# for text/plain or text/html. + +ForceType text/plain + -- To view, visit https://gerrit.wikimedia.org/r/318074 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I29820d95b0bf77af591352a1e9fbae5e3bf2731f Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Restore mc2009/mc2010 to standard settings
Elukey has submitted this change and it was merged. Change subject: Restore mc2009/mc2010 to standard settings .. Restore mc2009/mc2010 to standard settings These hosts have been used to experiment new settings and memcached version. Change-Id: I90f567514f6a6a170578877fa5b68745d8f31126 --- D hieradata/hosts/mc2009.yaml D hieradata/hosts/mc2010.yaml 2 files changed, 0 insertions(+), 15 deletions(-) Approvals: Elukey: Looks good to me, approved Giuseppe Lavagetto: Looks good to me, but someone else must approve jenkins-bot: Verified diff --git a/hieradata/hosts/mc2009.yaml b/hieradata/hosts/mc2009.yaml deleted file mode 100644 index 180e955..000 --- a/hieradata/hosts/mc2009.yaml +++ /dev/null @@ -1,10 +0,0 @@ -# The maximum number of slab classes is 64 from >= 1.4.25 -# so a different growth factor is needed to manage the same workload. -# More info: T129963 -memcached::version: '1.4.28-1.1+wmf1' -memcached::growth_factor: '1.15' -memcached::extended_options: - - 'slab_reassign' - - 'slab_automove' - - 'lru_crawler' - - 'lru_maintainer' \ No newline at end of file diff --git a/hieradata/hosts/mc2010.yaml b/hieradata/hosts/mc2010.yaml deleted file mode 100644 index 28c7b4b..000 --- a/hieradata/hosts/mc2010.yaml +++ /dev/null @@ -1,5 +0,0 @@ -memcached::extended_options: - - 'slab_reassign' - - 'lru_crawler' - - 'maxconns_fast' - - 'hash_algorithm=murmur3' \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/318051 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I90f567514f6a6a170578877fa5b68745d8f31126 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: Giuseppe Lavagetto Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Restore mc2009/mc2010 to standard settings
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/318051 Change subject: Restore mc2009/mc2010 to standard settings .. Restore mc2009/mc2010 to standard settings These hosts have been used to experiment new settings and memcached version. Change-Id: I90f567514f6a6a170578877fa5b68745d8f31126 --- D hieradata/hosts/mc2009.yaml D hieradata/hosts/mc2010.yaml 2 files changed, 0 insertions(+), 15 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/51/318051/1 diff --git a/hieradata/hosts/mc2009.yaml b/hieradata/hosts/mc2009.yaml deleted file mode 100644 index 180e955..000 --- a/hieradata/hosts/mc2009.yaml +++ /dev/null @@ -1,10 +0,0 @@ -# The maximum number of slab classes is 64 from >= 1.4.25 -# so a different growth factor is needed to manage the same workload. -# More info: T129963 -memcached::version: '1.4.28-1.1+wmf1' -memcached::growth_factor: '1.15' -memcached::extended_options: - - 'slab_reassign' - - 'slab_automove' - - 'lru_crawler' - - 'lru_maintainer' \ No newline at end of file diff --git a/hieradata/hosts/mc2010.yaml b/hieradata/hosts/mc2010.yaml deleted file mode 100644 index 28c7b4b..000 --- a/hieradata/hosts/mc2010.yaml +++ /dev/null @@ -1,5 +0,0 @@ -memcached::extended_options: - - 'slab_reassign' - - 'lru_crawler' - - 'maxconns_fast' - - 'hash_algorithm=murmur3' \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/318051 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I90f567514f6a6a170578877fa5b68745d8f31126 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: logstash: Stop dropping mod_proxy_fcgi warnings
Elukey has submitted this change and it was merged.
Change subject: logstash: Stop dropping mod_proxy_fcgi warnings
..
logstash: Stop dropping mod_proxy_fcgi warnings
Bug: T73487
Change-Id: Icdb49b9e5f4a6ffbbd2b5f4c112b1ee99cae893d
---
M files/logstash/filter-syslog.conf
1 file changed, 0 insertions(+), 11 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/files/logstash/filter-syslog.conf
b/files/logstash/filter-syslog.conf
index 429e8ad..9710285 100644
--- a/files/logstash/filter-syslog.conf
+++ b/files/logstash/filter-syslog.conf
@@ -80,17 +80,6 @@
} # end [program] == "hhvm-fatal"
if [program] == "apache2" {
- # Ignore known mod_proxy_fcgi bugs
- if [message] =~ /AH01070: Error parsing script headers/ {
-drop {}
- }
- if [message] =~ /AH01068: Got bogus version \d/ {
-drop {}
- }
- if [message] =~ /AH01075: Error dispatching request to :/ {
-drop {}
- }
-
mutate {
replace => [ "type", "apache2" ]
}
--
To view, visit https://gerrit.wikimedia.org/r/306943
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Icdb49b9e5f4a6ffbbd2b5f4c112b1ee99cae893d
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: BryanDavis
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: Giuseppe Lavagetto
Gerrit-Reviewer: Hashar
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add user pmiazga to the 'deployment' group
Elukey has submitted this change and it was merged. Change subject: Add user pmiazga to the 'deployment' group .. Add user pmiazga to the 'deployment' group Remark: approved by manager and Ops meeting. Bug: T148477 Change-Id: I4eea4f75776be5d1dfae0372625dde91f088e3fb --- M modules/admin/data/data.yaml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index ca7abc8..101e8b1 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -55,7 +55,7 @@ tgr, phuedx, ejegg, twentyafterfour, legoktm, catrope, krenair, mobrovac, nuria, thcipriani, joal, eevans, mforns, dpatrick, dcausse, bsitzmann, mholloway-shell, dduvall, gehel, dereckson, - ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko] + ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko, pmiazga] privileges: ['ALL = (www-data,apache,mwdeploy,l10nupdate) NOPASSWD: ALL', 'ALL = NOPASSWD: /sbin/restart hhvm', 'ALL = NOPASSWD: /sbin/start hhvm', -- To view, visit https://gerrit.wikimedia.org/r/317539 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I4eea4f75776be5d1dfae0372625dde91f088e3fb Gerrit-PatchSet: 2 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add user pmiazga to the 'deployment' group
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/317539 Change subject: Add user pmiazga to the 'deployment' group .. Add user pmiazga to the 'deployment' group Remark: approved by manager and Ops meeting. Bug: T148477 Change-Id: I4eea4f75776be5d1dfae0372625dde91f088e3fb --- M modules/admin/data/data.yaml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/39/317539/1 diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index ca7abc8..101e8b1 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -55,7 +55,7 @@ tgr, phuedx, ejegg, twentyafterfour, legoktm, catrope, krenair, mobrovac, nuria, thcipriani, joal, eevans, mforns, dpatrick, dcausse, bsitzmann, mholloway-shell, dduvall, gehel, dereckson, - ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko] + ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko, pmiazga] privileges: ['ALL = (www-data,apache,mwdeploy,l10nupdate) NOPASSWD: ALL', 'ALL = NOPASSWD: /sbin/restart hhvm', 'ALL = NOPASSWD: /sbin/start hhvm', -- To view, visit https://gerrit.wikimedia.org/r/317539 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I4eea4f75776be5d1dfae0372625dde91f088e3fb Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add ppchelko to the deployment group
Elukey has submitted this change and it was merged. Change subject: Add ppchelko to the deployment group .. Add ppchelko to the deployment group Please note: this change should only be merged after the next ops meeting (if accepted of course). Bug: T148475 Change-Id: I611d533030bc81c275fab906c80ea3ea4968e94b --- M modules/admin/data/data.yaml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index 0f75c6e..ca7abc8 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -55,7 +55,7 @@ tgr, phuedx, ejegg, twentyafterfour, legoktm, catrope, krenair, mobrovac, nuria, thcipriani, joal, eevans, mforns, dpatrick, dcausse, bsitzmann, mholloway-shell, dduvall, gehel, dereckson, - ladsgroup, bawolff, zfilipin, addshore, niharika29] + ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko] privileges: ['ALL = (www-data,apache,mwdeploy,l10nupdate) NOPASSWD: ALL', 'ALL = NOPASSWD: /sbin/restart hhvm', 'ALL = NOPASSWD: /sbin/start hhvm', -- To view, visit https://gerrit.wikimedia.org/r/317121 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I611d533030bc81c275fab906c80ea3ea4968e94b Gerrit-PatchSet: 2 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Alex Monk Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add user pmiazga to analytics-privatedata-users
Elukey has submitted this change and it was merged. Change subject: Add user pmiazga to analytics-privatedata-users .. Add user pmiazga to analytics-privatedata-users Bug: T148477 Change-Id: I5e15ec33eb83baa39ab496b99f70734e7ab36fe0 --- M modules/admin/data/data.yaml 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index ec20ea4..0f75c6e 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -231,7 +231,7 @@ dcausse, bd808, tjones, mobrovac, jgirault, jdrewniak, ejegg, jdcc, ori, bmansurov, atgomez, amire80, kartik, pcoombe, foks, mpany, hjiang, jsamra, bcohn, jdlrobson, chelsyx, ovasileva, mtizzoni, panisson, paolotti, ciro, - psinger, flemmerich, mkroetzsch, akrausetud, filippo, nschaaf] + psinger, flemmerich, mkroetzsch, akrausetud, filippo, nschaaf, pmiazga] analytics-admins: gid: 732 description: Admin access to analytics cluster. -- To view, visit https://gerrit.wikimedia.org/r/317468 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I5e15ec33eb83baa39ab496b99f70734e7ab36fe0 Gerrit-PatchSet: 2 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Alex Monk Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add user pmiazga to analytics-privatedata-users
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/317468 Change subject: Add user pmiazga to analytics-privatedata-users .. Add user pmiazga to analytics-privatedata-users Bug: T148477 Change-Id: I5e15ec33eb83baa39ab496b99f70734e7ab36fe0 --- M modules/admin/data/data.yaml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/68/317468/1 diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index ec20ea4..0f75c6e 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -231,7 +231,7 @@ dcausse, bd808, tjones, mobrovac, jgirault, jdrewniak, ejegg, jdcc, ori, bmansurov, atgomez, amire80, kartik, pcoombe, foks, mpany, hjiang, jsamra, bcohn, jdlrobson, chelsyx, ovasileva, mtizzoni, panisson, paolotti, ciro, - psinger, flemmerich, mkroetzsch, akrausetud, filippo, nschaaf] + psinger, flemmerich, mkroetzsch, akrausetud, filippo, nschaaf, pmiazga] analytics-admins: gid: 732 description: Admin access to analytics cluster. -- To view, visit https://gerrit.wikimedia.org/r/317468 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I5e15ec33eb83baa39ab496b99f70734e7ab36fe0 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add user pmiazga and its related ssh key
Elukey has submitted this change and it was merged. Change subject: Add user pmiazga and its related ssh key .. Add user pmiazga and its related ssh key Piotr Miazga is a new hire. He will need access to deployment and analytics groups, that will be handled on a separate code change. Bug: T148477 Change-Id: If32f799c07b20e6d41f376ab5f1a047e46cf5551 --- M modules/admin/data/data.yaml 1 file changed, 8 insertions(+), 0 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified Dzahn: Looks good to me, but someone else must approve diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index 5af6066..ec20ea4 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -2162,3 +2162,11 @@ ssh_keys: - ssh-rsa B3NzaC1yc2EDAQABAAACAQDKpQEVtQw8XJCKvG7oNL9d+dveFqRpHCzduUvdzi6MK7zgtY9xUKF7ZcQObgP5fCpgLonDUu2b61WRzYgiPoBkhhnTKqRkCx6sT6KfsHaSE2iZefW5N7qEyRqpWCoXfmGnXJdP3r4cCgxBqdAJ/aoFNcvzpLUrcgw7wfI+gJ6VV897fwkH9SEnjVUxn/AZDAxsK6xa2gDmCUjG+Biwl6LsRCXtop87jtJ6SdgKmDkmGNh8t2y6YJxE0vjYuzNH2E2bEaYOgFRS/GULbNh+U+UqrN3sE4RTHUzAOqXh3iYShpNnwgm9nj41j3Q4h1S51muPEywRz60nU16JfaAvvvA3vyrS2L1P3zB7F+MbvhVVhdprmMvQf7hXbqSNhcxBWwvYfGGA67jy6hDL8WCd8+ql5F7tGw66FxStDjs2GYCiLp0zaDeBkl5Fx9GKjTXD8kCtA7Y8JDLuW1L/7ROX58C8tO6SyHjHHQO4hpSUavieUHt+2CFLYWvGtBZFStdL+esQmOC6qjX1KdwSQxlRXN9D436VxGusVdo3gkbaVvPFVx25gefWrKQ0/a17iU3eDL02sVkNa91Xs3AfeNNTqOv111UfoZjsatva7ZhBujeD131Ky7wolm1IVYtwfF4ifEpgLwjHtSzgfmdMLJSv4IweGYebc71JILMOHZCQGJ3/iw== volke...@wikimedia.org uid: 12186 + pmiazga: +ensure: present +gid: 500 +name: pmiazga +realname: Piotr Miazga +ssh_keys: + - ssh-rsa B3NzaC1yc2EDAQABAAABAQCsS1WgYh+EeHalBKDkWmlvhzfoWPFQcg6OYWKkWhraOd6mhGjDfaF6vst9xC4PCtvudDBMImNIrzLAT3uWoZxuLUIpsGLvIfPLk/vRtfQFN/L//ItSaOshFRVmVfEvVeNbmQKrgcfLvp2rmuJvf/1qJXNAgGIZK54xw3xjriZ/uem8xBZOJ9a7jcyUYlD8fk0rNcv2vpzQIY6ZiFJq8lYf8bja70IuBTPb93gmlCnE6r6wCb6PAWByL3zZJVffFi+A+DEMSNZ6bjEu6iQA1iQCcIPxXS8DTr0Zj+J1TeyqYUvuXmzqJb+xmOqv74fAqPu8A7YxX89LCExj5AzP+0Xr raynor@DellE6540 +uid: 15700 -- To view, visit https://gerrit.wikimedia.org/r/317142 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: If32f799c07b20e6d41f376ab5f1a047e46cf5551 Gerrit-PatchSet: 3 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Alex Monk Gerrit-Reviewer: Dzahn Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove mc1019 from role memcached (new host)
Elukey has submitted this change and it was merged.
Change subject: Remove mc1019 from role memcached (new host)
..
Remove mc1019 from role memcached (new host)
This host is new and not ready to be productionized.
While we have redis and memcached IPs configured in Hiera
it may cause confusion to have a host fully equipped but not
in any pool.
Bug: T137345
Change-Id: I26164e6cd573613c471c2ab59719c4a94ae0b177
---
M manifests/site.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
Dzahn: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/manifests/site.pp b/manifests/site.pp
index 1de2690..c336334 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1933,7 +1933,7 @@
include base::firewall
}
-node /^mc(10[01][0-9])\.eqiad\.wmnet/ {
+node /^mc10(0[1-9]|1[0-8])\.eqiad\.wmnet/ {
role(memcached)
}
--
To view, visit https://gerrit.wikimedia.org/r/317171
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I26164e6cd573613c471c2ab59719c4a94ae0b177
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Dzahn
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove mc1019 from role memcached (new host)
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/317171
Change subject: Remove mc1019 from role memcached (new host)
..
Remove mc1019 from role memcached (new host)
This host is new and not ready to be productionized.
While we have redis and memcached IPs configured in Hiera
it may cause confusion to have a host fully equipped but not
in any pool.
Bug: T137345
Change-Id: I26164e6cd573613c471c2ab59719c4a94ae0b177
---
M manifests/site.pp
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/71/317171/1
diff --git a/manifests/site.pp b/manifests/site.pp
index 1de2690..598bdc3 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -1933,7 +1933,7 @@
include base::firewall
}
-node /^mc(10[01][0-9])\.eqiad\.wmnet/ {
+node /^mc(10[01][0-8])\.eqiad\.wmnet/ {
role(memcached)
}
--
To view, visit https://gerrit.wikimedia.org/r/317171
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I26164e6cd573613c471c2ab59719c4a94ae0b177
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add user pmiazga and its related ssh key
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/317142 Change subject: Add user pmiazga and its related ssh key .. Add user pmiazga and its related ssh key Piotr Miazga is a new hire. He will need access to deployment and analytics groups, that will be handled on a separate code change. Bug: T148477 Change-Id: If32f799c07b20e6d41f376ab5f1a047e46cf5551 --- M modules/admin/data/data.yaml 1 file changed, 8 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/42/317142/1 diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index 5af6066..ec20ea4 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -2162,3 +2162,11 @@ ssh_keys: - ssh-rsa B3NzaC1yc2EDAQABAAACAQDKpQEVtQw8XJCKvG7oNL9d+dveFqRpHCzduUvdzi6MK7zgtY9xUKF7ZcQObgP5fCpgLonDUu2b61WRzYgiPoBkhhnTKqRkCx6sT6KfsHaSE2iZefW5N7qEyRqpWCoXfmGnXJdP3r4cCgxBqdAJ/aoFNcvzpLUrcgw7wfI+gJ6VV897fwkH9SEnjVUxn/AZDAxsK6xa2gDmCUjG+Biwl6LsRCXtop87jtJ6SdgKmDkmGNh8t2y6YJxE0vjYuzNH2E2bEaYOgFRS/GULbNh+U+UqrN3sE4RTHUzAOqXh3iYShpNnwgm9nj41j3Q4h1S51muPEywRz60nU16JfaAvvvA3vyrS2L1P3zB7F+MbvhVVhdprmMvQf7hXbqSNhcxBWwvYfGGA67jy6hDL8WCd8+ql5F7tGw66FxStDjs2GYCiLp0zaDeBkl5Fx9GKjTXD8kCtA7Y8JDLuW1L/7ROX58C8tO6SyHjHHQO4hpSUavieUHt+2CFLYWvGtBZFStdL+esQmOC6qjX1KdwSQxlRXN9D436VxGusVdo3gkbaVvPFVx25gefWrKQ0/a17iU3eDL02sVkNa91Xs3AfeNNTqOv111UfoZjsatva7ZhBujeD131Ky7wolm1IVYtwfF4ifEpgLwjHtSzgfmdMLJSv4IweGYebc71JILMOHZCQGJ3/iw== volke...@wikimedia.org uid: 12186 + pmiazga: +ensure: present +gid: 500 +name: pmiazga +realname: Piotr Miazga +ssh_keys: + - ssh-rsa B3NzaC1yc2EDAQABAAABAQCsS1WgYh+EeHalBKDkWmlvhzfoWPFQcg6OYWKkWhraOd6mhGjDfaF6vst9xC4PCtvudDBMImNIrzLAT3uWoZxuLUIpsGLvIfPLk/vRtfQFN/L//ItSaOshFRVmVfEvVeNbmQKrgcfLvp2rmuJvf/1qJXNAgGIZK54xw3xjriZ/uem8xBZOJ9a7jcyUYlD8fk0rNcv2vpzQIY6ZiFJq8lYf8bja70IuBTPb93gmlCnE6r6wCb6PAWByL3zZJVffFi+A+DEMSNZ6bjEu6iQA1iQCcIPxXS8DTr0Zj+J1TeyqYUvuXmzqJb+xmOqv74fAqPu8A7YxX89LCExj5AzP+0Xr raynor@DellE6540 +uid: 15700 -- To view, visit https://gerrit.wikimedia.org/r/317142 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: If32f799c07b20e6d41f376ab5f1a047e46cf5551 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add ppchelko to the deployment group
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/317121 Change subject: Add ppchelko to the deployment group .. Add ppchelko to the deployment group Please note: this change should only be merged after the next ops meeting (if accepted of course). Bug: T148475 Change-Id: I611d533030bc81c275fab906c80ea3ea4968e94b --- M modules/admin/data/data.yaml 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/21/317121/1 diff --git a/modules/admin/data/data.yaml b/modules/admin/data/data.yaml index 5af6066..016ad93 100644 --- a/modules/admin/data/data.yaml +++ b/modules/admin/data/data.yaml @@ -55,7 +55,7 @@ tgr, phuedx, ejegg, twentyafterfour, legoktm, catrope, krenair, mobrovac, nuria, thcipriani, joal, eevans, mforns, dpatrick, dcausse, bsitzmann, mholloway-shell, dduvall, gehel, dereckson, - ladsgroup, bawolff, zfilipin, addshore, niharika29] + ladsgroup, bawolff, zfilipin, addshore, niharika29, ppchelko] privileges: ['ALL = (www-data,apache,mwdeploy,l10nupdate) NOPASSWD: ALL', 'ALL = NOPASSWD: /sbin/restart hhvm', 'ALL = NOPASSWD: /sbin/start hhvm', -- To view, visit https://gerrit.wikimedia.org/r/317121 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I611d533030bc81c275fab906c80ea3ea4968e94b Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix yarn.w.o's mod_proxy_html configuration
Elukey has submitted this change and it was merged. Change subject: Fix yarn.w.o's mod_proxy_html configuration .. Fix yarn.w.o's mod_proxy_html configuration The visualization of Spark shell logs in the UI was broken due to a problem with mod_proxy_html. Ref: https://httpd.apache.org/docs/2.4/mod/mod_proxy_html.html#comment_3329 Bug: T147927 Change-Id: I68219d91ad1131dbe41131e3666c6b6dec3e718b --- M modules/statistics/templates/yarn.wikimedia.org.erb 1 file changed, 4 insertions(+), 4 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/statistics/templates/yarn.wikimedia.org.erb b/modules/statistics/templates/yarn.wikimedia.org.erb index 0f14d7a..3aca7ac 100644 --- a/modules/statistics/templates/yarn.wikimedia.org.erb +++ b/modules/statistics/templates/yarn.wikimedia.org.erb @@ -30,10 +30,10 @@ # The Jetty Webserver (responsible for the Yarn UI) returns HTML with hardcoded # links in the HTML hrefs, so we need a way to avoid broken links. -ProxyHTMLEnable On -# ProxyHTMLExtended forces mod_proxy_html to parse CSS/JS files too -# causing performance penalties in the overall rendering time. -# Unfortunately this is needed since some analytics1001 links are +# Not using 'ProxyHTMLEnable On' since it breaks the visualization of +# Spark shell logs in the UI. +# https://httpd.apache.org/docs/2.4/mod/mod_proxy_html.html#comment_3329 +SetOutputFilter INFLATE;proxy-html;DEFLATE # contained in embedded JS script that cause a broken UI experience # if left untouched. ProxyHTMLExtended On -- To view, visit https://gerrit.wikimedia.org/r/316969 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: I68219d91ad1131dbe41131e3666c6b6dec3e718b Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey Gerrit-Reviewer: Elukey Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix yarn.w.o's mod_proxy_html configuration
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/316969 Change subject: Fix yarn.w.o's mod_proxy_html configuration .. Fix yarn.w.o's mod_proxy_html configuration The visualization of Spark shell logs in the UI was broken due to a problem with mod_proxy_html. Ref: https://httpd.apache.org/docs/2.4/mod/mod_proxy_html.html#comment_3329 Bug: T147927 Change-Id: I68219d91ad1131dbe41131e3666c6b6dec3e718b --- M modules/statistics/templates/yarn.wikimedia.org.erb 1 file changed, 4 insertions(+), 4 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/69/316969/1 diff --git a/modules/statistics/templates/yarn.wikimedia.org.erb b/modules/statistics/templates/yarn.wikimedia.org.erb index 0f14d7a..3aca7ac 100644 --- a/modules/statistics/templates/yarn.wikimedia.org.erb +++ b/modules/statistics/templates/yarn.wikimedia.org.erb @@ -30,10 +30,10 @@ # The Jetty Webserver (responsible for the Yarn UI) returns HTML with hardcoded # links in the HTML hrefs, so we need a way to avoid broken links. -ProxyHTMLEnable On -# ProxyHTMLExtended forces mod_proxy_html to parse CSS/JS files too -# causing performance penalties in the overall rendering time. -# Unfortunately this is needed since some analytics1001 links are +# Not using 'ProxyHTMLEnable On' since it breaks the visualization of +# Spark shell logs in the UI. +# https://httpd.apache.org/docs/2.4/mod/mod_proxy_html.html#comment_3329 +SetOutputFilter INFLATE;proxy-html;DEFLATE # contained in embedded JS script that cause a broken UI experience # if left untouched. ProxyHTMLExtended On -- To view, visit https://gerrit.wikimedia.org/r/316969 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I68219d91ad1131dbe41131e3666c6b6dec3e718b Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Tune varnishkafka-webrequest parameters
Elukey has submitted this change and it was merged.
Change subject: Tune varnishkafka-webrequest parameters
..
Tune varnishkafka-webrequest parameters
The Analytics team discovered a lot of webrequests missing the end
datatime field ending up in data consistency errors.
Varnishlog has been used on various cp hosts with the following
configuration to spot anomalies:
sudo varnishlog -c -n frontend -L 5000 -T 1500
-q 'VSL or (Timestamp:Start and not Timestamp:Resp)' | tee timeouts.txt
The VSL timeouts settings (-L and -T) are the same used by Varnishkafka.
This query asks for any request that is either logged with a VSL timeout
or with a Start timestamp but not a Resp one. Two things came up:
1) A lot of requests with the HttpGarbage tag are discarded by Varnish but
logged. Example:
* << Request >> 173757699
- Begin req 173757698 rxreq
- Timestamp Start: 1476449182.479356 0.00 0.00
- Timestamp Req: 1476449182.479356 0.00 0.00
- BogoHeader Header has ctrl char 0x1f
- HttpGarbage"GET%00"
- ReqAcct643 0 643 28 0 28
- End
2) The VSL store overflow error is still present but happens less frequently.
The proposed solution for 1) is to avoid logging any request with
the HttpGarbage tag, and to raise the maximum number of incomplete requests
kept in memory to 1.
Bug: T148412
Change-Id: I68ada5789a848a676989c08590819625740b6bd8
---
M modules/role/manifests/cache/kafka/webrequest.pp
1 file changed, 11 insertions(+), 6 deletions(-)
Approvals:
Elukey: Looks good to me, approved
Ottomata: Looks good to me, but someone else must approve
Ema: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 65fcdd7..f541861 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -15,10 +15,11 @@
{
# Set varnish.arg.q or varnish.arg.m according to Varnish version
if (hiera('varnish_version4', false)) {
-# Background from T136314:
+# Background task: T136314
+# Background info about the parameters used:
# 'q':
-# Filter out PURGE requests and Pipe creation traffic.
-# A Varnish log containing Timestamp:Pipe does not carry
Timestamp:Resp,
+# 1) Filter out PURGE requests and Pipe creation traffic.
+# 2) A Varnish log containing Timestamp:Pipe does not carry
Timestamp:Resp,
# used by Analytics to bucket data on Hadoop and for data consistency
# checks. These requests indicate that Varnish tried to establish a
pipe
# channel between the client and the backend, an information that
@@ -30,13 +31,15 @@
# At the moment these requests get logged incorrectly and with partial
# data (due to the VSL timeout) so it makes sense to filter them out to
# remove noise from Analytics data.
+# 3) A request marked with the VSL tag 'HttpGarbage' indicates
unparseable
+# HTTP requests, generating spurious Varnish logs.
# 'T':
# VLS API timeout is the maximum time that Varnishkafka will wait
between
# "Begin" and "End" timestamps before flushing the available tags to a
log.
# When a timeout occurs most of the times the result is a webrequest
log
# missing values like the end timestamp.
#
-# Parameters modified during the upload migration:
+# VSL Timeout parameters modified during the upload migration:
# 'L':
# Sets the upper limit of incomplete transactions kept before the
oldest
# one is force completed. This setting keeps an upper bound
@@ -44,14 +47,16 @@
# A change in the -T timeout value has the side effect of keeping more
# incomplete transactions in memory for each varnishkafka query (in
our case
# it directly corresponds to a varnishkafka instance running).
+# The threshold has been raised to '5000' the first time (which removed
+# the bulk of the timeouts) and to '1' the second time.
# 'T':
# Raised the maximum timeout for incomplete records from '700' to
'1500'
# after setting the -L to '5000'. VSL timeouts were masked
# by VSL store overflow errors.
$varnish_opts = {
-'q' => 'ReqMethod ne "PURGE" and not Timestamp:Pipe and not
ReqHeader:Upgrade ~ "[wW]ebsocket"',
+'q' => 'ReqMethod ne "PURGE" and not Timestamp:Pipe and not
ReqHeader:Upgrade ~ "[wW]ebsocket" and not HttpGarbage',
'T' => '1500',
-'L' => '5000'
+'L' => '1'
}
$conf_template = 'varnishkafka/varnishkafka_v4.conf.erb'
} else {
--
To view, visit https://gerrit.wikimedia.org/r/316306
To unsubscribe, visit
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Tune varnishkafka-webrequest parameters
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/316306
Change subject: Tune varnishkafka-webrequest parameters
..
Tune varnishkafka-webrequest parameters
The Analytics team discovered a lot of webrequests missing the end
datatime field ending up in data consistency errors.
Varnishlog has been used on various cp hosts with the following
configuration to spot anomalies:
sudo varnishlog -c -n frontend -L 5000 -T 1500
-q 'VSL or (Timestamp:Start and not Timestamp:Resp)' | tee timeouts.txt
The VSL timeouts settings (-L and -T) are the same used by Varnishkafka.
This query asks for any request that is either logged with a VSL timeout
or with a Start timestamp but not a Resp one. Two things came up:
1) A lot of requests with the HttpGarbage tag are discarded by Varnish but
logged.
2) The VSL store overflow error is still present but happens less frequently.
The proposed solution for 1) is to avoid logging any request with
the HttpGarbage tag, and to raise the maximum number of incomplete requests
kept in memory to 1.
Change-Id: I68ada5789a848a676989c08590819625740b6bd8
---
M modules/role/manifests/cache/kafka/webrequest.pp
1 file changed, 11 insertions(+), 6 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/06/316306/1
diff --git a/modules/role/manifests/cache/kafka/webrequest.pp
b/modules/role/manifests/cache/kafka/webrequest.pp
index 65fcdd7..f541861 100644
--- a/modules/role/manifests/cache/kafka/webrequest.pp
+++ b/modules/role/manifests/cache/kafka/webrequest.pp
@@ -15,10 +15,11 @@
{
# Set varnish.arg.q or varnish.arg.m according to Varnish version
if (hiera('varnish_version4', false)) {
-# Background from T136314:
+# Background task: T136314
+# Background info about the parameters used:
# 'q':
-# Filter out PURGE requests and Pipe creation traffic.
-# A Varnish log containing Timestamp:Pipe does not carry
Timestamp:Resp,
+# 1) Filter out PURGE requests and Pipe creation traffic.
+# 2) A Varnish log containing Timestamp:Pipe does not carry
Timestamp:Resp,
# used by Analytics to bucket data on Hadoop and for data consistency
# checks. These requests indicate that Varnish tried to establish a
pipe
# channel between the client and the backend, an information that
@@ -30,13 +31,15 @@
# At the moment these requests get logged incorrectly and with partial
# data (due to the VSL timeout) so it makes sense to filter them out to
# remove noise from Analytics data.
+# 3) A request marked with the VSL tag 'HttpGarbage' indicates
unparseable
+# HTTP requests, generating spurious Varnish logs.
# 'T':
# VLS API timeout is the maximum time that Varnishkafka will wait
between
# "Begin" and "End" timestamps before flushing the available tags to a
log.
# When a timeout occurs most of the times the result is a webrequest
log
# missing values like the end timestamp.
#
-# Parameters modified during the upload migration:
+# VSL Timeout parameters modified during the upload migration:
# 'L':
# Sets the upper limit of incomplete transactions kept before the
oldest
# one is force completed. This setting keeps an upper bound
@@ -44,14 +47,16 @@
# A change in the -T timeout value has the side effect of keeping more
# incomplete transactions in memory for each varnishkafka query (in
our case
# it directly corresponds to a varnishkafka instance running).
+# The threshold has been raised to '5000' the first time (which removed
+# the bulk of the timeouts) and to '1' the second time.
# 'T':
# Raised the maximum timeout for incomplete records from '700' to
'1500'
# after setting the -L to '5000'. VSL timeouts were masked
# by VSL store overflow errors.
$varnish_opts = {
-'q' => 'ReqMethod ne "PURGE" and not Timestamp:Pipe and not
ReqHeader:Upgrade ~ "[wW]ebsocket"',
+'q' => 'ReqMethod ne "PURGE" and not Timestamp:Pipe and not
ReqHeader:Upgrade ~ "[wW]ebsocket" and not HttpGarbage',
'T' => '1500',
-'L' => '5000'
+'L' => '1'
}
$conf_template = 'varnishkafka/varnishkafka_v4.conf.erb'
} else {
--
To view, visit https://gerrit.wikimedia.org/r/316306
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I68ada5789a848a676989c08590819625740b6bd8
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/medi
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Allow the wmde LDAP group to access pivot.w.o
Elukey has uploaded a new change for review. https://gerrit.wikimedia.org/r/316217 Change subject: Allow the wmde LDAP group to access pivot.w.o .. Allow the wmde LDAP group to access pivot.w.o Change-Id: Ic9672cab4e516c394fa675612bfb2e691b8316f9 --- M modules/statistics/templates/pivot.wikimedia.org.erb 1 file changed, 1 insertion(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/17/316217/1 diff --git a/modules/statistics/templates/pivot.wikimedia.org.erb b/modules/statistics/templates/pivot.wikimedia.org.erb index c1df67f..5399cda 100644 --- a/modules/statistics/templates/pivot.wikimedia.org.erb +++ b/modules/statistics/templates/pivot.wikimedia.org.erb @@ -27,6 +27,7 @@ AuthLDAPURL "ldaps://ldap-labs.eqiad.wikimedia.org ldap-labs.codfw.wikimedia.org/ou=people,dc=wikimedia,dc=org?cn" Require ldap-group cn=wmf,ou=groups,dc=wikimedia,dc=org Require ldap-group cn=nda,ou=groups,dc=wikimedia,dc=org +Require ldap-group cn=wmde,ou=groups,dc=wikimedia,dc=org CustomLog /var/log/apache2/pivot.wikimedia.org-access.log wmf -- To view, visit https://gerrit.wikimedia.org/r/316217 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Ic9672cab4e516c394fa675612bfb2e691b8316f9 Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Elukey ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add dataCubes description to pivot config template
Elukey has submitted this change and it was merged. Change subject: Add dataCubes description to pivot config template .. Add dataCubes description to pivot config template Pivot provides a description for datacubes, we want to use that. 3 dataCubes described, pageview_hourly, pageview_daily, edit-history-test. Change-Id: Ia68dedf8524d498ce2bf8a3e68b419fb7684d4be --- M modules/pivot/templates/config.yaml.erb 1 file changed, 3 insertions(+), 0 deletions(-) Approvals: Elukey: Looks good to me, approved jenkins-bot: Verified diff --git a/modules/pivot/templates/config.yaml.erb b/modules/pivot/templates/config.yaml.erb index 6459038..1c5fcc5 100644 --- a/modules/pivot/templates/config.yaml.erb +++ b/modules/pivot/templates/config.yaml.erb @@ -39,6 +39,7 @@ dataCubes: - name: pageviews-daily title: Pageviews Daily +description: Small dataset - Starting July 2015 - Daily resolution - Monthly loading clusterName: druid source: pageviews-daily # The refresh rule describes how often the data cube looks for new data. Default: 'query'/PT1M (every minute) @@ -70,6 +71,7 @@ - name: edit-history-test title: Edit History test +description: WIP - Don't use for now (probably broken anyway) clusterName: druid source: edit-history-test # The refresh rule describes how often the data cube looks for new data. @@ -102,6 +104,7 @@ - name: pageviews-hourly title: Pageviews Hourly +description: Large dataset - Last 3 months - Hourly resolution - Daily loading clusterName: druid source: pageviews-hourly # The refresh rule describes how often the data cube looks for new data. -- To view, visit https://gerrit.wikimedia.org/r/315695 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ia68dedf8524d498ce2bf8a3e68b419fb7684d4be Gerrit-PatchSet: 3 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: Joal Gerrit-Reviewer: Elukey Gerrit-Reviewer: Joal Gerrit-Reviewer: Mforns Gerrit-Reviewer: Milimetric Gerrit-Reviewer: jenkins-bot <> ___ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Refactor memcached role to allow a more flexible hieradata c...
Elukey has submitted this change and it was merged.
Change subject: Refactor memcached role to allow a more flexible hieradata
config
..
Refactor memcached role to allow a more flexible hieradata config
There are currently multiple versions of the memcached package
installed across eqiad and codfw, mostly due to performance experiments.
The long term view is to have a single version installed everywhere,
but in the meantime we need a bit more flexibility.
Bug: T129963
Change-Id: I34be02040a888058d64c3de3f12ee61606daf396
---
M hieradata/hosts/mc1009.yaml
A hieradata/hosts/mc1010.yaml
A hieradata/hosts/mc2009.yaml
A hieradata/hosts/mc2010.yaml
M manifests/role/memcached.pp
5 files changed, 42 insertions(+), 39 deletions(-)
Approvals:
Ori.livneh: Looks good to me, but someone else must approve
Elukey: Looks good to me, approved
Giuseppe Lavagetto: Looks good to me, but someone else must approve
jenkins-bot: Verified
diff --git a/hieradata/hosts/mc1009.yaml b/hieradata/hosts/mc1009.yaml
index 5fc1999..1ae36de 100644
--- a/hieradata/hosts/mc1009.yaml
+++ b/hieradata/hosts/mc1009.yaml
@@ -1,3 +1,13 @@
debdeploy::grains:
debdeploy-memcached:
value: canary
+# The maximum number of slab classes is 64 from >= 1.4.25
+# so a different growth factor is needed to manage the same workload.
+# More info: T129963
+memcached::version: '1.4.25-2~wmf1'
+memcached::growth_factor: '1.15'
+memcached::extended_options:
+ - 'slab_reassign'
+ - 'slab_automove'
+ - 'lru_crawler'
+ - 'lru_maintainer'
\ No newline at end of file
diff --git a/hieradata/hosts/mc1010.yaml b/hieradata/hosts/mc1010.yaml
new file mode 100644
index 000..3dfd910
--- /dev/null
+++ b/hieradata/hosts/mc1010.yaml
@@ -0,0 +1,10 @@
+# The maximum number of slab classes is 64 from >= 1.4.25
+# so a different growth factor is needed to manage the same workload.
+# More info: T129963
+memcached::version: '1.4.25-2~wmf1'
+memcached::growth_factor: '1.15'
+memcached::extended_options:
+ - 'slab_reassign'
+ - 'slab_automove'
+ - 'lru_crawler'
+ - 'lru_maintainer'
\ No newline at end of file
diff --git a/hieradata/hosts/mc2009.yaml b/hieradata/hosts/mc2009.yaml
new file mode 100644
index 000..180e955
--- /dev/null
+++ b/hieradata/hosts/mc2009.yaml
@@ -0,0 +1,10 @@
+# The maximum number of slab classes is 64 from >= 1.4.25
+# so a different growth factor is needed to manage the same workload.
+# More info: T129963
+memcached::version: '1.4.28-1.1+wmf1'
+memcached::growth_factor: '1.15'
+memcached::extended_options:
+ - 'slab_reassign'
+ - 'slab_automove'
+ - 'lru_crawler'
+ - 'lru_maintainer'
\ No newline at end of file
diff --git a/hieradata/hosts/mc2010.yaml b/hieradata/hosts/mc2010.yaml
new file mode 100644
index 000..28c7b4b
--- /dev/null
+++ b/hieradata/hosts/mc2010.yaml
@@ -0,0 +1,5 @@
+memcached::extended_options:
+ - 'slab_reassign'
+ - 'lru_crawler'
+ - 'maxconns_fast'
+ - 'hash_algorithm=murmur3'
\ No newline at end of file
diff --git a/manifests/role/memcached.pp b/manifests/role/memcached.pp
index adbb6ce..a53a609 100644
--- a/manifests/role/memcached.pp
+++ b/manifests/role/memcached.pp
@@ -12,48 +12,16 @@
'labs' => 3000,
}
-# The following hosts will get different package versions
-# deployed manually. This workaround should avoid
-# disabling puppet for the whole duration of the test.
+# There are different package versions available due to a performance test,
+# most of them are deployed/installed manually.
# More info: T129963
-if $::hostname == 'mc2009' {
-$version = '1.4.28-1.1+wmf1'
-} elsif $::hostname =~ /mc10(09|10)/ {
-$version = '1.4.25-2~wmf1'
-} else {
-$version = os_version('debian >= jessie || ubuntu >= trusty') ? {
-true=> 'present',
-default => '1.4.15-0wmf1',
-}
+$version = os_version('debian >= jessie || ubuntu >= trusty') ? {
+true=> hiera('memcached::version', 'present'),
+default => '1.4.15-0wmf1',
}
-# The following hosts are configured with newer memcached versions
-# (1.4.25-2~wmf1 and 1.4.28-1.1+wmf1) as part of a performance experiment.
-# The maximum number of slab classes is 64 from >= 1.4.25
-# so a different growth factor is needed to manage the same workload.
-# More info: T129963
-if $::hostname == 'mc2009' or $::hostname =~ /mc10(09|10)/ {
-$growth_factor= 1.15
-$extended_options = [
-'slab_reassign',
-'slab_automove',
-'lru_crawler',
-'lru_maintainer',
-]
-} elsif $::hostname == 'mc2010' {
-$growth_factor= 1.05
-$extended_options = [
-'slab_reassign',
-'lru_crawler',
-'maxconns_fast',
-'hash_algorithm=murmur3',
-]
-} else {
-$growth_factor= 1.05
-$extend
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add reference to the hw reclaim task for aqs100[123]
Elukey has submitted this change and it was merged.
Change subject: Add reference to the hw reclaim task for aqs100[123]
..
Add reference to the hw reclaim task for aqs100[123]
Change-Id: If3b2336eba8828ea4cf6471b18c08c4162094f57
---
M manifests/site.pp
1 file changed, 1 insertion(+), 1 deletion(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/manifests/site.pp b/manifests/site.pp
index 79059e6..3674bec 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -136,7 +136,7 @@
include base::firewall
}
-# Old Analytics Query Service (now decommissioned)
+# Old Analytics Query Service (hw reclaim task: T147926)
node /aqs100[123]\.eqiad\.wmnet/ {
role(spare::system)
}
--
To view, visit https://gerrit.wikimedia.org/r/315497
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: If3b2336eba8828ea4cf6471b18c08c4162094f57
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add reference to the hw reclaim task for aqs100[123]
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/315497
Change subject: Add reference to the hw reclaim task for aqs100[123]
..
Add reference to the hw reclaim task for aqs100[123]
Change-Id: If3b2336eba8828ea4cf6471b18c08c4162094f57
---
M manifests/site.pp
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/97/315497/1
diff --git a/manifests/site.pp b/manifests/site.pp
index 79059e6..3674bec 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -136,7 +136,7 @@
include base::firewall
}
-# Old Analytics Query Service (now decommissioned)
+# Old Analytics Query Service (hw reclaim task: T147926)
node /aqs100[123]\.eqiad\.wmnet/ {
role(spare::system)
}
--
To view, visit https://gerrit.wikimedia.org/r/315497
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If3b2336eba8828ea4cf6471b18c08c4162094f57
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix remaining issues with the Pivot UI
Elukey has submitted this change and it was merged.
Change subject: Fix remaining issues with the Pivot UI
..
Fix remaining issues with the Pivot UI
A missing key in the config file prevents Pivot to properly bootstrap,
the /etc/pivot directory needs to be ensured.
Bug: T138262
Change-Id: I7120b3530331d6be7e3d28237a242a9a7b0de1b2
---
M modules/pivot/manifests/init.pp
M modules/pivot/templates/config.yaml.erb
2 files changed, 9 insertions(+), 0 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/pivot/manifests/init.pp b/modules/pivot/manifests/init.pp
index 42ef4d7..70b2b2b 100644
--- a/modules/pivot/manifests/init.pp
+++ b/modules/pivot/manifests/init.pp
@@ -92,12 +92,20 @@
source => 'puppet:///modules/pivot/pivot.profile.firejail',
}
+file { '/etc/pivot':
+ensure => directory,
+owner => root,
+group => root,
+mode => '0755',
+}
+
file { '/etc/pivot/config.yaml':
ensure => present,
owner => 'root',
group => 'root',
mode=> '0444',
content => template('pivot/config.yaml.erb'),
+require => File['/etc/pivot'],
}
systemd::syslog { 'pivot':
diff --git a/modules/pivot/templates/config.yaml.erb
b/modules/pivot/templates/config.yaml.erb
index 99d52b0..6459038 100644
--- a/modules/pivot/templates/config.yaml.erb
+++ b/modules/pivot/templates/config.yaml.erb
@@ -106,6 +106,7 @@
source: pageviews-hourly
# The refresh rule describes how often the data cube looks for new data.
# Default: 'query'/PT1M (every minute)
+refreshRule:
rule: query
#defaultTimezone: Etc/UTC
--
To view, visit https://gerrit.wikimedia.org/r/315486
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I7120b3530331d6be7e3d28237a242a9a7b0de1b2
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix remaining issues with the Pivot UI
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/315486
Change subject: Fix remaining issues with the Pivot UI
..
Fix remaining issues with the Pivot UI
A missing key in the config file prevents Pivot to properly bootstrap,
the /etc/pivot directory needs to be ensured.
Bug: T138262
Change-Id: I7120b3530331d6be7e3d28237a242a9a7b0de1b2
---
M modules/pivot/manifests/init.pp
M modules/pivot/templates/config.yaml.erb
2 files changed, 9 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/86/315486/1
diff --git a/modules/pivot/manifests/init.pp b/modules/pivot/manifests/init.pp
index 42ef4d7..70b2b2b 100644
--- a/modules/pivot/manifests/init.pp
+++ b/modules/pivot/manifests/init.pp
@@ -92,12 +92,20 @@
source => 'puppet:///modules/pivot/pivot.profile.firejail',
}
+file { '/etc/pivot':
+ensure => directory,
+owner => root,
+group => root,
+mode => '0755',
+}
+
file { '/etc/pivot/config.yaml':
ensure => present,
owner => 'root',
group => 'root',
mode=> '0444',
content => template('pivot/config.yaml.erb'),
+require => File['/etc/pivot'],
}
systemd::syslog { 'pivot':
diff --git a/modules/pivot/templates/config.yaml.erb
b/modules/pivot/templates/config.yaml.erb
index 99d52b0..6459038 100644
--- a/modules/pivot/templates/config.yaml.erb
+++ b/modules/pivot/templates/config.yaml.erb
@@ -106,6 +106,7 @@
source: pageviews-hourly
# The refresh rule describes how often the data cube looks for new data.
# Default: 'query'/PT1M (every minute)
+refreshRule:
rule: query
#defaultTimezone: Etc/UTC
--
To view, visit https://gerrit.wikimedia.org/r/315486
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I7120b3530331d6be7e3d28237a242a9a7b0de1b2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix template render bug for the Pivot UI
Elukey has submitted this change and it was merged.
Change subject: Fix template render bug for the Pivot UI
..
Fix template render bug for the Pivot UI
Bug: T138262
Change-Id: Idf7ddc9fad91dd0daff24a63bad596c7e15b9091
---
M modules/pivot/manifests/init.pp
1 file changed, 5 insertions(+), 5 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/pivot/manifests/init.pp b/modules/pivot/manifests/init.pp
index 011e35e..42ef4d7 100644
--- a/modules/pivot/manifests/init.pp
+++ b/modules/pivot/manifests/init.pp
@@ -93,11 +93,11 @@
}
file { '/etc/pivot/config.yaml':
-ensure => present,
-owner => 'root',
-group => 'root',
-mode => '0444',
-source => 'puppet:///modules/pivot/templates/config.yaml.erb',
+ensure => present,
+owner => 'root',
+group => 'root',
+mode=> '0444',
+content => template('pivot/config.yaml.erb'),
}
systemd::syslog { 'pivot':
--
To view, visit https://gerrit.wikimedia.org/r/315481
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Idf7ddc9fad91dd0daff24a63bad596c7e15b9091
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
Gerrit-Reviewer: Elukey
Gerrit-Reviewer: jenkins-bot <>
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix template render bug for the Pivot UI
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/315481
Change subject: Fix template render bug for the Pivot UI
..
Fix template render bug for the Pivot UI
Bug: T138262
Change-Id: Idf7ddc9fad91dd0daff24a63bad596c7e15b9091
---
M modules/pivot/manifests/init.pp
1 file changed, 5 insertions(+), 5 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/81/315481/1
diff --git a/modules/pivot/manifests/init.pp b/modules/pivot/manifests/init.pp
index 011e35e..42ef4d7 100644
--- a/modules/pivot/manifests/init.pp
+++ b/modules/pivot/manifests/init.pp
@@ -93,11 +93,11 @@
}
file { '/etc/pivot/config.yaml':
-ensure => present,
-owner => 'root',
-group => 'root',
-mode => '0444',
-source => 'puppet:///modules/pivot/templates/config.yaml.erb',
+ensure => present,
+owner => 'root',
+group => 'root',
+mode=> '0444',
+content => template('pivot/config.yaml.erb'),
}
systemd::syslog { 'pivot':
--
To view, visit https://gerrit.wikimedia.org/r/315481
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idf7ddc9fad91dd0daff24a63bad596c7e15b9091
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Elukey
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add config file for the Pivot UI
Elukey has submitted this change and it was merged.
Change subject: Add config file for the Pivot UI
..
Add config file for the Pivot UI
Bug: T138262
Change-Id: If98e77173b9c15fcb40ac433cf338020da84a7a1
---
M modules/pivot/manifests/init.pp
A modules/pivot/templates/config.yaml.erb
M modules/pivot/templates/initscripts/pivot.systemd.erb
3 files changed, 182 insertions(+), 18 deletions(-)
Approvals:
Elukey: Looks good to me, approved
jenkins-bot: Verified
diff --git a/modules/pivot/manifests/init.pp b/modules/pivot/manifests/init.pp
index d175968..011e35e 100644
--- a/modules/pivot/manifests/init.pp
+++ b/modules/pivot/manifests/init.pp
@@ -17,24 +17,48 @@
#
# === Parameters
#
-# $port The port used by Pivot to accept HTTP connections.
-#Default: 9090
-# $druid_broker The fully qualified domain name (like
druid1001.eqiad.wmnet)
-#of the Druid Broker that the Pivot UI will contact.
-#Default: undef
-# $deployment_user Scap deployment user.
-#Default: 'analytics_deploy'
-# $scap_repo Scap repository.
-#Default: 'analytics/pivot/deploy'
-# $contact_group Contact group for alerts.
-#Default: 'admins'
-
+# [*port*]
+# The port used by Pivot to accept HTTP connections.
+# Default: 9090
+#
+# [*druid_broker*]
+# The fully qualified domain name (like druid1001.eqiad.wmnet)
+# of the Druid Broker that the Pivot UI will contact.
+# Default: undef
+#
+# [*query_timeout*]
+# The timeout to set on the Druid queries in ms.
+# Default: 4
+#
+# [*source_refresh_ms*]
+# How often should Druid sources be reloaded in ms.
+# Default: 15000
+#
+# [*schema_refresh_ms*]
+# How often should Druid source schema be reloaded in ms.
+# Default: 12
+#
+# [*deployment_user*]
+# Scap deployment user.
+# Default: 'analytics_deploy'
+#
+# [*scap_repo*]
+# Scap repository.
+# Default: 'analytics/pivot/deploy'
+#
+# [*contact_group*]
+# Contact group for alerts.
+# Default: 'admins'
+#
class pivot(
-$port= 9090,
-$druid_broker= undef,
-$deployment_user = 'analytics_deploy',
-$scap_repo = 'analytics/pivot/deploy',
-$contact_group = 'admins',
+$port = 9090,
+$druid_broker = undef,
+$query_timeout = 4,
+$source_refresh_ms = 15000,
+$schema_refresh_ms = 12,
+$deployment_user = 'analytics_deploy',
+$scap_repo = 'analytics/pivot/deploy',
+$contact_group = 'admins',
) {
requires_os('debian >= jessie')
@@ -68,6 +92,14 @@
source => 'puppet:///modules/pivot/pivot.profile.firejail',
}
+file { '/etc/pivot/config.yaml':
+ensure => present,
+owner => 'root',
+group => 'root',
+mode => '0444',
+source => 'puppet:///modules/pivot/templates/config.yaml.erb',
+}
+
systemd::syslog { 'pivot':
readable_by => 'all',
base_dir=> '/var/log',
@@ -80,6 +112,7 @@
require => [
Scap::Target['analytics/pivot/deploy'],
File['/etc/firejail/pivot.profile'],
+File['/etc/pivot/config.yaml'],
User['pivot'],
Systemd::Syslog['pivot'],
],
diff --git a/modules/pivot/templates/config.yaml.erb
b/modules/pivot/templates/config.yaml.erb
new file mode 100644
index 000..99d52b0
--- /dev/null
+++ b/modules/pivot/templates/config.yaml.erb
@@ -0,0 +1,131 @@
+# generated by Pivot version 0.9.39
+# for a more detailed walk-through go to:
https://github.com/implydata/pivot/blob/v0.9.39/docs/configuration.md
+
+# The port on which the Pivot server will listen on
+port: <%= @port %>
+
+clusters:
+ - name: druid
+
+# The database type of the cluster
+type: druid
+
+# The host (hostname:port) of the cluster. In the Druid case this must be
the broker.
+host: <%= @druid_broker %>
+
+# The timeout to set on the queries in ms.
+timeout: <%= @query_timeout %>
+
+# Should the sources of this cluster be automatically scanned and new
sources added as data cubes.
+sourceListScan: auto
+
+# Should the list of sources be reloaded every time that Pivot is loaded.
+#sourceListRefreshOnLoad: false
+
+# How often should sources be reloaded in ms.
+sourceListRefreshInterval: <%= @source_refresh_ms %>
+
+# Should sources be scanned for additional dimensions every time that
Pivot is loaded.
+#sourceReintrospectOnLoad: false
+
+# How often should source schema be reloaded in ms.
+sourceReintrospectInterval: <%= @schema_refresh_ms %>
+
+# Database specific (druid) ===
+
+# The introspection strategy for the Druid external.
+#introspectionStrategy: segment-metadata-fallback
+
+dataCubes:
+ - name: pageviews-daily
+title: Pageviews Daily
+cl
[MediaWiki-commits] [Gerrit] operations/puppet[production]: Add config file for the Pivot UI
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/315480
Change subject: Add config file for the Pivot UI
..
Add config file for the Pivot UI
Bug: T138262
Change-Id: If98e77173b9c15fcb40ac433cf338020da84a7a1
---
M modules/pivot/manifests/init.pp
A modules/pivot/templates/config.yaml.erb
M modules/pivot/templates/initscripts/pivot.systemd.erb
3 files changed, 182 insertions(+), 18 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/80/315480/1
diff --git a/modules/pivot/manifests/init.pp b/modules/pivot/manifests/init.pp
index d175968..011e35e 100644
--- a/modules/pivot/manifests/init.pp
+++ b/modules/pivot/manifests/init.pp
@@ -17,24 +17,48 @@
#
# === Parameters
#
-# $port The port used by Pivot to accept HTTP connections.
-#Default: 9090
-# $druid_broker The fully qualified domain name (like
druid1001.eqiad.wmnet)
-#of the Druid Broker that the Pivot UI will contact.
-#Default: undef
-# $deployment_user Scap deployment user.
-#Default: 'analytics_deploy'
-# $scap_repo Scap repository.
-#Default: 'analytics/pivot/deploy'
-# $contact_group Contact group for alerts.
-#Default: 'admins'
-
+# [*port*]
+# The port used by Pivot to accept HTTP connections.
+# Default: 9090
+#
+# [*druid_broker*]
+# The fully qualified domain name (like druid1001.eqiad.wmnet)
+# of the Druid Broker that the Pivot UI will contact.
+# Default: undef
+#
+# [*query_timeout*]
+# The timeout to set on the Druid queries in ms.
+# Default: 4
+#
+# [*source_refresh_ms*]
+# How often should Druid sources be reloaded in ms.
+# Default: 15000
+#
+# [*schema_refresh_ms*]
+# How often should Druid source schema be reloaded in ms.
+# Default: 12
+#
+# [*deployment_user*]
+# Scap deployment user.
+# Default: 'analytics_deploy'
+#
+# [*scap_repo*]
+# Scap repository.
+# Default: 'analytics/pivot/deploy'
+#
+# [*contact_group*]
+# Contact group for alerts.
+# Default: 'admins'
+#
class pivot(
-$port= 9090,
-$druid_broker= undef,
-$deployment_user = 'analytics_deploy',
-$scap_repo = 'analytics/pivot/deploy',
-$contact_group = 'admins',
+$port = 9090,
+$druid_broker = undef,
+$query_timeout = 4,
+$source_refresh_ms = 15000,
+$schema_refresh_ms = 12,
+$deployment_user = 'analytics_deploy',
+$scap_repo = 'analytics/pivot/deploy',
+$contact_group = 'admins',
) {
requires_os('debian >= jessie')
@@ -68,6 +92,14 @@
source => 'puppet:///modules/pivot/pivot.profile.firejail',
}
+file { '/etc/pivot/config.yaml':
+ensure => present,
+owner => 'root',
+group => 'root',
+mode => '0444',
+source => 'puppet:///modules/pivot/templates/config.yaml.erb',
+}
+
systemd::syslog { 'pivot':
readable_by => 'all',
base_dir=> '/var/log',
@@ -80,6 +112,7 @@
require => [
Scap::Target['analytics/pivot/deploy'],
File['/etc/firejail/pivot.profile'],
+File['/etc/pivot/config.yaml'],
User['pivot'],
Systemd::Syslog['pivot'],
],
diff --git a/modules/pivot/templates/config.yaml.erb
b/modules/pivot/templates/config.yaml.erb
new file mode 100644
index 000..99d52b0
--- /dev/null
+++ b/modules/pivot/templates/config.yaml.erb
@@ -0,0 +1,131 @@
+# generated by Pivot version 0.9.39
+# for a more detailed walk-through go to:
https://github.com/implydata/pivot/blob/v0.9.39/docs/configuration.md
+
+# The port on which the Pivot server will listen on
+port: <%= @port %>
+
+clusters:
+ - name: druid
+
+# The database type of the cluster
+type: druid
+
+# The host (hostname:port) of the cluster. In the Druid case this must be
the broker.
+host: <%= @druid_broker %>
+
+# The timeout to set on the queries in ms.
+timeout: <%= @query_timeout %>
+
+# Should the sources of this cluster be automatically scanned and new
sources added as data cubes.
+sourceListScan: auto
+
+# Should the list of sources be reloaded every time that Pivot is loaded.
+#sourceListRefreshOnLoad: false
+
+# How often should sources be reloaded in ms.
+sourceListRefreshInterval: <%= @source_refresh_ms %>
+
+# Should sources be scanned for additional dimensions every time that
Pivot is loaded.
+#sourceReintrospectOnLoad: false
+
+# How often should source schema be reloaded in ms.
+sourceReintrospectInterval: <%= @schema_refresh_ms %>
+
+# Database specific (druid) ===
+
+# The introspection strategy for the Druid external.
+#introspectionStrategy: segment-metadata-fallback
+
+dataCubes:
+ - name: pag
[MediaWiki-commits] [Gerrit] analytics/aqs[master]: Map null values to zeros in output in per-article endpoint
Elukey has uploaded a new change for review.
https://gerrit.wikimedia.org/r/315284
Change subject: Map null values to zeros in output in per-article endpoint
..
Map null values to zeros in output in per-article endpoint
View counts for an article that are equal to 0 in any of
the dimensions are being stored as 'null' in cassandra, aqs code
should do the translation from one to the other.
Bug: T144521
Change-Id: Idf1c517605714c3cf1cfad044d3e8763cc0f2ece
---
M sys/pageviews.js
M test/aqs_test_module.yaml
M test/features/pageviews/pageviews.js
3 files changed, 15 insertions(+), 1 deletion(-)
git pull ssh://gerrit.wikimedia.org:29418/analytics/aqs
refs/changes/84/315284/1
diff --git a/sys/pageviews.js b/sys/pageviews.js
index a203e41..9c37a4a 100644
--- a/sys/pageviews.js
+++ b/sys/pageviews.js
@@ -165,6 +165,10 @@
item.access = rp.access;
item.agent = rp.agent;
item.views = item[viewKey(rp.access, rp.agent)];
+// map null to zero for view counts, we store null in
cassandra for efficiency
+if (item.views === null) {
+item.views = 0;
+}
removeDenormalizedColumns(item);
});
}
diff --git a/test/aqs_test_module.yaml b/test/aqs_test_module.yaml
index fa93af9..f5f9af7 100644
--- a/test/aqs_test_module.yaml
+++ b/test/aqs_test_module.yaml
@@ -30,7 +30,7 @@
da: '{{request.params.views}}5'
db: '{{request.params.views}}6'
ds: '{{request.params.views}}7'
- du: '{{request.params.views}}8'
+ du: null
maa: '{{request.params.views}}9'
mab: '{{request.params.views}}10'
mas: '{{request.params.views}}11'
diff --git a/test/features/pageviews/pageviews.js
b/test/features/pageviews/pageviews.js
index ea5f20b..72a1f7f 100644
--- a/test/features/pageviews/pageviews.js
+++ b/test/features/pageviews/pageviews.js
@@ -15,6 +15,7 @@
// NOTE: this tests using the projects/aqs_default.yaml config, so
// it doesn't know about the /metrics root like the prod config does
var articleEndpoint =
'/pageviews/per-article/en.wikipedia/desktop/spider/one/daily/20150701/20150703';
+var articleEndpointMobile =
'/pageviews/per-article/en.wikipedia/desktop/user/one/daily/20150701/20150703';
var projectEndpoint =
'/pageviews/aggregate/en.wikipedia/all-access/all-agents/hourly/1969010100/1971010100';
var topsEndpoint =
'/pageviews/top/en.wikipedia/mobile-web/2015/01/all-days';
var projectEndpointStrip =
'/pageviews/aggregate/www.en.wikipedia.org/all-access/all-agents/hourly/1969010100/1971010100';
@@ -71,6 +72,15 @@
});
});
+it('should return integer zero if view count is null', function () {
+return preq.get({
+uri: server.config.aqsURL + articleEndpointMobile
+
+}).then(function (res) {
+assert.deepEqual(res.body.items.length, 1);
+assert.deepEqual(res.body.items[0].views, 0);
+});
+});
function r(s, replaceSpaces) {
var weirdArticleTitle = 'dash - space : colon % percent / slash';
--
To view, visit https://gerrit.wikimedia.org/r/315284
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idf1c517605714c3cf1cfad044d3e8763cc0f2ece
Gerrit-PatchSet: 1
Gerrit-Project: analytics/aqs
Gerrit-Branch: master
Gerrit-Owner: Elukey
Gerrit-Reviewer: Nuria
___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
