from:"Ema \(Code Review\)"

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Point pinkunicorn's varnish-fe to its own varnish-be

2016-10-14 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/315920

Change subject: Point pinkunicorn's varnish-fe to its own varnish-be
..

Point pinkunicorn's varnish-fe to its own varnish-be

pinkunicorn is currently using our standard varnish config in the sense
that traffic entering its frontend chashes to our standard pool of
varnish backends. It makes sense to send traffic to its own backend
instead to test a full varnish 4 cache_text setup.

Bug: T131503
Change-Id: Ied9f0e1c43f19328291025ba9ded0bb165cd68fb
---
M hieradata/hosts/cp1008.yaml
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/20/315920/1

diff --git a/hieradata/hosts/cp1008.yaml b/hieradata/hosts/cp1008.yaml
index d93343e..e9d076e 100644
--- a/hieradata/hosts/cp1008.yaml
+++ b/hieradata/hosts/cp1008.yaml
@@ -2,3 +2,6 @@
   debdeploy-cp:
 value: canary
 varnish_version4: true
+cache::text::nodes:
+eqiad:
+  - 'cp1008.wikimedia.org'

-- 
To view, visit https://gerrit.wikimedia.org/r/315920
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ied9f0e1c43f19328291025ba9ded0bb165cd68fb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Point pinkunicorn's varnish-fe to its own varnish-be

2016-10-14 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: Point pinkunicorn's varnish-fe to its own varnish-be
..


Point pinkunicorn's varnish-fe to its own varnish-be

pinkunicorn is currently using our standard varnish config in the sense
that traffic entering its frontend chashes to our standard pool of
varnish backends. It makes sense to send traffic to its own backend
instead to test a full varnish 4 cache_text setup.

Bug: T131503
Change-Id: Ied9f0e1c43f19328291025ba9ded0bb165cd68fb
---
M hieradata/hosts/cp1008.yaml
1 file changed, 5 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/hieradata/hosts/cp1008.yaml b/hieradata/hosts/cp1008.yaml
index d93343e..097c046 100644
--- a/hieradata/hosts/cp1008.yaml
+++ b/hieradata/hosts/cp1008.yaml
@@ -2,3 +2,8 @@
   debdeploy-cp:
 value: canary
 varnish_version4: true
+cache::text::nodes:
+eqiad:
+  - 'cp1008.wikimedia.org'
+codfw:
+  - 'cp1008.wikimedia.org'

-- 
To view, visit https://gerrit.wikimedia.org/r/315920
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ied9f0e1c43f19328291025ba9ded0bb165cd68fb
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: role::cache::instances add pinkunicorn to production conditi...

2016-10-14 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: role::cache::instances add pinkunicorn to production conditional
..


role::cache::instances add pinkunicorn to production conditional

Change-Id: I8bc9214ea164416d1f259096dc88bfa8cd655cc4
---
M modules/role/manifests/cache/instances.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/role/manifests/cache/instances.pp 
b/modules/role/manifests/cache/instances.pp
index 0f653fd..e9c996f 100644
--- a/modules/role/manifests/cache/instances.pp
+++ b/modules/role/manifests/cache/instances.pp
@@ -70,7 +70,7 @@
 # the production conditional is sad (vs using hiera), but I
 # don't know of a better way to factor this out at the moment,
 # and it may all change later...
-if $::realm != 'production' {
+if $::realm != 'production' or $::hostname == 'cp1008' {
 $becaches_filtered = hash_deselect_re('^cache_codfw', $backend_caches)
 } else {
 $becaches_filtered = $backend_caches

-- 
To view, visit https://gerrit.wikimedia.org/r/315924
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I8bc9214ea164416d1f259096dc88bfa8cd655cc4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cp1008: set varnish::dynamic_directors to false

2016-10-14 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/315944

Change subject: cp1008: set varnish::dynamic_directors to false
..

cp1008: set varnish::dynamic_directors to false

We do not want pinkunicorn to use dynamic directors, it should send
requests to its own varnish backend instead for testing purposes.

Bug: T131503
Change-Id: I122912bff274c3f555270804fcf412680692affd
---
M hieradata/hosts/cp1008.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/44/315944/1

diff --git a/hieradata/hosts/cp1008.yaml b/hieradata/hosts/cp1008.yaml
index 097c046..afee624 100644
--- a/hieradata/hosts/cp1008.yaml
+++ b/hieradata/hosts/cp1008.yaml
@@ -2,6 +2,7 @@
   debdeploy-cp:
 value: canary
 varnish_version4: true
+varnish::dynamic_directors: false
 cache::text::nodes:
 eqiad:
   - 'cp1008.wikimedia.org'

-- 
To view, visit https://gerrit.wikimedia.org/r/315944
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I122912bff274c3f555270804fcf412680692affd
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cp1008: set varnish::dynamic_directors to false

2016-10-14 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cp1008: set varnish::dynamic_directors to false
..


cp1008: set varnish::dynamic_directors to false

We do not want pinkunicorn to use dynamic directors, it should send
requests to its own varnish backend instead for testing purposes.

Bug: T131503
Change-Id: I122912bff274c3f555270804fcf412680692affd
---
M hieradata/hosts/cp1008.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/cp1008.yaml b/hieradata/hosts/cp1008.yaml
index 097c046..afee624 100644
--- a/hieradata/hosts/cp1008.yaml
+++ b/hieradata/hosts/cp1008.yaml
@@ -2,6 +2,7 @@
   debdeploy-cp:
 value: canary
 varnish_version4: true
+varnish::dynamic_directors: false
 cache::text::nodes:
 eqiad:
   - 'cp1008.wikimedia.org'

-- 
To view, visit https://gerrit.wikimedia.org/r/315944
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I122912bff274c3f555270804fcf412680692affd
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Port varnishrls to Varnish 4 VSL API

2016-10-18 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: Port varnishrls to Varnish 4 VSL API
..


Port varnishrls to Varnish 4 VSL API

Bug: T131353
Change-Id: Iaaa469fcc46c690216da97d28d1715548af5aaf8
---
A modules/varnish/files/varnishrls4
M modules/varnish/manifests/logging/rls.pp
2 files changed, 90 insertions(+), 1 deletion(-)

Approvals:
  Elukey: Looks good to me, but someone else must approve
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/varnishrls4 
b/modules/varnish/files/varnishrls4
new file mode 100755
index 000..dd03f29
--- /dev/null
+++ b/modules/varnish/files/varnishrls4
@@ -0,0 +1,82 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+  varnishrls
+  ~~
+
+  Accumulate browser cache hit ratio and total request volume statistics
+  for ResourceLoader requests (/w/load.php) and report to StatsD.
+
+  Usage: varnishrls [--statsd-server SERVER] [--key-prefix PREFIX]
+
+--statsd-server SERVER  statsd server (default: none; echo to stdout)
+--key-prefix PREFIX metric key prefix (default: varnish.clients)
+
+  Copyright 2015 Ori Livneh 
+  Copyright 2015 Gilles Dubuc 
+  Copyright 2016 Emanuele Rocca 
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+"""
+from __future__ import division
+
+import re
+import varnishlog
+import varnishprocessor
+
+class ResourceLoaderVarnishLogProcessor(varnishprocessor.VarnishLogProcessor):
+description = 'ResourceLoader Browser Cache Hit Ratio StatsD Reporter'
+key_prefix = 'ResourceLoader'
+
+def process_transaction(self, transaction):
+"""Process a single completed transaction."""
+status_code = transaction['RespStatus']
+metric_keys = ['reqs.all', 'resps.' + status_code]
+
+if 'BerespHeader' in transaction:
+metric_keys.append('reqs.if_none_match')
+
+cache_control_header = transaction.get('BereqHeader')
+cache_control = 'no'
+if cache_control_header:
+match = re.search(r'(?<=max-age=)\d+', cache_control_header)
+if match:
+cache_control = 'short' if match.group() == '300' else 'long'
+metric_keys.append('responses.%s_cache_control.%s' %
+   (cache_control, status_code))
+
+for key in metric_keys:
+self.stats[key] = self.stats.get(key, 0) + 1
+
+if self.stats['reqs.all'] > 1:
+self.flush_stats()
+
+def start(self):
+varnishlog.varnishlog((
+('n', 'frontend'), # Consider the frontend Varnish instance
+('c', None),   # Only consider interactions with the client
+('i', 'RespStatus'),   # Get RespStatus for the HTTP status code
+('i', 'BerespHeader'), # Get BerespHeader for If-None-Match header
+('i', 'BereqHeader'),  # Get BereqHeader for Cache-control header
+('i', 'ReqURL'),   # Get ReqURL to match /w/load.php
+('i', 'Timestamp'),# Get Timestamp to delimit requests
+('C', ''), # Use case-insensitive matching
+('I', '^(/w/load\.php'  # ...to match ResourceLoader ReqURLs
+  '|if-none-match:' # ...or If-None-Match BerespHeaders
+  '|cache-control:' # ...or Cache-control BereqHeaders
+  '|([\d.]+ ?){6}$' # ...or Timestamp lines
+  '|[1-5]\d{2}$)'), # ...or RxStatus codes
+), self.handle_log_record)
+
+lp = ResourceLoaderVarnishLogProcessor()
diff --git a/modules/varnish/manifests/logging/rls.pp 
b/modules/varnish/manifests/logging/rls.pp
index 6e70173..cf28b8f 100644
--- a/modules/varnish/manifests/logging/rls.pp
+++ b/modules/varnish/manifests/logging/rls.pp
@@ -18,8 +18,15 @@
 define varnish::logging::rls( $statsd_server = 'statsd' ) {
 include varnish::common
 
+if (hiera('varnish_version4', false)) {
+# Use v4 version of varnishrls
+$varnish4_python_suffix = '4'
+} else {
+$varnish4_python_suffix = ''
+}
+
 file { '/usr/local/bin/varnishrls':
-source  => 'puppet:///modules/varnish/varnishrls',
+source  => 
"puppet:///modules/varnish/varnishrls${varnish4_python_suffix}",
 owner   => 'root',
 group   => 'root',
 mode=> '0555',

-- 
To view, visit

[MediaWiki-commits] [Gerrit] operations...nginx[wmf-1.11.4]: nginx (1.11.4-1+wmf4) jessie-wikimedia; urgency=medium

2016-10-25 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/317790

Change subject: nginx (1.11.4-1+wmf4) jessie-wikimedia; urgency=medium
..

nginx (1.11.4-1+wmf4) jessie-wikimedia; urgency=medium

  * Create nginx-{full,light,extras}-dbg by hand. Autogenerated -dbgsym
packages are supported by debhelper 9.20151219 or greater, while Jessie
ships 9.20150101.

 -- Emanuele Rocca   Tue, 25 Oct 2016 11:10:36 +0200

Change-Id: I01266b7bccf9856add6780e2c71f9504a6977d4c
---
M debian/changelog
M debian/control
M debian/rules
3 files changed, 53 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/software/nginx 
refs/changes/90/317790/1

diff --git a/debian/changelog b/debian/changelog
index 60d282f..9e45952 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
+nginx (1.11.4-1+wmf4) jessie-wikimedia; urgency=medium
+
+  * Create nginx-{full,light,extras}-dbg by hand. Autogenerated -dbgsym
+packages are supported by debhelper 9.20151219 or greater, while Jessie
+ships 9.20150101.
+
+ -- Emanuele Rocca   Tue, 25 Oct 2016 11:10:36 +0200
+
 nginx (1.11.4-1+wmf3) jessie-wikimedia; urgency=medium
 
   * control: depend on libssl11-dev (WMF packaging)
diff --git a/debian/control b/debian/control
index ac2d3f6..265a1df 100644
--- a/debian/control
+++ b/debian/control
@@ -107,6 +107,20 @@
  THIRD PARTY MODULES: Auth PAM, DAV Ext, Echo, HTTP Substitutions, Upstream
  Fair Queue.
 
+Package: nginx-full-dbg
+Architecture: any
+Section: debug
+Priority: extra
+Depends: nginx-full (= ${binary:Version}), ${misc:Depends}
+Conflicts: nginx-extras-dbg, nginx-light-dbg
+Description: nginx web/proxy server (standard version) - debugging symbols
+ Nginx ("engine X") is a high-performance web and reverse proxy server
+ created by Igor Sysoev. It can be used both as a standalone web server
+ and as a proxy to reduce the load on back-end HTTP or mail servers.
+ .
+ This package provides debugging symbols for nginx-full, to assist in debugging
+ issues that you may find. It should not be required for normal operation.
+
 Package: nginx-light
 Architecture: any
 Priority: extra
@@ -134,6 +148,21 @@
  Pool, WebDAV, Upstream.
  .
  THIRD PARTY MODULES: Echo.
+
+Package: nginx-light-dbg
+Architecture: any
+Section: debug
+Priority: extra
+Depends: nginx-light (= ${binary:Version}), ${misc:Depends}
+Conflicts: nginx-extras-dbg, nginx-full-dbg
+Description: nginx web/proxy server (basic version) - debugging symbols
+ Nginx ("engine X") is a high-performance web and reverse proxy server
+ created by Igor Sysoev. It can be used both as a standalone web server
+ and as a proxy to reduce the load on back-end HTTP or mail servers.
+ .
+ This package provides debugging symbols for nginx-light, to assist in
+ debugging issues that you may find. It should not be required for normal
+ operation.
 
 Package: nginx-extras
 Architecture: any
@@ -185,6 +214,21 @@
  Headers More, Embedded Lua, HTTP Substitutions, Nchan, Upload Progress,
  Upstream Fair Queue.
 
+Package: nginx-extras-dbg
+Architecture: any
+Section: debug
+Priority: extra
+Depends: nginx-extras (= ${binary:Version}), ${misc:Depends}
+Conflicts: nginx-full-dbg, nginx-light-dbg
+Description: nginx web/proxy server (extended version) - debugging symbols
+ Nginx ("engine X") is a high-performance web and reverse proxy server
+ created by Igor Sysoev. It can be used both as a standalone web server
+ and as a proxy to reduce the load on back-end HTTP or mail servers.
+ .
+ This package provides debugging symbols for nginx-extras, to assist in
+ debugging issues that you may find. It should not be required for normal
+ operation.
+
 Package: libnginx-mod-http-geoip
 Architecture: any
 Depends: ${misc:Depends}, ${shlibs:Depends}
diff --git a/debian/rules b/debian/rules
index 7547660..7ed11f0 100755
--- a/debian/rules
+++ b/debian/rules
@@ -161,7 +161,7 @@
$(MAKE) -C $(BUILDDIR_$*) build
 
 strip.arch.%:
-   dh_strip --package=nginx-$(*) -O--dbgsym-migration='nginx-$(*)-dbg (<< 
1.10.1-3~)'
+   dh_strip --dbg-package=nginx-$(*)-dbg
 
 strip.mods.%:
dh_strip --package=libnginx-mod-$(*) -O--automatic-dbgsym

-- 
To view, visit https://gerrit.wikimedia.org/r/317790
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I01266b7bccf9856add6780e2c71f9504a6977d4c
Gerrit-PatchSet: 1
Gerrit-Project: operations/software/nginx
Gerrit-Branch: wmf-1.11.4
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...varnish4[debian-wmf]: 4.1.3-1wm4: gethdr_extrachance

2016-11-14 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/321384

Change subject: 4.1.3-1wm4: gethdr_extrachance
..

4.1.3-1wm4: gethdr_extrachance

Change-Id: Id47a7fc006a81bf0435819fefd94c8a0c825d2d9
---
M debian/changelog
A debian/patches/0005-gethdr_extrachance.patch
D debian/patches/0005-remove_bad_extrachance_code.patch
M debian/patches/series
4 files changed, 54 insertions(+), 47 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/varnish4 
refs/changes/84/321384/1

diff --git a/debian/changelog b/debian/changelog
index 297cfc9..319edd8 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+varnish (4.1.3-1wm4) jessie-wikimedia; urgency=medium
+
+  * Make extrachance upper bound configurable.
+
+ -- Emanuele Rocca   Fri, 11 Nov 2016 12:01:18 +0100
+
 varnish (4.1.3-1wm3) jessie-wikimedia; urgency=medium
 
   * Add 0005-remove_bad_extrachance_code.patch: the extrachance code is
diff --git a/debian/patches/0005-gethdr_extrachance.patch 
b/debian/patches/0005-gethdr_extrachance.patch
new file mode 100644
index 000..f8144ad
--- /dev/null
+++ b/debian/patches/0005-gethdr_extrachance.patch
@@ -0,0 +1,47 @@
+diff --git a/bin/varnishd/cache/cache_backend.c 
b/bin/varnishd/cache/cache_backend.c
+index 039d3b2..4324894 100644
+--- a/bin/varnishd/cache/cache_backend.c
 b/bin/varnishd/cache/cache_backend.c
+@@ -177,7 +177,7 @@ static int __match_proto__(vdi_gethdrs_f)
+ vbe_dir_gethdrs(const struct director *d, struct worker *wrk,
+ struct busyobj *bo)
+ {
+-  int i, extrachance = 1;
++  int i, extrachance = cache_param->gethdr_extrachance;
+   struct backend *bp;
+   struct vbc *vbc;
+ 
+@@ -232,7 +232,7 @@ vbe_dir_gethdrs(const struct director *d, struct worker 
*wrk,
+   bo->req->req_body_status != REQ_BODY_CACHED)
+   break;
+   VSC_C_main->backend_retry++;
+-  } while (extrachance);
++  } while (extrachance-- > 0);
+   return (-1);
+ }
+ 
+diff --git a/include/tbl/params.h b/include/tbl/params.h
+index 5c3f46b..ec0a8cf 100644
+--- a/include/tbl/params.h
 b/include/tbl/params.h
+@@ -458,6 +458,20 @@ PARAM(
+ )
+ 
+ PARAM(
++  /* name */  gethdr_extrachance,
++  /* typ */   uint,
++  /* min */   "0",
++  /* max */   "100",
++  /* default */   "1",
++  /* units */ NULL,
++  /* flags */ EXPERIMENTAL,
++  /* s-text */
++  "Number of extra chances in vbe_dir_gethdrs.",
++  /* l-text */"",
++  /* func */  NULL
++)
++
++PARAM(
+   /* name */  gzip_buffer,
+   /* typ */   bytes_u,
+   /* min */   "2k",
diff --git a/debian/patches/0005-remove_bad_extrachance_code.patch 
b/debian/patches/0005-remove_bad_extrachance_code.patch
deleted file mode 100644
index d159f8a..000
--- a/debian/patches/0005-remove_bad_extrachance_code.patch
+++ /dev/null
@@ -1,46 +0,0 @@
-From: Brandon Black 
-Date: Tue Nov  8 11:59:43 UTC 2016
-Subject: remove_bad_extrachance_code
-
-Bug: T150247
-diff --git a/bin/varnishd/cache/cache_backend.c 
b/bin/varnishd/cache/cache_backend.c
-index 039d3b2..f88bf8e 100644
 a/bin/varnishd/cache/cache_backend.c
-+++ b/bin/varnishd/cache/cache_backend.c
-@@ -177,7 +177,7 @@ static int __match_proto__(vdi_gethdrs_f)
- vbe_dir_gethdrs(const struct director *d, struct worker *wrk,
- struct busyobj *bo)
- {
--  int i, extrachance = 1;
-+  int i; /*, extrachance = 1;*/
-   struct backend *bp;
-   struct vbc *vbc;
- 
-@@ -201,8 +201,10 @@ vbe_dir_gethdrs(const struct director *d, struct worker 
*wrk,
-   return (-1);
-   }
-   AN(bo->htc);
-+  /*
-   if (vbc->state != VBC_STATE_STOLEN)
-   extrachance = 0;
-+  */
- 
-   i = V1F_SendReq(wrk, bo, >acct.bereq_hdrbytes, 0);
- 
-@@ -225,6 +227,7 @@ vbe_dir_gethdrs(const struct director *d, struct worker 
*wrk,
-*/
-   vbe_dir_finish(d, wrk, bo);
-   AZ(bo->htc);
-+  /*
-   if (i < 0)
-   break;
-   if (bo->req != NULL &&
-@@ -232,7 +235,8 @@ vbe_dir_gethdrs(const struct director *d, struct worker 
*wrk,
-   bo->req->req_body_status != REQ_BODY_CACHED)
-   break;
-   VSC_C_main->backend_retry++;
--  } while (extrachance);
-+  } while (extrachance);*/
-+  } while (0);
-   return (-1);
- }
diff --git a/debian/patches/series b/debian/patches/series
index 273c363..44fc4a0 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1,4 +1,4 @@
 0001-varnishd-persistent-addrarg.patch
 0002-varnishd-persistent-fallocate.patch
 0004-varnishd-persistent-mmap-exit.patch
-0005-remove_bad_extrachance_code.patch

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove Varnishkafka APT pinning

2016-11-22 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: Remove Varnishkafka APT pinning
..


Remove Varnishkafka APT pinning

Bug: T150660
Change-Id: If49fd22de7037ee3d2fbd2b50161a6b2a59ff421
---
M modules/role/manifests/cache/kafka.pp
1 file changed, 0 insertions(+), 11 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/role/manifests/cache/kafka.pp 
b/modules/role/manifests/cache/kafka.pp
index 2e67ad4..4a7891b 100644
--- a/modules/role/manifests/cache/kafka.pp
+++ b/modules/role/manifests/cache/kafka.pp
@@ -7,17 +7,6 @@
 # NOTE: This is used by inheriting classes role::cache::kafka::*
 $kafka_brokers = $kafka_config['brokers']['array']
 
-apt::pin { 'varnishkafka':
-ensure   => 'absent',
-pin  => '',
-priority => '',
-}
-
-# Make the Varnishkafka class depend on APT pinning. We want to ensure
-# varnishkafka is not apt-get installed before the pinning file is
-# created/removed.
-Apt::Pin['varnishkafka'] -> Class['Varnishkafka']
-
 # Make sure varnishes are configured and started for the first time
 # before the instances as well, or they fail to start initially...
 Service <| tag == 'varnish_instance' |> -> Varnishkafka::Instance <| |>

-- 
To view, visit https://gerrit.wikimedia.org/r/322911
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: If49fd22de7037ee3d2fbd2b50161a6b2a59ff421
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: rename scripts depending on varnishlog.py

2016-11-25 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnish: rename scripts depending on varnishlog.py
..


varnish: rename scripts depending on varnishlog.py

We do not need to have two different versions of the python varnishlog
scripts now that the upgrade to Varnish 4 is finished.

Bug: T150660
Change-Id: I5b1024c268a9ddf21d5596bc5338d54e9286ea97
---
M modules/varnish/files/varnishmedia
D modules/varnish/files/varnishmedia4
M modules/varnish/files/varnishprocessor/varnishprocessor.py
D modules/varnish/files/varnishprocessor4/__init__.py
D modules/varnish/files/varnishprocessor4/varnishprocessor.py
M modules/varnish/files/varnishreqstats
D modules/varnish/files/varnishreqstats4
M modules/varnish/files/varnishrls
D modules/varnish/files/varnishrls4
M modules/varnish/files/varnishstatsd
D modules/varnish/files/varnishstatsd4
M modules/varnish/files/varnishxcache
D modules/varnish/files/varnishxcache4
M modules/varnish/files/varnishxcps
D modules/varnish/files/varnishxcps4
M modules/varnish/manifests/common.pp
M modules/varnish/manifests/logging/media.pp
M modules/varnish/manifests/logging/reqstats.pp
M modules/varnish/manifests/logging/rls.pp
M modules/varnish/manifests/logging/statsd.pp
M modules/varnish/manifests/logging/xcache.pp
M modules/varnish/manifests/logging/xcps.pp
22 files changed, 80 insertions(+), 1,014 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/varnishmedia 
b/modules/varnish/files/varnishmedia
index 2466134..27a0535 100755
--- a/modules/varnish/files/varnishmedia
+++ b/modules/varnish/files/varnishmedia
@@ -42,7 +42,7 @@
 
 def process_transaction(self, transaction):
 """Process a single completed transaction."""
-status_code = transaction.get('TxStatus')
+status_code = transaction.get('RespStatus')
 if status_code is None:
 return
 
@@ -56,11 +56,11 @@
 
 def start(self):
 varnishlog.varnishlog((
-('m', 'RxURL:/thumb/'), # Only look at thumb requests
-('n', 'frontend'),  # Consider the frontend Varnish instance
-('i', 'TxStatus'),  # Get TxStatus for the HTTP status code
-('i', 'RxURL'), # Get RxURL to delimit requests
-('i', 'ReqEnd'),# Get ReqEnd to delimit requests
+('q', 'ReqURL ~ "/thumb/"'), # Only look at thumb requests
+('n', 'frontend'),   # Only frontend Varnish instance
+('i', 'RespStatus'), # RespStatus for the HTTP status code
+('i', 'ReqURL'), # ReqURL to delimit requests
+('i', 'Timestamp'),  # Timestamp to delimit requests
 ), self.handle_log_record)
 
 lp = MediaVarnishLogProcessor()
diff --git a/modules/varnish/files/varnishmedia4 
b/modules/varnish/files/varnishmedia4
deleted file mode 100755
index 27a0535..000
--- a/modules/varnish/files/varnishmedia4
+++ /dev/null
@@ -1,66 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-"""
-  varnishmedia
-  
-
-  Accumulate browser cache hit ratio and total request volume statistics
-  for media requests and report to StatsD.
-
-  Usage: varnishmedia [--statsd-server SERVER] [--key-prefix PREFIX]
-
---statsd-server SERVER  statsd server (default: none; echo to stdout)
---key-prefix PREFIX metric key prefix (default: 
media.thumbnail.varnish)
-
-  Copyright 2015 Ori Livneh 
-  Copyright 2015 Gilles Dubuc 
-
-  Licensed under the Apache License, Version 2.0 (the "License");
-  you may not use this file except in compliance with the License.
-  You may obtain a copy of the License at
-
-  http://www.apache.org/licenses/LICENSE-2.0
-
-  Unless required by applicable law or agreed to in writing, software
-  distributed under the License is distributed on an "AS IS" BASIS,
-  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-  See the License for the specific language governing permissions and
-  limitations under the License.
-
-"""
-from __future__ import division
-
-import re
-
-import varnishlog
-import varnishprocessor
-
-
-class MediaVarnishLogProcessor(varnishprocessor.VarnishLogProcessor):
-description = 'Media Browser Cache Hit Ratio StatsD Reporter'
-key_prefix = 'media.thumbnail.varnish'
-
-def process_transaction(self, transaction):
-"""Process a single completed transaction."""
-status_code = transaction.get('RespStatus')
-if status_code is None:
-return
-
-metric_keys = ['reqs.all', 'resps.' + status_code]
-
-for key in metric_keys:
-self.stats[key] = self.stats.get(key, 0) + 1
-
-if self.stats['reqs.all'] > 1:
-self.flush_stats()
-
-def start(self):
-varnishlog.varnishlog((
-('q', 'ReqURL ~ "/thumb/"'), # Only

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishapi.py: add VSM_Close C binding

2016-11-25 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnishapi.py: add VSM_Close C binding
..


varnishapi.py: add VSM_Close C binding

Bug: T151561
Change-Id: Ib6a2a91b81e3ab179b3db1c36eedbc9b3c00961a
Ref: https://github.com/xcir/python-varnishapi/pull/67
---
M modules/varnish/files/varnishapi.py
1 file changed, 4 insertions(+), 0 deletions(-)

Approvals:
  Elukey: Looks good to me, but someone else must approve
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/varnishapi.py 
b/modules/varnish/files/varnishapi.py
index 3dac16e..33deaf2 100644
--- a/modules/varnish/files/varnishapi.py
+++ b/modules/varnish/files/varnishapi.py
@@ -289,7 +289,11 @@
 #VSM_Seq;
 #VSM_Head;
 #VSM_Find_Chunk;
+
 #VSM_Close;
+self.VSM_Close = lib.VSM_Close
+self.VSM_Close.argtypes = [c_void_p]
+
 #VSM_iter0;
 #VSM_intern;
 #

-- 
To view, visit https://gerrit.wikimedia.org/r/323530
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib6a2a91b81e3ab179b3db1c36eedbc9b3c00961a
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishapi.py: add VSM_Close C binding

2016-11-25 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/323530

Change subject: varnishapi.py: add VSM_Close C binding
..

varnishapi.py: add VSM_Close C binding

Bug: T151561
Change-Id: Ib6a2a91b81e3ab179b3db1c36eedbc9b3c00961a
Ref: https://github.com/xcir/python-varnishapi/pull/67
---
M modules/varnish/files/varnishapi.py
1 file changed, 4 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/30/323530/1

diff --git a/modules/varnish/files/varnishapi.py 
b/modules/varnish/files/varnishapi.py
index 3dac16e..33deaf2 100644
--- a/modules/varnish/files/varnishapi.py
+++ b/modules/varnish/files/varnishapi.py
@@ -289,7 +289,11 @@
 #VSM_Seq;
 #VSM_Head;
 #VSM_Find_Chunk;
+
 #VSM_Close;
+self.VSM_Close = lib.VSM_Close
+self.VSM_Close.argtypes = [c_void_p]
+
 #VSM_iter0;
 #VSM_intern;
 #

-- 
To view, visit https://gerrit.wikimedia.org/r/323530
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib6a2a91b81e3ab179b3db1c36eedbc9b3c00961a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Remove varnish::apt_preferences

2016-11-21 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/322703

Change subject: Remove varnish::apt_preferences
..

Remove varnish::apt_preferences

There is no need for APT pinning of Varnish packages now that all hosts
are running v4.

Bug: T150660
Change-Id: I29a80155e5ea8b6b9fb8204486ee862fca63fdc4
---
D modules/varnish/manifests/apt_preferences.pp
M modules/varnish/manifests/packages.pp
2 files changed, 1 insertion(+), 12 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/03/322703/1

diff --git a/modules/varnish/manifests/apt_preferences.pp 
b/modules/varnish/manifests/apt_preferences.pp
deleted file mode 100644
index df8eb78..000
--- a/modules/varnish/manifests/apt_preferences.pp
+++ /dev/null
@@ -1,8 +0,0 @@
-class varnish::apt_preferences {
-# XXX: to remove
-apt::pin { 'varnish':
-ensure   => 'absent',
-pin  => '',
-priority => '',
-}
-}
diff --git a/modules/varnish/manifests/packages.pp 
b/modules/varnish/manifests/packages.pp
index 9f78765..3701b54 100644
--- a/modules/varnish/manifests/packages.pp
+++ b/modules/varnish/manifests/packages.pp
@@ -1,13 +1,10 @@
 class varnish::packages($version='installed') {
-require varnish::apt_preferences
-
 package { [
 'varnish',
 'varnish-dbg',
 'libvarnishapi1',
 ]:
 ensure  => $version,
-require => Class['varnish::apt_preferences'],
 }
 
 # Install VMODs on Varnish 4 instances
@@ -21,6 +18,6 @@
 'libvmod-vslp',
 ]:
 ensure  => 'installed',
-require => [ Class['varnish::apt_preferences'], 
Package['libvmod-header'] ],
+require => Package['libvmod-header'],
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/322703
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I29a80155e5ea8b6b9fb8204486ee862fca63fdc4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_upload: stop graphiq.com buggy javascript

2016-11-23 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/323135

Change subject: cache_upload: stop graphiq.com buggy javascript
..

cache_upload: stop graphiq.com buggy javascript

A bug in graphiq.com's javascript is responsible for thousands of
pointless requests to cache_upload.

Turn graphiq.com 404s into 401s to mitigate the impact.

Bug: T151444
Change-Id: Id34a902cc29b95e7b73bb641fe0cd7e5a521b8ce
---
M modules/varnish/templates/upload-frontend.inc.vcl.erb
1 file changed, 6 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/35/323135/1

diff --git a/modules/varnish/templates/upload-frontend.inc.vcl.erb 
b/modules/varnish/templates/upload-frontend.inc.vcl.erb
index 3e80280..8c27b1c 100644
--- a/modules/varnish/templates/upload-frontend.inc.vcl.erb
+++ b/modules/varnish/templates/upload-frontend.inc.vcl.erb
@@ -91,6 +91,12 @@
 }
 
 sub cluster_fe_deliver {
+   // graphiq.com javascript bug (T151444)
+   if (resp.status == 404 && req.http.Referer ~ 
"^https://w.graphiq.com/w/;) {
+   set resp.status = 401;
+   set resp.http.WWW-Authenticate = {"Basic realm="Buggy request, 
please report at https://www.graphiq.com/contact-us""};
+   }
+
call upload_common_deliver;
 
// The upload cluster does not serve page views or authenticated 
requests, so keep

-- 
To view, visit https://gerrit.wikimedia.org/r/323135
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id34a902cc29b95e7b73bb641fe0cd7e5a521b8ce
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: remove error_synth

2016-11-22 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/322908

Change subject: varnish: remove error_synth
..

varnish: remove error_synth

Bug: T150660
Change-Id: Ib4c9e172d54ecd0a4a36ab30a3b53caa3c97c461
---
M modules/varnish/templates/text-frontend.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
M modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
4 files changed, 16 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/08/322908/1

diff --git a/modules/varnish/templates/text-frontend.inc.vcl.erb 
b/modules/varnish/templates/text-frontend.inc.vcl.erb
index 2acd9af..4014c98 100644
--- a/modules/varnish/templates/text-frontend.inc.vcl.erb
+++ b/modules/varnish/templates/text-frontend.inc.vcl.erb
@@ -41,7 +41,7 @@
} else {
set req.http.Location = "http://; + 
req.http.MobileHost + req.url;
}
-   <%= error_synth(666, "Found") -%>
+   return (synth(666, "Found"));
}
unset req.http.MobileHost;
}
@@ -50,15 +50,15 @@
 sub cluster_fe_recv_pre_purge {
// Forged UAs on zerodot. This largely handles lazywebtools below, 
incidentally.
if (req.http.host ~ "zero\.wikipedia\.org" && req.http.User-Agent && 
req.http.User-Agent ~ "Facebookbot|Googlebot") {
-   <%= error_synth(403, "Noise") -%>
+   return (synth(403, "Noise"));
}
 
if (req.http.referer && req.http.referer ~ 
"^http://(www\.(keeprefreshing|refreshthis|refresh-page|urlreload)\.com|tuneshub\.blogspot\.com|itunes24x7\.blogspot\.com|autoreload\.net|www\.lazywebtools\.co\.uk)/")
 {
-   <%= error_synth(403, "Noise") -%>
+   return (synth(403, "Noise"));
}
 
if (<%= @req_method %> == "POST" && req.url ~ 
"index\.php\?option=com_jce=plugin=imgmanager=imgmanager=form=")
 {
-   <%= error_synth(403, "Noise") -%>
+   return (synth(403, "Noise"));
}
 
// FIXME: we're seeing an issue with Range requests and gzip/gunzip.
@@ -140,7 +140,7 @@
 sub cluster_fe_recv {
// Experiment on dealing with a buggy UA that's spamming requests in 
T141786
if (req.http.User-Agent ~ "Windows NT .*Chrome/41\.0\.2272\.76" && 
req.url == "/" && req.http.X-Connection-Properties ~ "SSL=TLSv1.1; 
C=ECDHE-ECDSA-AES128-SHA;") {
-   <%= error_synth(741, "Buggy request, please report at 
https://phabricator.wikimedia.org/T141786;) -%>
+   return (synth(741, "Buggy request, please report at 
https://phabricator.wikimedia.org/T141786;));
}
 
call mobile_redirect;
@@ -157,7 +157,7 @@
// shortener URLs can be validated and restricted here
if (req.http.host == "<%= @vcl_config.fetch('shortener_domain') %>") {
if (req.url !~ 
"^/[23456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz$_]*$") {
-   <%= error_synth(404, "Short URL Not Found") -%>
+   return (synth(404, "Short URL Not Found"));
}
}
 
@@ -186,7 +186,7 @@
   && req.url ~ "^/wiki/" && req.http.host !~ "\.m\."
   && req.http.Cookie !~ "Browser-Security=Awful"
   && std.random(0,100) < 2.0) {
-   <%= error_synth(787, "Browser Security Warning Redirect") -%>
+   return (synth(787, "Browser Security Warning Redirect"));
}
 
call text_common_recv;
diff --git a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
index ace6b80..b8a79db 100644
--- a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
@@ -13,7 +13,7 @@
 sub vcl_recv {
if (client.ip !~ wikimedia_trust) {
// Do not allow direct access to non-frontend layers
-   <%= error_synth(403, "Access denied") -%>
+   return (synth(403, "Access denied"));
}
 
call wm_common_recv_early;
@@ -26,7 +26,7 @@
if (req.restarts == 0) {
if (req.http.X-DCPath) {
if (req.http.X-DCPath ~ "<%= @site %>") {
-   <%= error_synth(508, "Loop Detected") -%>
+   return (synth(508, "Loop Detected"));
}
set req.http.X-DCPath = req.http.X-DCPath + ", <%= 
@site %>";
} else {
diff --git a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
index 6201b88..0c36d32 100644
--- a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
+++

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: rename varnishlog4.py into varnishlog.py

2016-11-28 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/323822

Change subject: varnish: rename varnishlog4.py into varnishlog.py
..

varnish: rename varnishlog4.py into varnishlog.py

Bug: T150660
Change-Id: I8de988f5686c64e7ab84a4f73dd09750b33f923b
---
M modules/varnish/files/varnishlog.py
D modules/varnish/files/varnishlog4.py
M modules/varnish/manifests/common.pp
3 files changed, 169 insertions(+), 406 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/22/323822/1

diff --git a/modules/varnish/files/varnishlog.py 
b/modules/varnish/files/varnishlog.py
index e218a99..2301420 100755
--- a/modules/varnish/files/varnishlog.py
+++ b/modules/varnish/files/varnishlog.py
@@ -1,11 +1,11 @@
 # -*- coding: utf-8 -*-
+
 """
   varnishlog.py
   ~
 
-  Python 2 and 3 bindings for the Varnish shared log API.
-  This uses ctypes to interface with the Varnish C API,
-  and abstracts away those details.
+  varnishlog.varnishlog calls a given function for every VSL entry that
+  matches filters provided in vsl_args.
 
   Usage:
 
@@ -16,19 +16,32 @@
 varnishlog(
 [
 ('n', 'frontend'),
-('i', 'RxMethod'),
+('i', 'ReqURL'),
 ('c', '')
 ],
-my_callback
-])
+my_callback)
 
-  In the callback, remote_party will either be 'client',
-  'backend', or None.
+  By default, varnishlog will group transactions by request. A specific
+  transaction grouping mode can be used to override the default.
+
+  For example:
+
+varnishlog(
+[
+('g', 'session'),
+# ...
+],
+my_callback)
+
+  See https://www.varnish-cache.org/docs/trunk/reference/vsl-query.html for
+  more details.
+
+  This module depends on python-varnishapi:
+  https://github.com/xcir/python-varnishapi
 
 
-  Copyright 2015
-  Ori Livneh ,
-  Andrew Otto 
+  Copyright 2016 Emanuele Rocca
+  Copyright 2016 Wikimedia Foundation, Inc.
 
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
@@ -41,197 +54,165 @@
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-
-  Authors: Ori Livneh and Andrew Otto
-
 """
-import ctypes
-import ctypes.util
-import functools
-import unittest
+
 import sys
+import time
+import signal
+import inspect
+
+import varnishapi
 
 
-varnishapi_so = ctypes.util.find_library('varnishapi')
-if varnishapi_so is None:
-raise OSError('Unable to locate varnishapi library.')
-varnishapi = ctypes.CDLL(varnishapi_so)
-varnishapi.VSM_New.restype = ctypes.c_void_p
+class VarnishCallbackHandler:
 
-VSL_handler_f = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_void_p, ctypes.c_uint,
- ctypes.c_uint, ctypes.c_uint, ctypes.c_uint,
- ctypes.POINTER(ctypes.c_char),
- ctypes.c_ulonglong)
+def __init__(self, args, callback):
+"""Set up a callback handler. The callback function will be executed
+for all transactions matching the given args.
 
-# Index of Varnish log tags ('RxURL', 'ReqEnd', etc.)
-VSL_Tags = (ctypes.c_char_p * 256).in_dll(varnishapi, 'VSL_tags')
+args must be a list of varnishlog arguments such as:
+[ '-i', 'ReqURL', '-n', 'frontend']
 
-# This flag indicates that the log record was generated as a result
-# of communication with a client. Defined in varnishapi.h.
-VSL_S_CLIENT = 1
+callback needs to be a function with the following signature:
+callback(transaction_id, tag, value, remote_party)
+"""
+self.vap = varnishapi.VarnishLog(args)
+self.callback = callback
 
-# This flag indicates that the log record was generated as a result
-# of communication with a backend server. Defined in varnishapi.h.
-VSL_S_BACKEND = 2
+if self.vap.error:
+raise Exception(self.vap.error)
+
+signal.signal(signal.SIGINT, self.fini)
+signal.signal(signal.SIGTERM, self.fini)
+self.keep_running = True
+
+def fini(self, signum=None, frame=None):
+"""Exit gracefully"""
+self.keep_running = False
+self.vap.Fini()
+
+def execute(self):
+"""Loop and execute callback"""
+while self.keep_running:
+ret = self.vap.Dispatch(self.vap_callback)
+if self.vap.error:
+sys.stderr.write("Error in execute(): %s\n" % self.vap.error)
+self.vap.error = ''
+
+if ret == 0:
+time.sleep(0.01)
+
+def vap_callback(self, vap, cbd, priv):
+"""Callback passed to varnishapi.VarnishLog.Dispatch.
+
+cbd is a dictionary

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: rename varnishlog4.py into varnishlog.py

2016-11-28 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnish: rename varnishlog4.py into varnishlog.py
..


varnish: rename varnishlog4.py into varnishlog.py

Bug: T150660
Change-Id: I8de988f5686c64e7ab84a4f73dd09750b33f923b
---
M modules/varnish/files/varnishlog.py
D modules/varnish/files/varnishlog4.py
M modules/varnish/manifests/common.pp
3 files changed, 169 insertions(+), 406 deletions(-)

Approvals:
  Elukey: Looks good to me, but someone else must approve
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/varnishlog.py 
b/modules/varnish/files/varnishlog.py
index e218a99..2301420 100755
--- a/modules/varnish/files/varnishlog.py
+++ b/modules/varnish/files/varnishlog.py
@@ -1,11 +1,11 @@
 # -*- coding: utf-8 -*-
+
 """
   varnishlog.py
   ~
 
-  Python 2 and 3 bindings for the Varnish shared log API.
-  This uses ctypes to interface with the Varnish C API,
-  and abstracts away those details.
+  varnishlog.varnishlog calls a given function for every VSL entry that
+  matches filters provided in vsl_args.
 
   Usage:
 
@@ -16,19 +16,32 @@
 varnishlog(
 [
 ('n', 'frontend'),
-('i', 'RxMethod'),
+('i', 'ReqURL'),
 ('c', '')
 ],
-my_callback
-])
+my_callback)
 
-  In the callback, remote_party will either be 'client',
-  'backend', or None.
+  By default, varnishlog will group transactions by request. A specific
+  transaction grouping mode can be used to override the default.
+
+  For example:
+
+varnishlog(
+[
+('g', 'session'),
+# ...
+],
+my_callback)
+
+  See https://www.varnish-cache.org/docs/trunk/reference/vsl-query.html for
+  more details.
+
+  This module depends on python-varnishapi:
+  https://github.com/xcir/python-varnishapi
 
 
-  Copyright 2015
-  Ori Livneh ,
-  Andrew Otto 
+  Copyright 2016 Emanuele Rocca
+  Copyright 2016 Wikimedia Foundation, Inc.
 
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
@@ -41,197 +54,165 @@
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-
-  Authors: Ori Livneh and Andrew Otto
-
 """
-import ctypes
-import ctypes.util
-import functools
-import unittest
+
 import sys
+import time
+import signal
+import inspect
+
+import varnishapi
 
 
-varnishapi_so = ctypes.util.find_library('varnishapi')
-if varnishapi_so is None:
-raise OSError('Unable to locate varnishapi library.')
-varnishapi = ctypes.CDLL(varnishapi_so)
-varnishapi.VSM_New.restype = ctypes.c_void_p
+class VarnishCallbackHandler:
 
-VSL_handler_f = ctypes.CFUNCTYPE(ctypes.c_int, ctypes.c_void_p, ctypes.c_uint,
- ctypes.c_uint, ctypes.c_uint, ctypes.c_uint,
- ctypes.POINTER(ctypes.c_char),
- ctypes.c_ulonglong)
+def __init__(self, args, callback):
+"""Set up a callback handler. The callback function will be executed
+for all transactions matching the given args.
 
-# Index of Varnish log tags ('RxURL', 'ReqEnd', etc.)
-VSL_Tags = (ctypes.c_char_p * 256).in_dll(varnishapi, 'VSL_tags')
+args must be a list of varnishlog arguments such as:
+[ '-i', 'ReqURL', '-n', 'frontend']
 
-# This flag indicates that the log record was generated as a result
-# of communication with a client. Defined in varnishapi.h.
-VSL_S_CLIENT = 1
+callback needs to be a function with the following signature:
+callback(transaction_id, tag, value, remote_party)
+"""
+self.vap = varnishapi.VarnishLog(args)
+self.callback = callback
 
-# This flag indicates that the log record was generated as a result
-# of communication with a backend server. Defined in varnishapi.h.
-VSL_S_BACKEND = 2
+if self.vap.error:
+raise Exception(self.vap.error)
+
+signal.signal(signal.SIGINT, self.fini)
+signal.signal(signal.SIGTERM, self.fini)
+self.keep_running = True
+
+def fini(self, signum=None, frame=None):
+"""Exit gracefully"""
+self.keep_running = False
+self.vap.Fini()
+
+def execute(self):
+"""Loop and execute callback"""
+while self.keep_running:
+ret = self.vap.Dispatch(self.vap_callback)
+if self.vap.error:
+sys.stderr.write("Error in execute(): %s\n" % self.vap.error)
+self.vap.error = ''
+
+if ret == 0:
+time.sleep(0.01)
+
+def vap_callback(self, vap, cbd, priv):
+"""Callback passed to varnishapi.VarnishLog.Dispatch.
+
+cbd is a

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: make python varnish scripts skip PURGE requests

2016-11-28 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/323829

Change subject: varnish: make python varnish scripts skip PURGE requests
..

varnish: make python varnish scripts skip PURGE requests

Add VSL queries to filter out PURGE requests where it makes sense.

Bug: T151643
Change-Id: I9f242b57ea092bfbf04016b3a3c675aca4aefeb4
---
M modules/varnish/files/varnishmedia
M modules/varnish/files/varnishrls
M modules/varnish/files/varnishstatsd
M modules/varnish/files/varnishxcache
M modules/varnish/files/varnishxcps
5 files changed, 11 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/29/323829/1

diff --git a/modules/varnish/files/varnishmedia 
b/modules/varnish/files/varnishmedia
index 27a0535..89728b0 100755
--- a/modules/varnish/files/varnishmedia
+++ b/modules/varnish/files/varnishmedia
@@ -56,11 +56,12 @@
 
 def start(self):
 varnishlog.varnishlog((
-('q', 'ReqURL ~ "/thumb/"'), # Only look at thumb requests
-('n', 'frontend'),   # Only frontend Varnish instance
-('i', 'RespStatus'), # RespStatus for the HTTP status code
-('i', 'ReqURL'), # ReqURL to delimit requests
-('i', 'Timestamp'),  # Timestamp to delimit requests
+('q', 'ReqMethod ne "PURGE"'),  # Skip PURGE requests
+('q', 'ReqURL ~ "/thumb/"'),# Only look at thumb requests
+('n', 'frontend'),  # Only frontend Varnish instance
+('i', 'RespStatus'),# RespStatus for the HTTP status 
code
+('i', 'ReqURL'),# ReqURL to delimit requests
+('i', 'Timestamp'), # Timestamp to delimit requests
 ), self.handle_log_record)
 
 lp = MediaVarnishLogProcessor()
diff --git a/modules/varnish/files/varnishrls b/modules/varnish/files/varnishrls
index 6949546..cfaf5a1 100755
--- a/modules/varnish/files/varnishrls
+++ b/modules/varnish/files/varnishrls
@@ -65,7 +65,7 @@
 
 def start(self):
 # VSL query matching ResourceLoader ReqURLs
-query = 'ReqURL ~ "^/w/load.php"'
+query = 'ReqURL ~ "^/w/load.php" and ReqMethod ne "PURGE"'
 
 varnishlog.varnishlog((
 ('q', query), # VSL query
diff --git a/modules/varnish/files/varnishstatsd 
b/modules/varnish/files/varnishstatsd
index 9a0089b..41965a2 100755
--- a/modules/varnish/files/varnishstatsd
+++ b/modules/varnish/files/varnishstatsd
@@ -46,6 +46,8 @@
 UDP_MTU_BYTES = 1472
 
 vsl_args = [
+# Skip PURGE requests
+('q', 'ReqMethod ne "PURGE"'),
 # Backend HTTP response status code (eg: 200, 404)
 ('i', 'BerespStatus'),
 # Backend HTTP request method (eg: GET, POST)
diff --git a/modules/varnish/files/varnishxcache 
b/modules/varnish/files/varnishxcache
index c2b3ec3..ebbcead 100755
--- a/modules/varnish/files/varnishxcache
+++ b/modules/varnish/files/varnishxcache
@@ -120,6 +120,7 @@
 
 
 varnishlog.varnishlog((
+('q', 'ReqMethod ne "PURGE"'),
 ('I', 'RespHeader:^X-Cache:'),
 ('n', 'frontend'),
 ), vsl_callback)
diff --git a/modules/varnish/files/varnishxcps 
b/modules/varnish/files/varnishxcps
index 73ddf8d..13289da 100755
--- a/modules/varnish/files/varnishxcps
+++ b/modules/varnish/files/varnishxcps
@@ -97,6 +97,7 @@
 vsl_callback.next_pub = time.time() + args.interval
 
 varnishlog.varnishlog((
+('q', 'ReqMethod ne "PURGE"'),
 ('I', 'ReqHeader:^X-Connection-Properties:'),
 ('n', 'frontend'),
 ), vsl_callback)

-- 
To view, visit https://gerrit.wikimedia.org/r/323829
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9f242b57ea092bfbf04016b3a3c675aca4aefeb4
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: make python varnish scripts skip PURGE requests

2016-11-28 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnish: make python varnish scripts skip PURGE requests
..


varnish: make python varnish scripts skip PURGE requests

Add VSL queries to filter out PURGE requests where it makes sense.

Bug: T151643
Change-Id: I9f242b57ea092bfbf04016b3a3c675aca4aefeb4
---
M modules/varnish/files/varnishmedia
M modules/varnish/files/varnishrls
M modules/varnish/files/varnishstatsd
M modules/varnish/files/varnishxcache
M modules/varnish/files/varnishxcps
5 files changed, 11 insertions(+), 6 deletions(-)

Approvals:
  Elukey: Looks good to me, but someone else must approve
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/varnishmedia 
b/modules/varnish/files/varnishmedia
index 27a0535..89728b0 100755
--- a/modules/varnish/files/varnishmedia
+++ b/modules/varnish/files/varnishmedia
@@ -56,11 +56,12 @@
 
 def start(self):
 varnishlog.varnishlog((
-('q', 'ReqURL ~ "/thumb/"'), # Only look at thumb requests
-('n', 'frontend'),   # Only frontend Varnish instance
-('i', 'RespStatus'), # RespStatus for the HTTP status code
-('i', 'ReqURL'), # ReqURL to delimit requests
-('i', 'Timestamp'),  # Timestamp to delimit requests
+('q', 'ReqMethod ne "PURGE"'),  # Skip PURGE requests
+('q', 'ReqURL ~ "/thumb/"'),# Only look at thumb requests
+('n', 'frontend'),  # Only frontend Varnish instance
+('i', 'RespStatus'),# RespStatus for the HTTP status 
code
+('i', 'ReqURL'),# ReqURL to delimit requests
+('i', 'Timestamp'), # Timestamp to delimit requests
 ), self.handle_log_record)
 
 lp = MediaVarnishLogProcessor()
diff --git a/modules/varnish/files/varnishrls b/modules/varnish/files/varnishrls
index 6949546..cfaf5a1 100755
--- a/modules/varnish/files/varnishrls
+++ b/modules/varnish/files/varnishrls
@@ -65,7 +65,7 @@
 
 def start(self):
 # VSL query matching ResourceLoader ReqURLs
-query = 'ReqURL ~ "^/w/load.php"'
+query = 'ReqURL ~ "^/w/load.php" and ReqMethod ne "PURGE"'
 
 varnishlog.varnishlog((
 ('q', query), # VSL query
diff --git a/modules/varnish/files/varnishstatsd 
b/modules/varnish/files/varnishstatsd
index 9a0089b..41965a2 100755
--- a/modules/varnish/files/varnishstatsd
+++ b/modules/varnish/files/varnishstatsd
@@ -46,6 +46,8 @@
 UDP_MTU_BYTES = 1472
 
 vsl_args = [
+# Skip PURGE requests
+('q', 'ReqMethod ne "PURGE"'),
 # Backend HTTP response status code (eg: 200, 404)
 ('i', 'BerespStatus'),
 # Backend HTTP request method (eg: GET, POST)
diff --git a/modules/varnish/files/varnishxcache 
b/modules/varnish/files/varnishxcache
index c2b3ec3..ebbcead 100755
--- a/modules/varnish/files/varnishxcache
+++ b/modules/varnish/files/varnishxcache
@@ -120,6 +120,7 @@
 
 
 varnishlog.varnishlog((
+('q', 'ReqMethod ne "PURGE"'),
 ('I', 'RespHeader:^X-Cache:'),
 ('n', 'frontend'),
 ), vsl_callback)
diff --git a/modules/varnish/files/varnishxcps 
b/modules/varnish/files/varnishxcps
index 73ddf8d..13289da 100755
--- a/modules/varnish/files/varnishxcps
+++ b/modules/varnish/files/varnishxcps
@@ -97,6 +97,7 @@
 vsl_callback.next_pub = time.time() + args.interval
 
 varnishlog.varnishlog((
+('q', 'ReqMethod ne "PURGE"'),
 ('I', 'ReqHeader:^X-Connection-Properties:'),
 ('n', 'frontend'),
 ), vsl_callback)

-- 
To view, visit https://gerrit.wikimedia.org/r/323829
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9f242b57ea092bfbf04016b3a3c675aca4aefeb4
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: rename scripts depending on varnishlog.py

2016-11-24 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/323423

Change subject: varnish: rename scripts depending on varnishlog.py
..

varnish: rename scripts depending on varnishlog.py

We do not need to have two different versions of the python varnishlog
scripts now that the upgrade to Varnish 4 is finished.

Bug: T150660
Change-Id: I5b1024c268a9ddf21d5596bc5338d54e9286ea97
---
M modules/varnish/files/varnishmedia
D modules/varnish/files/varnishmedia4
M modules/varnish/files/varnishprocessor/varnishprocessor.py
D modules/varnish/files/varnishprocessor4/__init__.py
D modules/varnish/files/varnishprocessor4/varnishprocessor.py
M modules/varnish/files/varnishreqstats
D modules/varnish/files/varnishreqstats4
M modules/varnish/files/varnishrls
D modules/varnish/files/varnishrls4
M modules/varnish/files/varnishstatsd
D modules/varnish/files/varnishstatsd4
M modules/varnish/files/varnishxcache
D modules/varnish/files/varnishxcache4
M modules/varnish/files/varnishxcps
D modules/varnish/files/varnishxcps4
M modules/varnish/manifests/common.pp
M modules/varnish/manifests/logging/media.pp
M modules/varnish/manifests/logging/reqstats.pp
M modules/varnish/manifests/logging/rls.pp
M modules/varnish/manifests/logging/statsd.pp
M modules/varnish/manifests/logging/xcache.pp
M modules/varnish/manifests/logging/xcps.pp
22 files changed, 80 insertions(+), 1,014 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/23/323423/1

diff --git a/modules/varnish/files/varnishmedia 
b/modules/varnish/files/varnishmedia
index 2466134..27a0535 100755
--- a/modules/varnish/files/varnishmedia
+++ b/modules/varnish/files/varnishmedia
@@ -42,7 +42,7 @@
 
 def process_transaction(self, transaction):
 """Process a single completed transaction."""
-status_code = transaction.get('TxStatus')
+status_code = transaction.get('RespStatus')
 if status_code is None:
 return
 
@@ -56,11 +56,11 @@
 
 def start(self):
 varnishlog.varnishlog((
-('m', 'RxURL:/thumb/'), # Only look at thumb requests
-('n', 'frontend'),  # Consider the frontend Varnish instance
-('i', 'TxStatus'),  # Get TxStatus for the HTTP status code
-('i', 'RxURL'), # Get RxURL to delimit requests
-('i', 'ReqEnd'),# Get ReqEnd to delimit requests
+('q', 'ReqURL ~ "/thumb/"'), # Only look at thumb requests
+('n', 'frontend'),   # Only frontend Varnish instance
+('i', 'RespStatus'), # RespStatus for the HTTP status code
+('i', 'ReqURL'), # ReqURL to delimit requests
+('i', 'Timestamp'),  # Timestamp to delimit requests
 ), self.handle_log_record)
 
 lp = MediaVarnishLogProcessor()
diff --git a/modules/varnish/files/varnishmedia4 
b/modules/varnish/files/varnishmedia4
deleted file mode 100755
index 27a0535..000
--- a/modules/varnish/files/varnishmedia4
+++ /dev/null
@@ -1,66 +0,0 @@
-#!/usr/bin/env python
-# -*- coding: utf-8 -*-
-"""
-  varnishmedia
-  
-
-  Accumulate browser cache hit ratio and total request volume statistics
-  for media requests and report to StatsD.
-
-  Usage: varnishmedia [--statsd-server SERVER] [--key-prefix PREFIX]
-
---statsd-server SERVER  statsd server (default: none; echo to stdout)
---key-prefix PREFIX metric key prefix (default: 
media.thumbnail.varnish)
-
-  Copyright 2015 Ori Livneh 
-  Copyright 2015 Gilles Dubuc 
-
-  Licensed under the Apache License, Version 2.0 (the "License");
-  you may not use this file except in compliance with the License.
-  You may obtain a copy of the License at
-
-  http://www.apache.org/licenses/LICENSE-2.0
-
-  Unless required by applicable law or agreed to in writing, software
-  distributed under the License is distributed on an "AS IS" BASIS,
-  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-  See the License for the specific language governing permissions and
-  limitations under the License.
-
-"""
-from __future__ import division
-
-import re
-
-import varnishlog
-import varnishprocessor
-
-
-class MediaVarnishLogProcessor(varnishprocessor.VarnishLogProcessor):
-description = 'Media Browser Cache Hit Ratio StatsD Reporter'
-key_prefix = 'media.thumbnail.varnish'
-
-def process_transaction(self, transaction):
-"""Process a single completed transaction."""
-status_code = transaction.get('RespStatus')
-if status_code is None:
-return
-
-metric_keys = ['reqs.all', 'resps.' + status_code]
-
-for key in metric_keys:
-self.stats[key] = self.stats.get(key, 0) + 1
-
-if self.stats['reqs.all'] > 1:
-self.flush_stats()
-
-def start(self):
-

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache: get rid of varnish 3 compatibility code

2016-11-18 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/322252

Change subject: cache: get rid of varnish 3 compatibility code
..

cache: get rid of varnish 3 compatibility code

Now that cache_text has been fully upgraded to Varnish 4 we do not have
any more servers running Varnish 3. Remove v3-specific code.

Bug: T150660
Bug: T131499
Change-Id: I2e2b33a988a74d7ca973c786337702d487353e28
---
M modules/role/manifests/cache/base.pp
M modules/role/manifests/cache/instances.pp
M modules/role/manifests/cache/kafka.pp
M modules/role/manifests/cache/kafka/eventlogging.pp
M modules/role/manifests/cache/kafka/statsv.pp
M modules/role/manifests/cache/kafka/webrequest.pp
M modules/role/manifests/cache/text.pp
M modules/tlsproxy/manifests/instance.pp
M modules/tlsproxy/manifests/localssl.pp
M modules/tlsproxy/templates/localssl.erb
M modules/tlsproxy/templates/nginx.conf.erb
M modules/varnish/manifests/apt_preferences.pp
M modules/varnish/manifests/common.pp
M modules/varnish/manifests/common/vcl.pp
M modules/varnish/manifests/instance.pp
M modules/varnish/manifests/logging/media.pp
M modules/varnish/manifests/logging/reqstats.pp
M modules/varnish/manifests/logging/rls.pp
M modules/varnish/manifests/logging/statsd.pp
M modules/varnish/manifests/logging/xcache.pp
M modules/varnish/manifests/logging/xcps.pp
M modules/varnish/manifests/packages.pp
M modules/varnish/templates/analytics.inc.vcl.erb
M modules/varnish/templates/errorpage.inc.vcl.erb
M modules/varnish/templates/geoip.inc.vcl.erb
M modules/varnish/templates/initscripts/varnish.systemd.erb
M modules/varnish/templates/normalize_path.inc.vcl.erb
M modules/varnish/templates/text-backend.inc.vcl.erb
M modules/varnish/templates/text-common.inc.vcl.erb
M modules/varnish/templates/text-frontend.inc.vcl.erb
M modules/varnish/templates/vcl/directors.vcl.tpl.erb
M modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
M modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
34 files changed, 141 insertions(+), 609 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/52/322252/1

diff --git a/modules/role/manifests/cache/base.pp 
b/modules/role/manifests/cache/base.pp
index f7c9e14..39b3c58 100644
--- a/modules/role/manifests/cache/base.pp
+++ b/modules/role/manifests/cache/base.pp
@@ -35,28 +35,14 @@
 site => $zero_site,
 }
 
-###
-# Varnish4 Transition
-###
-
-$varnish_version4 = hiera('varnish_version4', false)
-
-if $varnish_version4 {
-salt::grain { 'varnish_version':
-ensure  => present,
-replace => true,
-value   => 4,
-}
-} else {
-salt::grain { 'varnish_version':
-ensure  => present,
-replace => true,
-value   => 3,
-}
+salt::grain { 'varnish_version':
+ensure  => present,
+replace => true,
+value   => 4,
 }
 
 # XXX: temporary, we need this to mitigate T145661
-if $::realm == 'production' and $varnish_version4 {
+if $::realm == 'production' {
 $hnodes = hiera("cache::${cache_cluster}::nodes")
 $all_nodes = array_concat($hnodes['eqiad'], $hnodes['esams'], 
$hnodes['ulsfo'], $hnodes['codfw'])
 $times = cron_splay($all_nodes, 'weekly', 
"${cache_cluster}-backend-restarts")
@@ -177,13 +163,8 @@
 varnish::setup_filesystem { $filesystems: }
 Varnish::Setup_filesystem <| |> -> Varnish::Instance <| |>
 
-if ($varnish_version4) {
-# https://www.varnish-cache.org/docs/trunk/phk/persistent.html
-$persistent_name = 'deprecated_persistent'
-}
-else {
-$persistent_name = 'persistent'
-}
+# https://www.varnish-cache.org/docs/trunk/phk/persistent.html
+$persistent_name = 'deprecated_persistent'
 
 # This is the "normal" persistent storage varnish args, for consuming all 
available space
 # (upload uses something more complex than this based on our storage vars 
above as well!)
diff --git a/modules/role/manifests/cache/instances.pp 
b/modules/role/manifests/cache/instances.pp
index d5e84e2..a334ba9 100644
--- a/modules/role/manifests/cache/instances.pp
+++ b/modules/role/manifests/cache/instances.pp
@@ -15,11 +15,7 @@
 $cluster_nodes
 ) {
 
-if hiera('varnish_version4', false) {
-$chash_dir = 'vslp'
-} else {
-$chash_dir = 'chash'
-}
+$chash_dir = 'vslp'
 
 $cache_route_table = hiera('cache::route_table')
 $cache_route = $cache_route_table[$::site]
diff --git a/modules/role/manifests/cache/kafka.pp 
b/modules/role/manifests/cache/kafka.pp
index 5383e9c..2e67ad4 100644
--- a/modules/role/manifests/cache/kafka.pp
+++

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Revert "tlsproxy: turn proxy_request_buffering off for v4"

2016-11-11 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: Revert "tlsproxy: turn proxy_request_buffering off for v4"
..


Revert "tlsproxy: turn proxy_request_buffering off for v4"

We're seeing buggy POST requests hitting our varnish frontends. Those
requests have Content-Length > 0 but empty request body, and they result
in 503s.

Turning proxy_request_buffering on seems to improve the situation.

Bug: T150503
This reverts commit 5d3c208969b6a93f71eb374a54a13e97203eb93f.

Change-Id: I1cb4cc3bb200a3f3866ad721672a5f7369115d26
---
M modules/tlsproxy/templates/localssl.erb
1 file changed, 0 insertions(+), 3 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/tlsproxy/templates/localssl.erb 
b/modules/tlsproxy/templates/localssl.erb
index 0219940..491e5ac 100644
--- a/modules/tlsproxy/templates/localssl.erb
+++ b/modules/tlsproxy/templates/localssl.erb
@@ -69,9 +69,6 @@
 
proxy_redirect off;
proxy_buffering off;
-<% if @varnish_version4 -%>
-   proxy_request_buffering off;
-<% end -%>
}
 }
 <% if @redir_port -%>

-- 
To view, visit https://gerrit.wikimedia.org/r/321000
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1cb4cc3bb200a3f3866ad721672a5f7369115d26
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache: get rid of varnish 3 compatibility code

2016-11-21 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache: get rid of varnish 3 compatibility code
..


cache: get rid of varnish 3 compatibility code

Now that cache_text has been fully upgraded to Varnish 4 we do not have
any more servers running Varnish 3. Remove v3-specific code.

Bug: T150660
Bug: T131499
Change-Id: I2e2b33a988a74d7ca973c786337702d487353e28
---
M modules/role/manifests/cache/base.pp
M modules/role/manifests/cache/instances.pp
M modules/role/manifests/cache/kafka.pp
M modules/role/manifests/cache/kafka/eventlogging.pp
M modules/role/manifests/cache/kafka/statsv.pp
M modules/role/manifests/cache/kafka/webrequest.pp
M modules/role/manifests/cache/text.pp
M modules/tlsproxy/manifests/instance.pp
M modules/tlsproxy/manifests/localssl.pp
M modules/tlsproxy/templates/localssl.erb
M modules/tlsproxy/templates/nginx.conf.erb
M modules/varnish/manifests/apt_preferences.pp
M modules/varnish/manifests/common.pp
M modules/varnish/manifests/common/vcl.pp
M modules/varnish/manifests/instance.pp
M modules/varnish/manifests/logging/media.pp
M modules/varnish/manifests/logging/reqstats.pp
M modules/varnish/manifests/logging/rls.pp
M modules/varnish/manifests/logging/statsd.pp
M modules/varnish/manifests/logging/xcache.pp
M modules/varnish/manifests/logging/xcps.pp
M modules/varnish/manifests/packages.pp
M modules/varnish/templates/analytics.inc.vcl.erb
M modules/varnish/templates/errorpage.inc.vcl.erb
M modules/varnish/templates/geoip.inc.vcl.erb
M modules/varnish/templates/initscripts/varnish.systemd.erb
M modules/varnish/templates/normalize_path.inc.vcl.erb
M modules/varnish/templates/text-backend.inc.vcl.erb
M modules/varnish/templates/text-common.inc.vcl.erb
M modules/varnish/templates/text-frontend.inc.vcl.erb
M modules/varnish/templates/vcl/directors.vcl.tpl.erb
M modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
M modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
34 files changed, 141 insertions(+), 609 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve



diff --git a/modules/role/manifests/cache/base.pp 
b/modules/role/manifests/cache/base.pp
index f7c9e14..39b3c58 100644
--- a/modules/role/manifests/cache/base.pp
+++ b/modules/role/manifests/cache/base.pp
@@ -35,28 +35,14 @@
 site => $zero_site,
 }
 
-###
-# Varnish4 Transition
-###
-
-$varnish_version4 = hiera('varnish_version4', false)
-
-if $varnish_version4 {
-salt::grain { 'varnish_version':
-ensure  => present,
-replace => true,
-value   => 4,
-}
-} else {
-salt::grain { 'varnish_version':
-ensure  => present,
-replace => true,
-value   => 3,
-}
+salt::grain { 'varnish_version':
+ensure  => present,
+replace => true,
+value   => 4,
 }
 
 # XXX: temporary, we need this to mitigate T145661
-if $::realm == 'production' and $varnish_version4 {
+if $::realm == 'production' {
 $hnodes = hiera("cache::${cache_cluster}::nodes")
 $all_nodes = array_concat($hnodes['eqiad'], $hnodes['esams'], 
$hnodes['ulsfo'], $hnodes['codfw'])
 $times = cron_splay($all_nodes, 'weekly', 
"${cache_cluster}-backend-restarts")
@@ -177,13 +163,8 @@
 varnish::setup_filesystem { $filesystems: }
 Varnish::Setup_filesystem <| |> -> Varnish::Instance <| |>
 
-if ($varnish_version4) {
-# https://www.varnish-cache.org/docs/trunk/phk/persistent.html
-$persistent_name = 'deprecated_persistent'
-}
-else {
-$persistent_name = 'persistent'
-}
+# https://www.varnish-cache.org/docs/trunk/phk/persistent.html
+$persistent_name = 'deprecated_persistent'
 
 # This is the "normal" persistent storage varnish args, for consuming all 
available space
 # (upload uses something more complex than this based on our storage vars 
above as well!)
diff --git a/modules/role/manifests/cache/instances.pp 
b/modules/role/manifests/cache/instances.pp
index d5e84e2..a334ba9 100644
--- a/modules/role/manifests/cache/instances.pp
+++ b/modules/role/manifests/cache/instances.pp
@@ -15,11 +15,7 @@
 $cluster_nodes
 ) {
 
-if hiera('varnish_version4', false) {
-$chash_dir = 'vslp'
-} else {
-$chash_dir = 'chash'
-}
+$chash_dir = 'vslp'
 
 $cache_route_table = hiera('cache::route_table')
 $cache_route = $cache_route_table[$::site]
diff --git a/modules/role/manifests/cache/kafka.pp 
b/modules/role/manifests/cache/kafka.pp
index 5383e9c..2e67ad4 100644
--- a/modules/role/manifests/cache/kafka.pp
+++

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache: remove persistent storage support

2016-11-21 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/322691

Change subject: cache: remove persistent storage support
..

cache: remove persistent storage support

We've switched to file storage on all systems running Varnish 4 in
50fdd2e. As we're not running Varnish 3 anymore, remove persistent
storage support altogether.

Bug: T150660
Change-Id: I27ffdf044128c5f7775609093fc433ce19784ca2
---
M modules/role/manifests/cache/base.pp
1 file changed, 0 insertions(+), 26 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/91/322691/1

diff --git a/modules/role/manifests/cache/base.pp 
b/modules/role/manifests/cache/base.pp
index 39b3c58..8274056 100644
--- a/modules/role/manifests/cache/base.pp
+++ b/modules/role/manifests/cache/base.pp
@@ -133,22 +133,6 @@
 # Storage configuration
 ###
 
-# mma: mmap addrseses for fixed persistent storage on x86_64 Linux:
-#  This scheme fits 4x fixed memory mappings of up to 4TB each
-#  into the range 0x5000 - 0x5FFF, which on
-#  x86_64 Linux is in the middle of the user address space and thus
-#  unlikely to ever be used by normal, auto-addressed allocations,
-#  as those grow in from the edges (typically from the top, but
-#  possibly from the bottom depending).  Regardless of which
-#  direction heap grows from, there's 32TB or more for normal
-#  allocations to chew through before they reach our fixed range.
-$mma = [
-'0x5000',
-'0x5400',
-'0x5800',
-'0x5C00',
-]
-
 # everything from here down is related to backend storage/weight config
 
 $storage_size = $::hostname ? {
@@ -162,16 +146,6 @@
 $filesystems = unique($storage_parts)
 varnish::setup_filesystem { $filesystems: }
 Varnish::Setup_filesystem <| |> -> Varnish::Instance <| |>
-
-# https://www.varnish-cache.org/docs/trunk/phk/persistent.html
-$persistent_name = 'deprecated_persistent'
-
-# This is the "normal" persistent storage varnish args, for consuming all 
available space
-# (upload uses something more complex than this based on our storage vars 
above as well!)
-$persistent_storage_args = join([
-"-s 
main1=${persistent_name},/srv/${storage_parts[0]}/varnish.main1,${storage_size}G,${mma[0]}",
-"-s 
main2=${persistent_name},/srv/${storage_parts[1]}/varnish.main2,${storage_size}G,${mma[1]}",
-], ' ')
 
 $file_storage_args = join([
 "-s 
main1=file,/srv/${storage_parts[0]}/varnish.main1,${storage_size}G",

-- 
To view, visit https://gerrit.wikimedia.org/r/322691
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I27ffdf044128c5f7775609093fc433ce19784ca2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache: remove persistent storage support

2016-11-21 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache: remove persistent storage support
..


cache: remove persistent storage support

We've switched to file storage on all systems running Varnish 4 in
50fdd2e. As we're not running Varnish 3 anymore, remove persistent
storage support altogether.

Bug: T150660
Change-Id: I27ffdf044128c5f7775609093fc433ce19784ca2
---
M modules/role/manifests/cache/base.pp
1 file changed, 0 insertions(+), 26 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified
  BBlack: Looks good to me, but someone else must approve



diff --git a/modules/role/manifests/cache/base.pp 
b/modules/role/manifests/cache/base.pp
index 39b3c58..8274056 100644
--- a/modules/role/manifests/cache/base.pp
+++ b/modules/role/manifests/cache/base.pp
@@ -133,22 +133,6 @@
 # Storage configuration
 ###
 
-# mma: mmap addrseses for fixed persistent storage on x86_64 Linux:
-#  This scheme fits 4x fixed memory mappings of up to 4TB each
-#  into the range 0x5000 - 0x5FFF, which on
-#  x86_64 Linux is in the middle of the user address space and thus
-#  unlikely to ever be used by normal, auto-addressed allocations,
-#  as those grow in from the edges (typically from the top, but
-#  possibly from the bottom depending).  Regardless of which
-#  direction heap grows from, there's 32TB or more for normal
-#  allocations to chew through before they reach our fixed range.
-$mma = [
-'0x5000',
-'0x5400',
-'0x5800',
-'0x5C00',
-]
-
 # everything from here down is related to backend storage/weight config
 
 $storage_size = $::hostname ? {
@@ -162,16 +146,6 @@
 $filesystems = unique($storage_parts)
 varnish::setup_filesystem { $filesystems: }
 Varnish::Setup_filesystem <| |> -> Varnish::Instance <| |>
-
-# https://www.varnish-cache.org/docs/trunk/phk/persistent.html
-$persistent_name = 'deprecated_persistent'
-
-# This is the "normal" persistent storage varnish args, for consuming all 
available space
-# (upload uses something more complex than this based on our storage vars 
above as well!)
-$persistent_storage_args = join([
-"-s 
main1=${persistent_name},/srv/${storage_parts[0]}/varnish.main1,${storage_size}G,${mma[0]}",
-"-s 
main2=${persistent_name},/srv/${storage_parts[1]}/varnish.main2,${storage_size}G,${mma[1]}",
-], ' ')
 
 $file_storage_args = join([
 "-s 
main1=file,/srv/${storage_parts[0]}/varnish.main1,${storage_size}G",

-- 
To view, visit https://gerrit.wikimedia.org/r/322691
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I27ffdf044128c5f7775609093fc433ce19784ca2
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...varnish4[debian-wmf]: 4.1.3-1wm4: gethdr_extrachance

2016-11-15 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: 4.1.3-1wm4: gethdr_extrachance
..


4.1.3-1wm4: gethdr_extrachance

Change-Id: Id47a7fc006a81bf0435819fefd94c8a0c825d2d9
---
M debian/changelog
A debian/patches/0005-gethdr_extrachance.patch
D debian/patches/0005-remove_bad_extrachance_code.patch
M debian/patches/series
4 files changed, 54 insertions(+), 47 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/debian/changelog b/debian/changelog
index 297cfc9..319edd8 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+varnish (4.1.3-1wm4) jessie-wikimedia; urgency=medium
+
+  * Make extrachance upper bound configurable.
+
+ -- Emanuele Rocca   Fri, 11 Nov 2016 12:01:18 +0100
+
 varnish (4.1.3-1wm3) jessie-wikimedia; urgency=medium
 
   * Add 0005-remove_bad_extrachance_code.patch: the extrachance code is
diff --git a/debian/patches/0005-gethdr_extrachance.patch 
b/debian/patches/0005-gethdr_extrachance.patch
new file mode 100644
index 000..f8144ad
--- /dev/null
+++ b/debian/patches/0005-gethdr_extrachance.patch
@@ -0,0 +1,47 @@
+diff --git a/bin/varnishd/cache/cache_backend.c 
b/bin/varnishd/cache/cache_backend.c
+index 039d3b2..4324894 100644
+--- a/bin/varnishd/cache/cache_backend.c
 b/bin/varnishd/cache/cache_backend.c
+@@ -177,7 +177,7 @@ static int __match_proto__(vdi_gethdrs_f)
+ vbe_dir_gethdrs(const struct director *d, struct worker *wrk,
+ struct busyobj *bo)
+ {
+-  int i, extrachance = 1;
++  int i, extrachance = cache_param->gethdr_extrachance;
+   struct backend *bp;
+   struct vbc *vbc;
+ 
+@@ -232,7 +232,7 @@ vbe_dir_gethdrs(const struct director *d, struct worker 
*wrk,
+   bo->req->req_body_status != REQ_BODY_CACHED)
+   break;
+   VSC_C_main->backend_retry++;
+-  } while (extrachance);
++  } while (extrachance-- > 0);
+   return (-1);
+ }
+ 
+diff --git a/include/tbl/params.h b/include/tbl/params.h
+index 5c3f46b..ec0a8cf 100644
+--- a/include/tbl/params.h
 b/include/tbl/params.h
+@@ -458,6 +458,20 @@ PARAM(
+ )
+ 
+ PARAM(
++  /* name */  gethdr_extrachance,
++  /* typ */   uint,
++  /* min */   "0",
++  /* max */   "100",
++  /* default */   "1",
++  /* units */ NULL,
++  /* flags */ EXPERIMENTAL,
++  /* s-text */
++  "Number of extra chances in vbe_dir_gethdrs.",
++  /* l-text */"",
++  /* func */  NULL
++)
++
++PARAM(
+   /* name */  gzip_buffer,
+   /* typ */   bytes_u,
+   /* min */   "2k",
diff --git a/debian/patches/0005-remove_bad_extrachance_code.patch 
b/debian/patches/0005-remove_bad_extrachance_code.patch
deleted file mode 100644
index d159f8a..000
--- a/debian/patches/0005-remove_bad_extrachance_code.patch
+++ /dev/null
@@ -1,46 +0,0 @@
-From: Brandon Black 
-Date: Tue Nov  8 11:59:43 UTC 2016
-Subject: remove_bad_extrachance_code
-
-Bug: T150247
-diff --git a/bin/varnishd/cache/cache_backend.c 
b/bin/varnishd/cache/cache_backend.c
-index 039d3b2..f88bf8e 100644
 a/bin/varnishd/cache/cache_backend.c
-+++ b/bin/varnishd/cache/cache_backend.c
-@@ -177,7 +177,7 @@ static int __match_proto__(vdi_gethdrs_f)
- vbe_dir_gethdrs(const struct director *d, struct worker *wrk,
- struct busyobj *bo)
- {
--  int i, extrachance = 1;
-+  int i; /*, extrachance = 1;*/
-   struct backend *bp;
-   struct vbc *vbc;
- 
-@@ -201,8 +201,10 @@ vbe_dir_gethdrs(const struct director *d, struct worker 
*wrk,
-   return (-1);
-   }
-   AN(bo->htc);
-+  /*
-   if (vbc->state != VBC_STATE_STOLEN)
-   extrachance = 0;
-+  */
- 
-   i = V1F_SendReq(wrk, bo, >acct.bereq_hdrbytes, 0);
- 
-@@ -225,6 +227,7 @@ vbe_dir_gethdrs(const struct director *d, struct worker 
*wrk,
-*/
-   vbe_dir_finish(d, wrk, bo);
-   AZ(bo->htc);
-+  /*
-   if (i < 0)
-   break;
-   if (bo->req != NULL &&
-@@ -232,7 +235,8 @@ vbe_dir_gethdrs(const struct director *d, struct worker 
*wrk,
-   bo->req->req_body_status != REQ_BODY_CACHED)
-   break;
-   VSC_C_main->backend_retry++;
--  } while (extrachance);
-+  } while (extrachance);*/
-+  } while (0);
-   return (-1);
- }
diff --git a/debian/patches/series b/debian/patches/series
index 273c363..44fc4a0 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1,4 +1,4 @@
 0001-varnishd-persistent-addrarg.patch
 0002-varnishd-persistent-fallocate.patch
 0004-varnishd-persistent-mmap-exit.patch
-0005-remove_bad_extrachance_code.patch

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishrls4: use VSL query and proper tags

2016-10-21 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnishrls4: use VSL query and proper tags
..


varnishrls4: use VSL query and proper tags

Extract transactions from the VSM log with a VSL query matching
ResourceLoader URLs. Further, use the right tags in process_transaction.

Bug: T131353
Change-Id: I4b3a5ccc8dedb5ff8b51c3a22ca9cef8efb803dc
---
M modules/varnish/files/varnishrls4
1 file changed, 14 insertions(+), 15 deletions(-)

Approvals:
  Elukey: Looks good to me, but someone else must approve
  Ema: Verified; Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve



diff --git a/modules/varnish/files/varnishrls4 
b/modules/varnish/files/varnishrls4
index dd03f29..6949546 100755
--- a/modules/varnish/files/varnishrls4
+++ b/modules/varnish/files/varnishrls4
@@ -35,6 +35,7 @@
 import varnishlog
 import varnishprocessor
 
+
 class ResourceLoaderVarnishLogProcessor(varnishprocessor.VarnishLogProcessor):
 description = 'ResourceLoader Browser Cache Hit Ratio StatsD Reporter'
 key_prefix = 'ResourceLoader'
@@ -44,10 +45,10 @@
 status_code = transaction['RespStatus']
 metric_keys = ['reqs.all', 'resps.' + status_code]
 
-if 'BerespHeader' in transaction:
+if 'ReqHeader' in transaction:
 metric_keys.append('reqs.if_none_match')
 
-cache_control_header = transaction.get('BereqHeader')
+cache_control_header = transaction.get('RespHeader')
 cache_control = 'no'
 if cache_control_header:
 match = re.search(r'(?<=max-age=)\d+', cache_control_header)
@@ -63,20 +64,18 @@
 self.flush_stats()
 
 def start(self):
+# VSL query matching ResourceLoader ReqURLs
+query = 'ReqURL ~ "^/w/load.php"'
+
 varnishlog.varnishlog((
-('n', 'frontend'), # Consider the frontend Varnish instance
-('c', None),   # Only consider interactions with the client
-('i', 'RespStatus'),   # Get RespStatus for the HTTP status code
-('i', 'BerespHeader'), # Get BerespHeader for If-None-Match header
-('i', 'BereqHeader'),  # Get BereqHeader for Cache-control header
-('i', 'ReqURL'),   # Get ReqURL to match /w/load.php
-('i', 'Timestamp'),# Get Timestamp to delimit requests
-('C', ''), # Use case-insensitive matching
-('I', '^(/w/load\.php'  # ...to match ResourceLoader ReqURLs
-  '|if-none-match:' # ...or If-None-Match BerespHeaders
-  '|cache-control:' # ...or Cache-control BereqHeaders
-  '|([\d.]+ ?){6}$' # ...or Timestamp lines
-  '|[1-5]\d{2}$)'), # ...or RxStatus codes
+('q', query), # VSL query
+('n', 'frontend'),# Consider the frontend Varnish instance
+('i', 'RespStatus'),  # Get RespStatus for the HTTP status code
+('i', 'ReqURL'),  # Get ReqURL to match /w/load.php
+('i', 'Timestamp'),   # Timestamp lines
+('C', ''),# Use case-insensitive matching
+('I', 'ReqHeader:if-none-match'),
+('I', 'RespHeader:cache-control'),
 ), self.handle_log_record)
 
 lp = ResourceLoaderVarnishLogProcessor()

-- 
To view, visit https://gerrit.wikimedia.org/r/316966
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4b3a5ccc8dedb5ff8b51c3a22ca9cef8efb803dc
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: beacon and CP

2016-11-01 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text varnishtest: beacon and CP
..


cache_text varnishtest: beacon and CP

Add tests for beacon endpoints and 'Connection Properties' cookie.

Bug: T131503
Change-Id: I92942de0be016e3eb8a015b3a5abc79c058c7dd2
---
A modules/varnish/files/tests/text/05-beacon.vtc
A modules/varnish/files/tests/text/06-connection-properties.vtc
2 files changed, 51 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/tests/text/05-beacon.vtc 
b/modules/varnish/files/tests/text/05-beacon.vtc
new file mode 100644
index 000..5afc6af
--- /dev/null
+++ b/modules/varnish/files/tests/text/05-beacon.vtc
@@ -0,0 +1,20 @@
+varnishtest "cache_text beacon endpoints"
+
+server s1 {
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+client c1 {
+txreq -url "/beacon/test" -hdr "Host: en.wikipedia.org" -hdr 
"X-Forwarded-Proto: https"
+rxresp
+expect resp.status == 204
+} -run
diff --git a/modules/varnish/files/tests/text/06-connection-properties.vtc 
b/modules/varnish/files/tests/text/06-connection-properties.vtc
new file mode 100644
index 000..bb273f1
--- /dev/null
+++ b/modules/varnish/files/tests/text/06-connection-properties.vtc
@@ -0,0 +1,31 @@
+varnishtest "cache_text Connection Properties"
+
+server s1 {
+rxreq
+txresp
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+client c1 {
+# Set CP ('Connection Properties') cookie
+txreq -url "/" -hdr "Host: en.wikipedia.org" -hdr "X-Forwarded-Proto: 
https" -hdr "X-Connection-Properties: H2=1"
+rxresp
+expect resp.status == 200
+expect resp.http.Set-Cookie == "CP=H2; Path=/; secure"
+
+# Explicitly unset the cookie if it exists. Support for HTTP/2 in a browser
+# session can flip if a device moves networks
+txreq -url "/" -hdr "Host: en.wikipedia.org" -hdr "X-Forwarded-Proto: 
https" -hdr "Cookie: CP=H2"
+rxresp
+expect resp.status == 200
+expect resp.http.Set-Cookie == "CP=H1; Expires=Thu, 01-Jan-1970 00:00:01 
GMT; Path=/; secure"
+} -run

-- 
To view, visit https://gerrit.wikimedia.org/r/318946
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I92942de0be016e3eb8a015b3a5abc79c058c7dd2
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: SystemTap Puppet module and role::systemtap::devserver

2016-11-01 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319083

Change subject: SystemTap Puppet module and role::systemtap::devserver
..

SystemTap Puppet module and role::systemtap::devserver

The systemtap puppet module allows configuring a SystemTap development
server as well as the runtime environment required to run compiled
SystemTap probes.

A role called systemtap::devserver is also introduced by this commit.

Change-Id: I24ad8d60a0ccc7616c33af4b0a323c39344a1444
---
A modules/role/manifests/systemtap/devserver.pp
A modules/systemtap/README.md
A modules/systemtap/manifests/devserver.pp
A modules/systemtap/manifests/runtime.pp
4 files changed, 79 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/83/319083/1

diff --git a/modules/role/manifests/systemtap/devserver.pp 
b/modules/role/manifests/systemtap/devserver.pp
new file mode 100644
index 000..7d6f969
--- /dev/null
+++ b/modules/role/manifests/systemtap/devserver.pp
@@ -0,0 +1,11 @@
+# == Class: role::systemtap::devserver
+#
+# Role to configure a SystemTap development server
+#
+class role::systemtap::devserver {
+include systemtap::devserver
+
+system::role { 'role::systemtap::devserver':
+description => 'SystemTap development environment'
+}
+}
diff --git a/modules/systemtap/README.md b/modules/systemtap/README.md
new file mode 100644
index 000..cea4d79
--- /dev/null
+++ b/modules/systemtap/README.md
@@ -0,0 +1,17 @@
+# SystemTap Puppet Module #
+
+A Puppet module for configuring SystemTap development servers and the runtime
+environment required to run compiled SystemTap probes.
+
+# Usage #
+
+To configure a development environment where SystemTap probes can be compiled,
+include systemtap::devserver. The following sample probe can be executed on the
+development server to check if the basic SystemTap functionalities are working
+fine: `stap -e 'probe oneshot { println("hello world") }'`.
+
+Production machines can be configured to run SystemTap probes by including
+systemtap::runtime.
+
+# See also #
+https://wikitech.wikimedia.org/wiki/SystemTap
diff --git a/modules/systemtap/manifests/devserver.pp 
b/modules/systemtap/manifests/devserver.pp
new file mode 100644
index 000..e0dacda
--- /dev/null
+++ b/modules/systemtap/manifests/devserver.pp
@@ -0,0 +1,31 @@
+# Class systemtap::devserver
+# Development environment for SystemTap probes
+#
+# Actions:
+#   Installs the systemtap package, the debugging symbols for the Linux kernel
+#   and the kernel headers matching the currently running kernel.
+#
+# Usage:
+#   include systemtap::devserver
+class systemtap::devserver {
+require_package([
+'build-essential',
+"linux-image-${::kernelrelease}-dbg",
+"linux-headers-${::kernelrelease}"
+])
+
+# require_package creates a dynamic intermediate class that makes declaring
+# dependencies a bit strange. Let's use package directly here.
+if !defined(Package['systemtap']) {
+package { 'systemtap':
+ensure => 'present',
+}
+}
+
+apt::pin { 'systemtap':
+package  => 'systemtap systemtap-common systemtap-runtime',
+pin  => 'release a=jessie-backports',
+priority => '1001',
+before   => Package['systemtap'],
+}
+}
diff --git a/modules/systemtap/manifests/runtime.pp 
b/modules/systemtap/manifests/runtime.pp
new file mode 100644
index 000..202de34
--- /dev/null
+++ b/modules/systemtap/manifests/runtime.pp
@@ -0,0 +1,20 @@
+# Class systemtap::runtime
+# SystemTap runtime environment
+#
+# Actions:
+#   Installs the systemtap-runtime package, necessary to run compiled SystemTap
+#   probes.
+#
+# Usage:
+#   include systemtap::runtime
+class systemtap::runtime {
+package { 'systemtap-runtime':
+ensure => 'present',
+}
+
+apt::pin { 'systemtap-runtime':
+pin  => 'release a=jessie-backports',
+priority => '1001',
+before   => Package['systemtap-runtime'],
+}
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/319083
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I24ad8d60a0ccc7616c33af4b0a323c39344a1444
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: extend Varnish targets generation to text/upload

2016-11-02 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: prometheus: extend Varnish targets generation to text/upload
..


prometheus: extend Varnish targets generation to text/upload

Bug: T147424
Change-Id: Ic4791ca393bdc60a474344232cba04b2c4e3bad0
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 10 insertions(+), 0 deletions(-)

Approvals:
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Ema: Verified; Looks good to me, approved



diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 5212298..ba451573 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -136,4 +136,14 @@
 targets_path => $targets_path,
 cache_name   => 'misc',
 }
+
+prometheus::varnish_2layer{ 'text':
+targets_path => $targets_path,
+cache_name   => 'text',
+}
+
+prometheus::varnish_2layer{ 'upload':
+targets_path => $targets_path,
+cache_name   => 'upload',
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/319293
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic4791ca393bdc60a474344232cba04b2c4e3bad0
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site: add varnish_exporter to esams/eqiad maps/misc

2016-11-02 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: site: add varnish_exporter to esams/eqiad maps/misc
..


site: add varnish_exporter to esams/eqiad maps/misc

Bug: T147424
Change-Id: I00508b0cc6b32799c84e1271fc89cdb2fef9051f
---
M manifests/site.pp
1 file changed, 4 insertions(+), 4 deletions(-)

Approvals:
  Filippo Giunchedi: Looks good to me, but someone else must approve
  Ema: Verified; Looks good to me, approved



diff --git a/manifests/site.pp b/manifests/site.pp
index f892ab2..cdf7220 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -309,12 +309,12 @@
 
 node /^cp10(45|5[18]|61)\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc, ipsec, prometheus::varnish_exporter)
 }
 
 node 'cp1046.eqiad.wmnet', 'cp1047.eqiad.wmnet', 'cp1059.eqiad.wmnet', 
'cp1060.eqiad.wmnet' {
 interface::add_ip6_mapped { 'main': }
-role(cache::maps, ipsec)
+role(cache::maps, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp10(4[89]|50|6[234]|7[1-4]|99)\.eqiad\.wmnet$/ {
@@ -349,12 +349,12 @@
 
 node /^cp300[3-6]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::maps, ipsec)
+role(cache::maps, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp30(0[789]|10)\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp301[1-4]\.esams\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/319291
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I00508b0cc6b32799c84e1271fc89cdb2fef9051f
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_: enable varnish-be weekly cron restart for all clusters

2016-11-02 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319284

Change subject: cache_: enable varnish-be weekly cron restart for all clusters
..

cache_: enable varnish-be weekly cron restart for all clusters

Install /etc/cron.d/varnish-backend-restart in r::c::base. All clusters
need it. Use hiera('cache::cluster') to get the proper list of nodes
according to the cluster name, and for cron_splay().

Bug: T149784
Change-Id: Ib0f7d25a2606d5bfefff57f4d53b2bf8f64eaacf
---
M modules/role/manifests/cache/base.pp
M modules/role/manifests/cache/upload.pp
2 files changed, 18 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/84/319284/1

diff --git a/modules/role/manifests/cache/base.pp 
b/modules/role/manifests/cache/base.pp
index a0820ad..c6f127f 100644
--- a/modules/role/manifests/cache/base.pp
+++ b/modules/role/manifests/cache/base.pp
@@ -54,6 +54,24 @@
 }
 }
 
+# XXX: temporary, we need this to mitigate T145661
+if $::realm == 'production' {
+$hnodes = hiera("cache::${cache_cluster}::nodes")
+$all_nodes = array_concat($hnodes['eqiad'], $hnodes['esams'], 
$hnodes['ulsfo'], $hnodes['codfw'])
+$times = cron_splay($all_nodes, 'weekly', 
"${cache_cluster}-backend-restarts")
+$be_restart_h = $times['hour']
+$be_restart_m = $times['minute']
+$be_restart_d = $times['weekday']
+
+file { '/etc/cron.d/varnish-backend-restart':
+mode=> '0444',
+owner   => 'root',
+group   => 'root',
+content => template('varnish/varnish-backend-restart.cron.erb'),
+require => File['/usr/local/sbin/varnish-backend-restart'],
+}
+}
+
 ###
 # Analytics/Logging stuff
 ###
diff --git a/modules/role/manifests/cache/upload.pp 
b/modules/role/manifests/cache/upload.pp
index 474d1ec..3b77876 100644
--- a/modules/role/manifests/cache/upload.pp
+++ b/modules/role/manifests/cache/upload.pp
@@ -124,22 +124,4 @@
 ::varnish::logging::media { 'media':
 statsd_server => hiera('statsd'),
 }
-
-# XXX: temporary, we need this to mitigate T145661
-if $::realm == 'production' {
-$hnodes = hiera('cache::upload::nodes')
-$all_nodes = array_concat($hnodes['eqiad'], $hnodes['esams'], 
$hnodes['ulsfo'], $hnodes['codfw'])
-$times = cron_splay($all_nodes, 'weekly', 'upload-backend-restarts')
-$be_restart_h = $times['hour']
-$be_restart_m = $times['minute']
-$be_restart_d = $times['weekday']
-
-file { '/etc/cron.d/varnish-backend-restart':
-mode=> '0444',
-owner   => 'root',
-group   => 'root',
-content => template('varnish/varnish-backend-restart.cron.erb'),
-require => File['/usr/local/sbin/varnish-backend-restart'],
-}
-}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/319284
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib0f7d25a2606d5bfefff57f4d53b2bf8f64eaacf
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site: add varnish_exporter to esams/eqiad maps/misc

2016-11-02 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319291

Change subject: site: add varnish_exporter to esams/eqiad maps/misc
..

site: add varnish_exporter to esams/eqiad maps/misc

Bug: T147424
Change-Id: I00508b0cc6b32799c84e1271fc89cdb2fef9051f
---
M manifests/site.pp
1 file changed, 4 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/91/319291/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 9b34095..3ac67cd 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -309,12 +309,12 @@
 
 node /^cp10(45|5[18]|61)\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc, ipsec, prometheus::varnish_exporter)
 }
 
 node 'cp1046.eqiad.wmnet', 'cp1047.eqiad.wmnet', 'cp1059.eqiad.wmnet', 
'cp1060.eqiad.wmnet' {
 interface::add_ip6_mapped { 'main': }
-role(cache::maps, ipsec)
+role(cache::maps, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp10(4[89]|50|6[234]|7[1-4]|99)\.eqiad\.wmnet$/ {
@@ -349,12 +349,12 @@
 
 node /^cp300[3-6]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::maps, ipsec)
+role(cache::maps, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp30(0[789]|10)\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::misc, ipsec)
+role(cache::misc, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp301[1-4]\.esams\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/319291
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I00508b0cc6b32799c84e1271fc89cdb2fef9051f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: prometheus: extend Varnish targets generation to text/upload

2016-11-02 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319293

Change subject: prometheus: extend Varnish targets generation to text/upload
..

prometheus: extend Varnish targets generation to text/upload

Bug: T147424
Change-Id: Ic4791ca393bdc60a474344232cba04b2c4e3bad0
---
M modules/role/manifests/prometheus/ops.pp
1 file changed, 10 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/93/319293/1

diff --git a/modules/role/manifests/prometheus/ops.pp 
b/modules/role/manifests/prometheus/ops.pp
index 5212298..ba451573 100644
--- a/modules/role/manifests/prometheus/ops.pp
+++ b/modules/role/manifests/prometheus/ops.pp
@@ -136,4 +136,14 @@
 targets_path => $targets_path,
 cache_name   => 'misc',
 }
+
+prometheus::varnish_2layer{ 'text':
+targets_path => $targets_path,
+cache_name   => 'text',
+}
+
+prometheus::varnish_2layer{ 'upload':
+targets_path => $targets_path,
+cache_name   => 'upload',
+}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/319293
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic4791ca393bdc60a474344232cba04b2c4e3bad0
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_: enable varnish-be weekly cron restart for all clusters

2016-11-02 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_: enable varnish-be weekly cron restart for all clusters
..


cache_: enable varnish-be weekly cron restart for all clusters

Install /etc/cron.d/varnish-backend-restart in r::c::base. All clusters
need it. Use hiera('cache::cluster') to get the proper list of nodes
according to the cluster name, and for cron_splay().

Bug: T149784
Change-Id: Ib0f7d25a2606d5bfefff57f4d53b2bf8f64eaacf
---
M modules/role/manifests/cache/base.pp
M modules/role/manifests/cache/upload.pp
2 files changed, 18 insertions(+), 18 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve



diff --git a/modules/role/manifests/cache/base.pp 
b/modules/role/manifests/cache/base.pp
index a0820ad..c6f127f 100644
--- a/modules/role/manifests/cache/base.pp
+++ b/modules/role/manifests/cache/base.pp
@@ -54,6 +54,24 @@
 }
 }
 
+# XXX: temporary, we need this to mitigate T145661
+if $::realm == 'production' {
+$hnodes = hiera("cache::${cache_cluster}::nodes")
+$all_nodes = array_concat($hnodes['eqiad'], $hnodes['esams'], 
$hnodes['ulsfo'], $hnodes['codfw'])
+$times = cron_splay($all_nodes, 'weekly', 
"${cache_cluster}-backend-restarts")
+$be_restart_h = $times['hour']
+$be_restart_m = $times['minute']
+$be_restart_d = $times['weekday']
+
+file { '/etc/cron.d/varnish-backend-restart':
+mode=> '0444',
+owner   => 'root',
+group   => 'root',
+content => template('varnish/varnish-backend-restart.cron.erb'),
+require => File['/usr/local/sbin/varnish-backend-restart'],
+}
+}
+
 ###
 # Analytics/Logging stuff
 ###
diff --git a/modules/role/manifests/cache/upload.pp 
b/modules/role/manifests/cache/upload.pp
index 474d1ec..3b77876 100644
--- a/modules/role/manifests/cache/upload.pp
+++ b/modules/role/manifests/cache/upload.pp
@@ -124,22 +124,4 @@
 ::varnish::logging::media { 'media':
 statsd_server => hiera('statsd'),
 }
-
-# XXX: temporary, we need this to mitigate T145661
-if $::realm == 'production' {
-$hnodes = hiera('cache::upload::nodes')
-$all_nodes = array_concat($hnodes['eqiad'], $hnodes['esams'], 
$hnodes['ulsfo'], $hnodes['codfw'])
-$times = cron_splay($all_nodes, 'weekly', 'upload-backend-restarts')
-$be_restart_h = $times['hour']
-$be_restart_m = $times['minute']
-$be_restart_d = $times['weekday']
-
-file { '/etc/cron.d/varnish-backend-restart':
-mode=> '0444',
-owner   => 'root',
-group   => 'root',
-content => template('varnish/varnish-backend-restart.cron.erb'),
-require => File['/usr/local/sbin/varnish-backend-restart'],
-}
-}
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/319284
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ib0f7d25a2606d5bfefff57f4d53b2bf8f64eaacf
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site: add varnish_exporter to text/upload

2016-11-02 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319295

Change subject: site: add varnish_exporter to text/upload
..

site: add varnish_exporter to text/upload

Bug: T147424
Change-Id: I69dd07393b05eb51a8124a700e7b434f93176464
---
M manifests/site.pp
1 file changed, 9 insertions(+), 9 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/95/319295/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 3ac67cd..c325564 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -303,7 +303,7 @@
 
 # cp1008: prod-like SSL test host
 node 'cp1008.wikimedia.org' {
-role(cache::text, authdns::testns)
+role(cache::text, authdns::testns, prometheus::varnish_exporter)
 interface::add_ip6_mapped { 'main': }
 }
 
@@ -319,22 +319,22 @@
 
 node /^cp10(4[89]|50|6[234]|7[1-4]|99)\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp10(5[2-5]|6[5-8])\.eqiad\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp20(0[147]|1[0369]|23)\.codfw\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp20(0[258]|1[147]|2[0246])\.codfw\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp20(0[39]|15|21)\.codfw\.wmnet$/ {
@@ -378,12 +378,12 @@
 
 node /^cp30[34][0123]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp30[34][4-9]\.esams\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload, ipsec, prometheus::varnish_exporter)
 }
 
 #
@@ -397,12 +397,12 @@
 
 node /^cp40(0[5-7]|1[3-5])\.ulsfo\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::upload, ipsec)
+role(cache::upload, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp40(0[89]|1[0678])\.ulsfo\.wmnet$/ {
 interface::add_ip6_mapped { 'main': }
-role(cache::text, ipsec)
+role(cache::text, ipsec, prometheus::varnish_exporter)
 }
 
 node /^cp40(1[129]|20)\.ulsfo\.wmnet$/ {

-- 
To view, visit https://gerrit.wikimedia.org/r/319295
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I69dd07393b05eb51a8124a700e7b434f93176464
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: SystemTap Puppet module and role::systemtap::devserver

2016-11-03 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: SystemTap Puppet module and role::systemtap::devserver
..


SystemTap Puppet module and role::systemtap::devserver

The systemtap puppet module allows configuring a SystemTap development
server as well as the runtime environment required to run compiled
SystemTap probes.

A role called systemtap::devserver is also introduced by this commit.

Change-Id: I24ad8d60a0ccc7616c33af4b0a323c39344a1444
---
A modules/role/manifests/systemtap/devserver.pp
A modules/systemtap/README.md
A modules/systemtap/manifests/devserver.pp
A modules/systemtap/manifests/runtime.pp
4 files changed, 83 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/role/manifests/systemtap/devserver.pp 
b/modules/role/manifests/systemtap/devserver.pp
new file mode 100644
index 000..7d6f969
--- /dev/null
+++ b/modules/role/manifests/systemtap/devserver.pp
@@ -0,0 +1,11 @@
+# == Class: role::systemtap::devserver
+#
+# Role to configure a SystemTap development server
+#
+class role::systemtap::devserver {
+include systemtap::devserver
+
+system::role { 'role::systemtap::devserver':
+description => 'SystemTap development environment'
+}
+}
diff --git a/modules/systemtap/README.md b/modules/systemtap/README.md
new file mode 100644
index 000..cea4d79
--- /dev/null
+++ b/modules/systemtap/README.md
@@ -0,0 +1,17 @@
+# SystemTap Puppet Module #
+
+A Puppet module for configuring SystemTap development servers and the runtime
+environment required to run compiled SystemTap probes.
+
+# Usage #
+
+To configure a development environment where SystemTap probes can be compiled,
+include systemtap::devserver. The following sample probe can be executed on the
+development server to check if the basic SystemTap functionalities are working
+fine: `stap -e 'probe oneshot { println("hello world") }'`.
+
+Production machines can be configured to run SystemTap probes by including
+systemtap::runtime.
+
+# See also #
+https://wikitech.wikimedia.org/wiki/SystemTap
diff --git a/modules/systemtap/manifests/devserver.pp 
b/modules/systemtap/manifests/devserver.pp
new file mode 100644
index 000..db873cd
--- /dev/null
+++ b/modules/systemtap/manifests/devserver.pp
@@ -0,0 +1,33 @@
+# Class systemtap::devserver
+# Development environment for SystemTap probes
+#
+# Actions:
+#   Installs the systemtap package, the debugging symbols for the Linux kernel
+#   and the kernel headers matching the currently running kernel.
+#
+# Usage:
+#   include systemtap::devserver
+class systemtap::devserver {
+require_package([
+'build-essential',
+"linux-image-${::kernelrelease}-dbg",
+"linux-headers-${::kernelrelease}"
+])
+
+# require_package creates a dynamic intermediate class that makes declaring
+# dependencies a bit strange. Let's use package directly here.
+if !defined(Package['systemtap']) {
+package { 'systemtap':
+ensure => 'present',
+}
+}
+
+if os_version('debian jessie') {
+apt::pin { 'systemtap':
+package  => 'systemtap systemtap-common systemtap-runtime',
+pin  => 'release a=jessie-backports',
+priority => '1001',
+before   => Package['systemtap'],
+}
+}
+}
diff --git a/modules/systemtap/manifests/runtime.pp 
b/modules/systemtap/manifests/runtime.pp
new file mode 100644
index 000..c52e619
--- /dev/null
+++ b/modules/systemtap/manifests/runtime.pp
@@ -0,0 +1,22 @@
+# Class systemtap::runtime
+# SystemTap runtime environment
+#
+# Actions:
+#   Installs the systemtap-runtime package, necessary to run compiled SystemTap
+#   probes.
+#
+# Usage:
+#   include systemtap::runtime
+class systemtap::runtime {
+package { 'systemtap-runtime':
+ensure => 'present',
+}
+
+if os_version('debian jessie') {
+apt::pin { 'systemtap-runtime':
+pin  => 'release a=jessie-backports',
+priority => '1001',
+before   => Package['systemtap-runtime'],
+}
+}
+}

-- 
To view, visit https://gerrit.wikimedia.org/r/319083
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I24ad8d60a0ccc7616c33af4b0a323c39344a1444
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Alexandros Kosiaris 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Faidon Liambotis 
Gerrit-Reviewer: Filippo Giunchedi 
Gerrit-Reviewer: Muehlenhoff 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: route around codfw in cache::route_table

2016-11-03 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text: route around codfw in cache::route_table
..


cache_text: route around codfw in cache::route_table

Bug: T131503
Change-Id: I9a9d4a9a3ecd4903c2af37680d68590fd9b64591
---
M hieradata/role/common/cache/text.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/hieradata/role/common/cache/text.yaml 
b/hieradata/role/common/cache/text.yaml
index ae2fe39..c9c45a6 100644
--- a/hieradata/role/common/cache/text.yaml
+++ b/hieradata/role/common/cache/text.yaml
@@ -24,5 +24,5 @@
 cache::route_table:
   eqiad: 'direct'
   codfw: 'eqiad'
-  ulsfo: 'codfw'
+  ulsfo: 'eqiad'
   esams: 'eqiad'

-- 
To view, visit https://gerrit.wikimedia.org/r/319345
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9a9d4a9a3ecd4903c2af37680d68590fd9b64591
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: route codfw straight to applayer

2016-11-03 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text: route codfw straight to applayer
..


cache_text: route codfw straight to applayer

Bug: T131503
Change-Id: I75f3941a3683f0caebc9ff29b1cc9b09d7adfbeb
---
M hieradata/role/common/cache/text.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/hieradata/role/common/cache/text.yaml 
b/hieradata/role/common/cache/text.yaml
index c9c45a6..2cc1cab 100644
--- a/hieradata/role/common/cache/text.yaml
+++ b/hieradata/role/common/cache/text.yaml
@@ -23,6 +23,6 @@
 #
 cache::route_table:
   eqiad: 'direct'
-  codfw: 'eqiad'
+  codfw: 'direct'
   ulsfo: 'eqiad'
   esams: 'eqiad'

-- 
To view, visit https://gerrit.wikimedia.org/r/319347
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I75f3941a3683f0caebc9ff29b1cc9b09d7adfbeb
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: upgrade codfw to Varnish 4

2016-11-03 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text: upgrade codfw to Varnish 4
..


cache_text: upgrade codfw to Varnish 4

Bug: T131503
Change-Id: I26014ad39fd40b06a356b77ea907a5e2e3908ab0
---
M hieradata/role/codfw/cache/text.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/hieradata/role/codfw/cache/text.yaml 
b/hieradata/role/codfw/cache/text.yaml
index 2baffc3..47982d7 100644
--- a/hieradata/role/codfw/cache/text.yaml
+++ b/hieradata/role/codfw/cache/text.yaml
@@ -1,3 +1,4 @@
 debdeploy::grains:
   debdeploy-cp-codfw-text:
 value: standard
+varnish_version4: true

-- 
To view, visit https://gerrit.wikimedia.org/r/319351
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I26014ad39fd40b06a356b77ea907a5e2e3908ab0
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish-backend-restart: workaround fallocate issues

2016-11-03 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319596

Change subject: varnish-backend-restart: workaround fallocate issues
..

varnish-backend-restart: workaround fallocate issues

Wait for the FS to catch up and realize there's enough free space for
fallocate.

Bug: T149881
Change-Id: Ib1f39f33aeeca0e6338a1c5d7fbca2bf6e763a4c
---
M modules/varnish/files/varnish-backend-restart
1 file changed, 3 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/96/319596/1

diff --git a/modules/varnish/files/varnish-backend-restart 
b/modules/varnish/files/varnish-backend-restart
index 3c59587..ca2b96b 100644
--- a/modules/varnish/files/varnish-backend-restart
+++ b/modules/varnish/files/varnish-backend-restart
@@ -12,6 +12,9 @@
 # Restart varnish and wipe storage
 /usr/sbin/service varnish stop
 rm -f /srv/sd*/varnish*
+# Wait for the FS to catch up and realize there's enough free space for
+# fallocate: T149881
+sleep 5
 /usr/sbin/service varnish start
 
 sleep 10

-- 
To view, visit https://gerrit.wikimedia.org/r/319596
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib1f39f33aeeca0e6338a1c5d7fbca2bf6e763a4c
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: switch to file storage backend on Varnish 4

2016-11-03 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319609

Change subject: cache_text: switch to file storage backend on Varnish 4
..

cache_text: switch to file storage backend on Varnish 4

On Varnish 4, the persistent storage backend is buggy (T142810) and
deprecated. Switch cache_text backends to -sfile on Varnish 4.

See T142848 for the full reasoning behind the choice of dropping
persistent storage.

Bug: T142810
Bug: T131503
Ref: https://www.varnish-cache.org/docs/trunk/phk/persistent.html
Change-Id: I3e2b0616d325d6cef4e7259d6aee2a6914f7f3dc
---
M modules/role/manifests/cache/text.pp
1 file changed, 8 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/09/319609/1

diff --git a/modules/role/manifests/cache/text.pp 
b/modules/role/manifests/cache/text.pp
index ce6f535..00a6146 100644
--- a/modules/role/manifests/cache/text.pp
+++ b/modules/role/manifests/cache/text.pp
@@ -113,6 +113,13 @@
 
 $common_runtime_params = ['default_ttl=2592000']
 
+if ($::role::cache::2layer::varnish_version4) {
+$text_storage_args = $::role::cache::base::file_storage_args
+}
+else {
+$text_storage_args = $::role::cache::base::persistent_storage_args
+}
+
 role::cache::instances { 'text':
 fe_mem_gb => ceiling(0.4 * $::memorysize_mb / 1024.0),
 fe_jemalloc_conf  => 'lg_dirty_mult:8,lg_chunk:19',
@@ -123,7 +130,7 @@
 be_vcl_config => $be_vcl_config,
 fe_extra_vcl  => ['text-common', 'zero', 'normalize_path', 
'geoip'],
 be_extra_vcl  => ['text-common', 'normalize_path'],
-be_storage=> $::role::cache::base::persistent_storage_args,
+be_storage=> $text_storage_args,
 fe_cache_be_opts  => $fe_cache_be_opts,
 be_cache_be_opts  => $be_cache_be_opts,
 cluster_nodes => hiera('cache::text::nodes'),

-- 
To view, visit https://gerrit.wikimedia.org/r/319609
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3e2b0616d325d6cef4e7259d6aee2a6914f7f3dc
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: site: apply role::systemtap::devserver to copper

2016-11-03 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319616

Change subject: site: apply role::systemtap::devserver to copper
..

site: apply role::systemtap::devserver to copper

Use copper as a SystemTap development server.

Change-Id: I74b71bf17501d5435ba1eb40de106cbd2ecf824f
---
M manifests/site.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/16/319616/1

diff --git a/manifests/site.pp b/manifests/site.pp
index 09d4eb3..117a577 100644
--- a/manifests/site.pp
+++ b/manifests/site.pp
@@ -298,7 +298,7 @@
 
 # Debian package/docker images building host in production
 node 'copper.eqiad.wmnet' {
-role(builder)
+role(builder, systemtap::devserver)
 }
 
 # cp1008: prod-like SSL test host

-- 
To view, visit https://gerrit.wikimedia.org/r/319616
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I74b71bf17501d5435ba1eb40de106cbd2ecf824f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: set X-Carrier based on XCIP

2016-10-27 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/318291

Change subject: cache_text varnishtest: set X-Carrier based on XCIP
..

cache_text varnishtest: set X-Carrier based on XCIP

Bug: T131503
Change-Id: I81a7a847071d314827e168cbe251dfa211fa9492
---
A modules/varnish/files/tests/text/04-x-carrier.vtc
1 file changed, 21 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/91/318291/1

diff --git a/modules/varnish/files/tests/text/04-x-carrier.vtc 
b/modules/varnish/files/tests/text/04-x-carrier.vtc
new file mode 100644
index 000..1cbf3ce
--- /dev/null
+++ b/modules/varnish/files/tests/text/04-x-carrier.vtc
@@ -0,0 +1,21 @@
+varnishtest "cache_text x-carrier"
+
+server s1 {
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+client c1 {
+txreq -url "/" -hdr "Host: en.wikipedia.org" -hdr "X-Forwarded-Proto: 
https" -hdr "X-Client-IP: 200.89.83.7" -hdr "X-Forwarded-For: 200.89.83.7"
+rxresp 
+expect resp.status == 200
+expect resp.http.X-Carrier == "706-02"
+} -run

-- 
To view, visit https://gerrit.wikimedia.org/r/318291
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I81a7a847071d314827e168cbe251dfa211fa9492
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: insecure POST forbidden

2016-10-27 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text varnishtest: insecure POST forbidden
..


cache_text varnishtest: insecure POST forbidden

Bug: T131503
Change-Id: I2242cf1c24648ff573ddea54a0f45b49c2c1194a
---
A modules/varnish/files/tests/text/03-insecure-post-forbidden.vtc
1 file changed, 28 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/tests/text/03-insecure-post-forbidden.vtc 
b/modules/varnish/files/tests/text/03-insecure-post-forbidden.vtc
new file mode 100644
index 000..cca2af5
--- /dev/null
+++ b/modules/varnish/files/tests/text/03-insecure-post-forbidden.vtc
@@ -0,0 +1,28 @@
+varnishtest "cache_text insecure post forbidden"
+
+server s1 {
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+client c1 {
+# Insecure GETs are redirected to secure URL
+txreq -url "/" -hdr "Host: en.wikipedia.org"
+rxresp
+expect resp.status == 301
+expect resp.http.location == "https://en.wikipedia.org/;
+
+# Insecure POSTs are forbidden
+txreq -req POST -url "/" -hdr "Host: en.wikipedia.org"
+rxresp
+expect resp.status == 403
+expect resp.body ~ "Error: 403, Insecure Request Forbidden - use HTTPS - 
https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2016-May/000110.html;
+} -run

-- 
To view, visit https://gerrit.wikimedia.org/r/318287
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2242cf1c24648ff573ddea54a0f45b49c2c1194a
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: frontend response headers

2016-10-27 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text varnishtest: frontend response headers
..


cache_text varnishtest: frontend response headers

Add 02-frontend-headers.vtc to check for cache_text response headers,
including proper generation of the GeoIP cookie.

Bug: T131503
Change-Id: I4fc0a7f5a97641b9bf33768058fea2fdba810063
---
A modules/varnish/files/tests/text/02-frontend-headers.vtc
1 file changed, 34 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/tests/text/02-frontend-headers.vtc 
b/modules/varnish/files/tests/text/02-frontend-headers.vtc
new file mode 100644
index 000..f84303b
--- /dev/null
+++ b/modules/varnish/files/tests/text/02-frontend-headers.vtc
@@ -0,0 +1,34 @@
+varnishtest "cache_text frontend headers"
+
+server s1 {
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+# We return Set-Cookie twice: the first time for WMF-Last-Access, the second
+# one for GeoIP. logexpect seems to be the only sane way to check that the
+# latter is also set.
+logexpect l1 -v v1 -g request {
+expect  0 1001  Begin   "^req .* rxreq"
+expect  * = RespHeader "Set-Cookie: GeoIP=NL:::52.38:4.90:v4"
+expect  * = End
+} -start
+
+client c1 {
+txreq -url "/" -hdr "Host: en.wikipedia.org" -hdr "X-Forwarded-Proto: 
https" -hdr "X-Client-IP: 91.198.174.192"
+rxresp
+expect resp.http.Strict-Transport-Security == "max-age=31536000; 
includeSubDomains; preload"
+expect resp.http.X-Analytics == "https=1;nocookies=1"
+expect resp.http.X-Client-IP == "91.198.174.192"
+expect resp.http.Set-Cookie ~ "WMF-Last-Access="
+} -run
+
+logexpect l1 -wait

-- 
To view, visit https://gerrit.wikimedia.org/r/318276
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4fc0a7f5a97641b9bf33768058fea2fdba810063
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: VCL: allow to load test versions of netmapper JSON files

2016-10-27 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/318320

Change subject: VCL: allow to load test versions of netmapper JSON files
..

VCL: allow to load test versions of netmapper JSON files

Do not hardcode /var/netmapper in wikimedia-frontend.vcl.erb, use a
template variable instead. Specify a different directory for the test
version of wikimedia_text-frontend.vcl to allow loading test versions of
carriers.json and proxies.json.

Change-Id: Icca24978a200ec45638dd4ff03182b3f33f23540
---
M modules/varnish/manifests/instance.pp
M modules/varnish/manifests/wikimedia_vcl.pp
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
3 files changed, 5 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/20/318320/1

diff --git a/modules/varnish/manifests/instance.pp 
b/modules/varnish/manifests/instance.pp
index 4588343..e02cbf5 100644
--- a/modules/varnish/manifests/instance.pp
+++ b/modules/varnish/manifests/instance.pp
@@ -26,6 +26,8 @@
 
 $websocket_support = hiera('cache::websocket_support', false)
 
+$netmapper_dir = '/var/netmapper'
+
 # $varnish_version4 is used to distinguish between v4 and v3 versions of
 # VCL code, as well as to pass the right parameters to varnishd. See
 # varnish.systemd.erb
diff --git a/modules/varnish/manifests/wikimedia_vcl.pp 
b/modules/varnish/manifests/wikimedia_vcl.pp
index 733c0be..a5af94d 100644
--- a/modules/varnish/manifests/wikimedia_vcl.pp
+++ b/modules/varnish/manifests/wikimedia_vcl.pp
@@ -2,6 +2,7 @@
 if $varnish_testing  {
 $varnish_include_path = '/usr/share/varnish/tests/'
 $dynamic_directors = false
+$netmapper_dir = $varnish_include_path
 }
 
 file { $title:
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index 738490e..a031f19 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -221,8 +221,8 @@
call wm_common_directors_init;
// again, netmapper only used in frontends, for recv_fe_ip_processing
// args here are map-name (for .map()), data file, and seconds between 
mtime checks for reload
-   netmapper.init("proxies", "/var/netmapper/proxies.json", 89);
-   netmapper.init("carriers", "/var/netmapper/carriers.json", 89);
+   netmapper.init("proxies", "<%= @netmapper_dir %>/proxies.json", 89);
+   netmapper.init("carriers", "<%= @netmapper_dir %>/carriers.json", 89);
/* Function vcl_init in <%= @vcl %>.inc.vcl will be appended here */
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/318320
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icca24978a200ec45638dd4ff03182b3f33f23540
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: insecure POST forbidden

2016-10-27 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/318287

Change subject: cache_text varnishtest: insecure POST forbidden
..

cache_text varnishtest: insecure POST forbidden

Bug: T131503
Change-Id: I2242cf1c24648ff573ddea54a0f45b49c2c1194a
---
A modules/varnish/files/tests/text/03-insecure-post-forbidden.vtc
1 file changed, 28 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/87/318287/1

diff --git a/modules/varnish/files/tests/text/03-insecure-post-forbidden.vtc 
b/modules/varnish/files/tests/text/03-insecure-post-forbidden.vtc
new file mode 100644
index 000..cca2af5
--- /dev/null
+++ b/modules/varnish/files/tests/text/03-insecure-post-forbidden.vtc
@@ -0,0 +1,28 @@
+varnishtest "cache_text insecure post forbidden"
+
+server s1 {
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+client c1 {
+# Insecure GETs are redirected to secure URL
+txreq -url "/" -hdr "Host: en.wikipedia.org"
+rxresp
+expect resp.status == 301
+expect resp.http.location == "https://en.wikipedia.org/;
+
+# Insecure POSTs are forbidden
+txreq -req POST -url "/" -hdr "Host: en.wikipedia.org"
+rxresp
+expect resp.status == 403
+expect resp.body ~ "Error: 403, Insecure Request Forbidden - use HTTPS - 
https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2016-May/000110.html;
+} -run

-- 
To view, visit https://gerrit.wikimedia.org/r/318287
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2242cf1c24648ff573ddea54a0f45b49c2c1194a
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishlog.py: remove trailing NULL byte only if present

2016-10-28 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/318514

Change subject: varnishlog.py: remove trailing NULL byte only if present
..

varnishlog.py: remove trailing NULL byte only if present

Instead of unconditionally removing the last element of the data string
returned by varnishapi.VarnishLog.Dispatch we should only remove NULL
bytes. Newer versions of python-varnishapi do not return a trailing NULL
byte, resulting in very broken varnishlog behavior (eg: GET -> GE).

Change-Id: I4de0f6a9b75db8dd395d810d39ab43e59e8233bf
---
M modules/varnish/files/varnishlog4.py
1 file changed, 2 insertions(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/14/318514/1

diff --git a/modules/varnish/files/varnishlog4.py 
b/modules/varnish/files/varnishlog4.py
index 68742c7..0ef53fa 100755
--- a/modules/varnish/files/varnishlog4.py
+++ b/modules/varnish/files/varnishlog4.py
@@ -129,8 +129,8 @@
 transaction_id = cbd['vxid']
 tag = vap.VSL_tags[cbd['tag']]
 
-# Remove trailing NULL byte
-value = cbd['data'][:-1]
+# Remove trailing NULL byte (if any)
+value = cbd['data'].rstrip(b'\x00')
 
 remote_type = cbd['type']
 

-- 
To view, visit https://gerrit.wikimedia.org/r/318514
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I4de0f6a9b75db8dd395d810d39ab43e59e8233bf
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Fix bashisms

2016-10-28 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: Fix bashisms
..


Fix bashisms

This commit fixes a few possible bashisms reported by checkbashisms. In
most situations we were relying on bash-specific behavior, in others on
dash-specific ones.

When in doubt, switch back the script to using bash explicitly.

Bug: T95064
Change-Id: Ifc841a52be47ca33ca8737a383ac6f7e2ed718c4
---
M modules/admin/files/home/otto/.liquidprompt/test.sh
M modules/apache/files/apache-status
M modules/mirrors/files/check_apt_mirror
M modules/mysql/templates/mysql.init.erb
M modules/puppetmaster/files/motd/99-obsolete
M modules/quarry/files/celeryd
M modules/shinken/files/init
7 files changed, 11 insertions(+), 9 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/admin/files/home/otto/.liquidprompt/test.sh 
b/modules/admin/files/home/otto/.liquidprompt/test.sh
index 1e8ed86..1694000 100755
--- a/modules/admin/files/home/otto/.liquidprompt/test.sh
+++ b/modules/admin/files/home/otto/.liquidprompt/test.sh
@@ -1,4 +1,6 @@
-#!/bin/sh
+#!/bin/bash
+# We've replaced /bin/sh with /bin/bash given that this script does not work
+# properly when /bin/sh is a symlink to /bin/dash. See T95064. -- ema
 
 
 print_ok()
diff --git a/modules/apache/files/apache-status 
b/modules/apache/files/apache-status
index 492e4e6..3d6e87a 100755
--- a/modules/apache/files/apache-status
+++ b/modules/apache/files/apache-status
@@ -3,4 +3,4 @@
 RESET="$(/usr/bin/tput sgr0)"
 file="/tmp/apache_status.$(/bin/hostname).$(/bin/date +%s).txt"
 /usr/bin/links -dump http://127.0.0.1/server-status -width 80 | tee "$file"
-echo "\nOutput saved to ${WHITE}${file}${RESET}."
+printf "\nOutput saved to ${WHITE}${file}${RESET}.\n"
diff --git a/modules/mirrors/files/check_apt_mirror 
b/modules/mirrors/files/check_apt_mirror
index 48634d3..b17ba20 100755
--- a/modules/mirrors/files/check_apt_mirror
+++ b/modules/mirrors/files/check_apt_mirror
@@ -9,8 +9,8 @@
exit 1
 fi
 
-HOSTNAME=$(hostname -f)
-STAMP=${TO}/project/trace/${HOSTNAME}
+FQDN=$(hostname -f)
+STAMP=${TO}/project/trace/${FQDN}
 WARN=6
 CRITICAL=12
 
diff --git a/modules/mysql/templates/mysql.init.erb 
b/modules/mysql/templates/mysql.init.erb
index 8aef417..d1cfba9 100755
--- a/modules/mysql/templates/mysql.init.erb
+++ b/modules/mysql/templates/mysql.init.erb
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 # Copyright Abandoned 1996 TCX DataKonsult AB & Monty Program KB & Detron HB
 # This file is public domain and comes with NO WARRANTY of any kind
 
diff --git a/modules/puppetmaster/files/motd/99-obsolete 
b/modules/puppetmaster/files/motd/99-obsolete
index 2188783..e3ff246 100755
--- a/modules/puppetmaster/files/motd/99-obsolete
+++ b/modules/puppetmaster/files/motd/99-obsolete
@@ -1,8 +1,8 @@
 #!/bin/sh
 
 echo
-echo "\033[1;31m"
+printf "\033[1;31m\n"
 echo "This system is no longer an active puppetmaster."
 echo "Everything you did here should now be done on puppetmaster1001."
-echo "\033[0m"
+printf "\033[0m\n"
 echo
diff --git a/modules/quarry/files/celeryd b/modules/quarry/files/celeryd
index 0fe704f..b1b8c0b 100644
--- a/modules/quarry/files/celeryd
+++ b/modules/quarry/files/celeryd
@@ -1,4 +1,4 @@
-#!/bin/sh -e
+#!/bin/bash -e
 # 
 #  celeryd - Starts the Celery worker daemon.
 # 
diff --git a/modules/shinken/files/init b/modules/shinken/files/init
index f8effea..5e88474 100755
--- a/modules/shinken/files/init
+++ b/modules/shinken/files/init
@@ -325,7 +325,7 @@
 }
 if [ ! -z "$pid" ]; then
 kill "$pid"
-sleep 0.5
+/bin/sleep 0.5
 ## TODO: instead of 'sleep 1': wait up to when pid file is removed 
(with timeout)?
 for i in 1 2 3
 do

-- 
To view, visit https://gerrit.wikimedia.org/r/316529
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ifc841a52be47ca33ca8737a383ac6f7e2ed718c4
Gerrit-PatchSet: 4
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Alex Monk 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Faidon Liambotis 
Gerrit-Reviewer: Giuseppe Lavagetto 
Gerrit-Reviewer: Jcrespo 
Gerrit-Reviewer: Ottomata 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishapi.py: import latest upstream version

2016-10-28 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnishapi.py: import latest upstream version
..


varnishapi.py: import latest upstream version

Update varnishapi.py to the latest version (d8b259a), which fixes a bug
affecting the -c switch.

Change-Id: I408738550f585c102c2a645c1c6e52ffc76d426c
Ref: https://github.com/xcir/python-varnishapi/issues/65
---
M modules/varnish/files/varnishapi.py
1 file changed, 284 insertions(+), 88 deletions(-)

Approvals:
  Elukey: Looks good to me, but someone else must approve
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/varnishapi.py 
b/modules/varnish/files/varnishapi.py
index 7fa71ac..3dac16e 100644
--- a/modules/varnish/files/varnishapi.py
+++ b/modules/varnish/files/varnishapi.py
@@ -26,7 +26,7 @@
 # SUCH DAMAGE.
 
 # https://github.com/xcir/python-varnishapi
-# v0.9-varnish40
+# v50.18
 
 from ctypes import *
 import getopt
@@ -256,35 +256,217 @@
 self.VSL_r__MAX = 9
 
 
-class LIBVARNISHAPI13:
+class LIBVARNISHAPI:
 
 def __init__(self, lib):
-self.VSL_CursorFile = lib.VSL_CursorFile
-self.VSL_CursorFile.restype = c_void_p
 
-self.VSL_CursorVSM = lib.VSL_CursorVSM
-self.VSL_CursorVSM.restype = c_void_p
+#LIBVARNISHAPI_1.0
+#VSM_New;
+self.VSM_New = lib.VSM_New
+self.VSM_New.restype = c_void_p
 
-self.VSL_Error = lib.VSL_Error
-self.VSL_Error.restype = c_char_p
+#VSM_Diag;
+#VSM_n_Arg;
+self.VSM_n_Arg = lib.VSM_n_Arg
+self.VSM_n_Arg.restype = c_int
+self.VSM_n_Arg.argtypes = [c_void_p, c_char_p]
 
-self.VSM_Error = lib.VSM_Error
-self.VSM_Error.restype = c_char_p
-
+#VSM_Name;
 self.VSM_Name = lib.VSM_Name
 self.VSM_Name.restype = c_char_p
+self.VSM_Name.argtypes = [c_void_p]
 
+#VSM_Delete;
+self.VSM_Delete = lib.VSM_Delete
+self.VSM_Delete.argtypes = [c_void_p]
+
+#VSM_Open;
+self.VSM_Open = lib.VSM_Open
+self.VSM_Open.restype = c_int
+self.VSM_Open.argtypes = [c_void_p]
+
+#VSM_ReOpen;
+#VSM_Seq;
+#VSM_Head;
+#VSM_Find_Chunk;
+#VSM_Close;
+#VSM_iter0;
+#VSM_intern;
+#
+#VSC_Setup;
+#VSC_Arg;
+#VSC_Open;
+#VSC_Main;
+#VSC_Iter;
+self.VSC_Iter = lib.VSC_Iter
+self.VSC_Iter.argtypes = [c_void_p, c_void_p, VSC_iter_f, c_void_p]
+
+#
+#VSL_Setup;
+#VSL_Open;
+#VSL_Arg;
+self.VSL_Arg = lib.VSL_Arg
+self.VSL_Arg.restype = c_int
+self.VSL_Arg.argtypes = [c_void_p, c_int, c_char_p]
+
+#VSL_H_Print;
+#VSL_Select;
+#VSL_NonBlocking;
+#VSL_Dispatch;
+#VSL_NextLog;
+#VSL_Matched;
+#
+#VCLI_WriteResult;
+#VCLI_ReadResult;
+#VCLI_AuthResponse;
+#
+## Variables
+#VSL_tags;
+
+#LIBVARNISHAPI_1.1
+# Functions:
+#VSL_Name2Tag;
+self.VSL_Name2Tag = lib.VSL_Name2Tag
+self.VSL_Name2Tag.restype = c_int
+self.VSL_Name2Tag.argtypes = [c_char_p, c_int]
+
+#LIBVARNISHAPI_1.2
+# Functions:
+#VSL_NextSLT;
+#VSM_Error;
+self.VSM_Error = lib.VSM_Error
+self.VSM_Error.restype = c_char_p
+self.VSM_Error.argtypes = [c_void_p]
+
+#VSM_Get;
+
+#LIBVARNISHAPI_1.3
+#VSM_Abandoned;
+#VSM_ResetError;
+self.VSM_ResetError = lib.VSM_ResetError
+self.VSM_ResetError.argtypes = [c_void_p]
+
+#VSM_StillValid;
+#VSC_Mgt;
+#VSC_LevelDesc;
+#VSL_New;
+self.VSL_New = lib.VSL_New
+self.VSL_New.restype = c_void_p
+
+#VSL_Delete;
+self.VSL_Delete = lib.VSL_Delete
+self.VSL_Delete.argtypes = [c_void_p]
+
+#VSL_Error;
+self.VSL_Error = lib.VSL_Error
+self.VSL_Error.restype = c_char_p
+self.VSL_Error.argtypes = [c_void_p]
+
+#VSL_ResetError;
+#VSL_CursorVSM;
+self.VSL_CursorVSM = lib.VSL_CursorVSM
+self.VSL_CursorVSM.restype = POINTER(VSL_cursor)
+self.VSL_CursorVSM.argtypes = [c_void_p, c_void_p, c_uint]
+
+#VSL_CursorFile;
+self.VSL_CursorFile = lib.VSL_CursorFile
+self.VSL_CursorFile.restype = POINTER(VSL_cursor)
+self.VSL_CursorFile.argtypes = [c_void_p, c_char_p, c_uint]
+
+#VSL_DeleteCursor;
+#VSL_Next;
+self.VSL_Next = lib.VSL_Next
+self.VSL_Next.restype = c_int
+self.VSL_Next.argtypes = [POINTER(VSL_cursor)]
+
+#VSL_Match;
+self.VSL_Match = lib.VSL_Match
+self.VSL_Match.restype = c_int
+self.VSL_Match.argtypes = [c_void_p, POINTER(VSL_cursor)]
+
+#VSL_Print;
+

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: set X-Carrier based on XCIP

2016-10-28 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/318519

Change subject: cache_text varnishtest: set X-Carrier based on XCIP
..

cache_text varnishtest: set X-Carrier based on XCIP

Test setting X-Carrier based on XCIP using a test version of
carriers.json.

Bug: T131503
Change-Id: I94f79faa7a4e5cdd8ad7a12e22d7c29fe0fd2f9f
---
A modules/varnish/files/tests/carriers.json
A modules/varnish/files/tests/text/04-x-carrier.vtc
2 files changed, 22 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/19/318519/1

diff --git a/modules/varnish/files/tests/carriers.json 
b/modules/varnish/files/tests/carriers.json
new file mode 100644
index 000..00e67b5
--- /dev/null
+++ b/modules/varnish/files/tests/carriers.json
@@ -0,0 +1 @@
+{"42-02": ["93.184.216.34/32"]}
diff --git a/modules/varnish/files/tests/text/04-x-carrier.vtc 
b/modules/varnish/files/tests/text/04-x-carrier.vtc
new file mode 100644
index 000..2960805
--- /dev/null
+++ b/modules/varnish/files/tests/text/04-x-carrier.vtc
@@ -0,0 +1,21 @@
+varnishtest "cache_text x-carrier"
+
+server s1 {
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+client c1 {
+txreq -url "/" -hdr "Host: en.wikipedia.org" -hdr "X-Forwarded-Proto: 
https" -hdr "X-Client-IP: 93.184.216.34" -hdr "X-Forwarded-For: 93.184.216.34"
+rxresp
+expect resp.status == 200
+expect resp.http.X-Carrier == "42-02"
+} -run

-- 
To view, visit https://gerrit.wikimedia.org/r/318519
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I94f79faa7a4e5cdd8ad7a12e22d7c29fe0fd2f9f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishlog.py: remove trailing NULL byte only if present

2016-10-28 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnishlog.py: remove trailing NULL byte only if present
..


varnishlog.py: remove trailing NULL byte only if present

Instead of unconditionally removing the last element of the data string
returned by varnishapi.VarnishLog.Dispatch we should only remove NULL
bytes. Newer versions of python-varnishapi do not return a trailing NULL
byte, resulting in very broken varnishlog behavior (eg: GET -> GE).

Change-Id: I4de0f6a9b75db8dd395d810d39ab43e59e8233bf
---
M modules/varnish/files/varnishlog4.py
1 file changed, 2 insertions(+), 2 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Volans: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/varnish/files/varnishlog4.py 
b/modules/varnish/files/varnishlog4.py
index 68742c7..0ef53fa 100755
--- a/modules/varnish/files/varnishlog4.py
+++ b/modules/varnish/files/varnishlog4.py
@@ -129,8 +129,8 @@
 transaction_id = cbd['vxid']
 tag = vap.VSL_tags[cbd['tag']]
 
-# Remove trailing NULL byte
-value = cbd['data'][:-1]
+# Remove trailing NULL byte (if any)
+value = cbd['data'].rstrip(b'\x00')
 
 remote_type = cbd['type']
 

-- 
To view, visit https://gerrit.wikimedia.org/r/318514
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4de0f6a9b75db8dd395d810d39ab43e59e8233bf
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: beacon and CP

2016-10-31 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/318946

Change subject: cache_text varnishtest: beacon and CP
..

cache_text varnishtest: beacon and CP

Add tests for beacon endpoints and 'Connection Properties' cookie.

Bug: T131503
Change-Id: I92942de0be016e3eb8a015b3a5abc79c058c7dd2
---
A modules/varnish/files/tests/text/05-beacon.vtc
A modules/varnish/files/tests/text/06-connection-properties.vtc
2 files changed, 51 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/46/318946/1

diff --git a/modules/varnish/files/tests/text/05-beacon.vtc 
b/modules/varnish/files/tests/text/05-beacon.vtc
new file mode 100644
index 000..5afc6af
--- /dev/null
+++ b/modules/varnish/files/tests/text/05-beacon.vtc
@@ -0,0 +1,20 @@
+varnishtest "cache_text beacon endpoints"
+
+server s1 {
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+client c1 {
+txreq -url "/beacon/test" -hdr "Host: en.wikipedia.org" -hdr 
"X-Forwarded-Proto: https"
+rxresp
+expect resp.status == 204
+} -run
diff --git a/modules/varnish/files/tests/text/06-connection-properties.vtc 
b/modules/varnish/files/tests/text/06-connection-properties.vtc
new file mode 100644
index 000..bb273f1
--- /dev/null
+++ b/modules/varnish/files/tests/text/06-connection-properties.vtc
@@ -0,0 +1,31 @@
+varnishtest "cache_text Connection Properties"
+
+server s1 {
+rxreq
+txresp
+rxreq
+txresp
+} -start
+
+varnish v1 -arg "-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/local/lib/ 
-o %o %s -lmaxminddb' -p vcc_allow_inline_c=true -p vcc_err_unref=false" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-frontend.vcl";
+} -start
+
+client c1 {
+# Set CP ('Connection Properties') cookie
+txreq -url "/" -hdr "Host: en.wikipedia.org" -hdr "X-Forwarded-Proto: 
https" -hdr "X-Connection-Properties: H2=1"
+rxresp
+expect resp.status == 200
+expect resp.http.Set-Cookie == "CP=H2; Path=/; secure"
+
+# Explicitly unset the cookie if it exists. Support for HTTP/2 in a browser
+# session can flip if a device moves networks
+txreq -url "/" -hdr "Host: en.wikipedia.org" -hdr "X-Forwarded-Proto: 
https" -hdr "Cookie: CP=H2"
+rxresp
+expect resp.status == 200
+expect resp.http.Set-Cookie == "CP=H1; Expires=Thu, 01-Jan-1970 00:00:01 
GMT; Path=/; secure"
+} -run

-- 
To view, visit https://gerrit.wikimedia.org/r/318946
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I92942de0be016e3eb8a015b3a5abc79c058c7dd2
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: proper caching of mangled URLs

2016-11-04 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319829

Change subject: cache_text varnishtest: proper caching of mangled URLs
..

cache_text varnishtest: proper caching of mangled URLs

Test proper caching of URLs mangled in misspass_mangle
(direct-backends).

Also add -p vcc_allow_inline_c=true to 01-w.wiki-shortener.vtc for v4
compatibility.

Bug: T131503
Change-Id: Id1b2d3ee765b42eb2aaffd2598a48a8eb63bee75
---
M modules/varnish/files/tests/text/01-w.wiki-shortener.vtc
A modules/varnish/files/tests/text/07-backend-misspass_mangle-proper-caching.vtc
2 files changed, 27 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/29/319829/1

diff --git a/modules/varnish/files/tests/text/01-w.wiki-shortener.vtc 
b/modules/varnish/files/tests/text/01-w.wiki-shortener.vtc
index 28cad3d..aa38011 100644
--- a/modules/varnish/files/tests/text/01-w.wiki-shortener.vtc
+++ b/modules/varnish/files/tests/text/01-w.wiki-shortener.vtc
@@ -6,7 +6,7 @@
 txresp
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git 
a/modules/varnish/files/tests/text/07-backend-misspass_mangle-proper-caching.vtc
 
b/modules/varnish/files/tests/text/07-backend-misspass_mangle-proper-caching.vtc
new file mode 100644
index 000..a18520d
--- /dev/null
+++ 
b/modules/varnish/files/tests/text/07-backend-misspass_mangle-proper-caching.vtc
@@ -0,0 +1,26 @@
+varnishtest "backend misspass_mangle proper caching"
+
+server s1 {
+rxreq
+# Expect proper URL rewriting by misspass_mangle
+expect req.url == 
"/wikimedia.org/v1/media/math/render/svg/d494a1039a0708e25bfeb9abb059bbefdaf2bcc3"
+txresp
+} -start
+
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-backend.vcl";
+} -start
+
+client c1 {
+txreq -url 
"/api/rest_v1/media/math/render/svg/d494a1039a0708e25bfeb9abb059bbefdaf2bcc3" 
-hdr "Host: wikimedia.org"
+rxresp
+expect resp.http.X-Cache-Int ~ "miss"
+
+txreq -url 
"/api/rest_v1/media/math/render/svg/d494a1039a0708e25bfeb9abb059bbefdaf2bcc3" 
-hdr "Host: wikimedia.org"
+rxresp
+expect resp.http.X-Cache-Int ~ "hit"
+} -run

-- 
To view, visit https://gerrit.wikimedia.org/r/319829
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id1b2d3ee765b42eb2aaffd2598a48a8eb63bee75
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: upgrade ulsfo to Varnish 4

2016-11-04 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text: upgrade ulsfo to Varnish 4
..


cache_text: upgrade ulsfo to Varnish 4

Bug: T131503
Change-Id: I0b0373218afedf44876f49031009daa2ef752bd0
---
M hieradata/role/ulsfo/cache/text.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve



diff --git a/hieradata/role/ulsfo/cache/text.yaml 
b/hieradata/role/ulsfo/cache/text.yaml
index fcaa704..1b64f25 100644
--- a/hieradata/role/ulsfo/cache/text.yaml
+++ b/hieradata/role/ulsfo/cache/text.yaml
@@ -1,3 +1,4 @@
 debdeploy::grains:
   debdeploy-cp-ulsfo-text:
 value: standard
+varnish_version4: true

-- 
To view, visit https://gerrit.wikimedia.org/r/319824
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I0b0373218afedf44876f49031009daa2ef752bd0
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text varnishtest: proper caching of mangled requests

2016-11-04 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text varnishtest: proper caching of mangled requests
..


cache_text varnishtest: proper caching of mangled requests

Test proper caching of requests mangled in misspass_mangle
(direct-backends).

Also add -p vcc_allow_inline_c=true to 01-w.wiki-shortener.vtc for v4
compatibility.

Bug: T131503
Change-Id: Id1b2d3ee765b42eb2aaffd2598a48a8eb63bee75
---
M modules/varnish/files/tests/text/01-w.wiki-shortener.vtc
A modules/varnish/files/tests/text/07-backend-misspass_mangle-proper-caching.vtc
2 files changed, 41 insertions(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/varnish/files/tests/text/01-w.wiki-shortener.vtc 
b/modules/varnish/files/tests/text/01-w.wiki-shortener.vtc
index 28cad3d..aa38011 100644
--- a/modules/varnish/files/tests/text/01-w.wiki-shortener.vtc
+++ b/modules/varnish/files/tests/text/01-w.wiki-shortener.vtc
@@ -6,7 +6,7 @@
 txresp
 } -start
 
-varnish v1 -arg "-p vcc_err_unref=false" -vcl+backend {
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
 backend vtc_backend {
 .host = "${s1_addr}"; .port = "${s1_port}";
 }
diff --git 
a/modules/varnish/files/tests/text/07-backend-misspass_mangle-proper-caching.vtc
 
b/modules/varnish/files/tests/text/07-backend-misspass_mangle-proper-caching.vtc
new file mode 100644
index 000..3aef83f
--- /dev/null
+++ 
b/modules/varnish/files/tests/text/07-backend-misspass_mangle-proper-caching.vtc
@@ -0,0 +1,40 @@
+varnishtest "backend misspass_mangle proper caching"
+
+server s1 {
+# First miss (/api/rest_v1/)
+rxreq
+# Expect proper URL rewriting by misspass_mangle
+expect req.url == 
"/wikimedia.org/v1/media/math/render/svg/d494a1039a0708e25bfeb9abb059bbefdaf2bcc3"
+txresp
+
+# Second miss (X-Subdomain)
+rxreq
+expect req.http.Host == "test.org"
+txresp
+} -start
+
+varnish v1 -arg "-p vcc_err_unref=false -p vcc_allow_inline_c=true" 
-vcl+backend {
+backend vtc_backend {
+.host = "${s1_addr}"; .port = "${s1_port}";
+}
+
+include "/usr/share/varnish/tests/wikimedia_text-backend.vcl";
+} -start
+
+client c1 {
+txreq -url 
"/api/rest_v1/media/math/render/svg/d494a1039a0708e25bfeb9abb059bbefdaf2bcc3" 
-hdr "Host: wikimedia.org"
+rxresp
+expect resp.http.X-Cache-Int ~ "miss"
+
+txreq -url 
"/api/rest_v1/media/math/render/svg/d494a1039a0708e25bfeb9abb059bbefdaf2bcc3" 
-hdr "Host: wikimedia.org"
+rxresp
+expect resp.http.X-Cache-Int ~ "hit"
+
+txreq -hdr "X-Subdomain: example.org" -hdr "x-dt-host: test.org"
+rxresp
+expect resp.http.X-Cache-Int ~ "miss"
+
+txreq -hdr "X-Subdomain: example.org" -hdr "x-dt-host: test.org"
+rxresp
+expect resp.http.X-Cache-Int ~ "hit"
+} -run

-- 
To view, visit https://gerrit.wikimedia.org/r/319829
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id1b2d3ee765b42eb2aaffd2598a48a8eb63bee75
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: route ulsfo to codfw

2016-11-04 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319823

Change subject: cache_text: route ulsfo to codfw
..

cache_text: route ulsfo to codfw

We've upgraded cache_text codfw to Varnish 4 (02118b7).

Route cache_text ulsfo to codfw before upgrading ulsfo as we've observed
multiple issues with Varnish 4 fetching from Varnish 3 in the past.

Bug: T131503
Change-Id: I3a7ec8572c6a3fa6315bd5e3b4a3c4c7e507354c
---
M hieradata/role/common/cache/text.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/23/319823/1

diff --git a/hieradata/role/common/cache/text.yaml 
b/hieradata/role/common/cache/text.yaml
index 2cc1cab..fb383e3 100644
--- a/hieradata/role/common/cache/text.yaml
+++ b/hieradata/role/common/cache/text.yaml
@@ -24,5 +24,5 @@
 cache::route_table:
   eqiad: 'direct'
   codfw: 'direct'
-  ulsfo: 'eqiad'
+  ulsfo: 'codfw'
   esams: 'eqiad'

-- 
To view, visit https://gerrit.wikimedia.org/r/319823
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I3a7ec8572c6a3fa6315bd5e3b4a3c4c7e507354c
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: upgrade ulsfo to Varnish 4

2016-11-04 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319824

Change subject: cache_text: upgrade ulsfo to Varnish 4
..

cache_text: upgrade ulsfo to Varnish 4

Bug: T131503
Change-Id: I0b0373218afedf44876f49031009daa2ef752bd0
---
M hieradata/role/ulsfo/cache/text.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/24/319824/1

diff --git a/hieradata/role/ulsfo/cache/text.yaml 
b/hieradata/role/ulsfo/cache/text.yaml
index fcaa704..1b64f25 100644
--- a/hieradata/role/ulsfo/cache/text.yaml
+++ b/hieradata/role/ulsfo/cache/text.yaml
@@ -1,3 +1,4 @@
 debdeploy::grains:
   debdeploy-cp-ulsfo-text:
 value: standard
+varnish_version4: true

-- 
To view, visit https://gerrit.wikimedia.org/r/319824
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I0b0373218afedf44876f49031009daa2ef752bd0
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: route ulsfo to codfw

2016-11-04 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text: route ulsfo to codfw
..


cache_text: route ulsfo to codfw

We've upgraded cache_text codfw to Varnish 4 (02118b7).

Route cache_text ulsfo to codfw before upgrading ulsfo as we've observed
multiple issues with Varnish 4 fetching from Varnish 3 in the past.

Bug: T131503
Change-Id: I3a7ec8572c6a3fa6315bd5e3b4a3c4c7e507354c
---
M hieradata/role/common/cache/text.yaml
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve



diff --git a/hieradata/role/common/cache/text.yaml 
b/hieradata/role/common/cache/text.yaml
index 2cc1cab..fb383e3 100644
--- a/hieradata/role/common/cache/text.yaml
+++ b/hieradata/role/common/cache/text.yaml
@@ -24,5 +24,5 @@
 cache::route_table:
   eqiad: 'direct'
   codfw: 'direct'
-  ulsfo: 'eqiad'
+  ulsfo: 'codfw'
   esams: 'eqiad'

-- 
To view, visit https://gerrit.wikimedia.org/r/319823
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I3a7ec8572c6a3fa6315bd5e3b4a3c4c7e507354c
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: route around codfw in cache::route_table

2016-11-02 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319345

Change subject: cache_text: route around codfw in cache::route_table
..

cache_text: route around codfw in cache::route_table

Bug: T131503
Change-Id: I9a9d4a9a3ecd4903c2af37680d68590fd9b64591
---
M hieradata/role/common/cache/text.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/45/319345/1

diff --git a/hieradata/role/common/cache/text.yaml 
b/hieradata/role/common/cache/text.yaml
index ae2fe39..c9c45a6 100644
--- a/hieradata/role/common/cache/text.yaml
+++ b/hieradata/role/common/cache/text.yaml
@@ -24,5 +24,5 @@
 cache::route_table:
   eqiad: 'direct'
   codfw: 'eqiad'
-  ulsfo: 'codfw'
+  ulsfo: 'eqiad'
   esams: 'eqiad'

-- 
To view, visit https://gerrit.wikimedia.org/r/319345
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9a9d4a9a3ecd4903c2af37680d68590fd9b64591
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: route codfw straight to applayer

2016-11-02 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319347

Change subject: cache_text: route codfw straight to applayer
..

cache_text: route codfw straight to applayer

Bug: T131503
Change-Id: I75f3941a3683f0caebc9ff29b1cc9b09d7adfbeb
---
M hieradata/role/common/cache/text.yaml
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/47/319347/1

diff --git a/hieradata/role/common/cache/text.yaml 
b/hieradata/role/common/cache/text.yaml
index c9c45a6..2cc1cab 100644
--- a/hieradata/role/common/cache/text.yaml
+++ b/hieradata/role/common/cache/text.yaml
@@ -23,6 +23,6 @@
 #
 cache::route_table:
   eqiad: 'direct'
-  codfw: 'eqiad'
+  codfw: 'direct'
   ulsfo: 'eqiad'
   esams: 'eqiad'

-- 
To view, visit https://gerrit.wikimedia.org/r/319347
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I75f3941a3683f0caebc9ff29b1cc9b09d7adfbeb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text: upload codfw to Varnish 4

2016-11-02 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/319351

Change subject: cache_text: upload codfw to Varnish 4
..

cache_text: upload codfw to Varnish 4

Bug: T131503
Change-Id: I26014ad39fd40b06a356b77ea907a5e2e3908ab0
---
M hieradata/role/codfw/cache/text.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/51/319351/1

diff --git a/hieradata/role/codfw/cache/text.yaml 
b/hieradata/role/codfw/cache/text.yaml
index 2baffc3..47982d7 100644
--- a/hieradata/role/codfw/cache/text.yaml
+++ b/hieradata/role/codfw/cache/text.yaml
@@ -1,3 +1,4 @@
 debdeploy::grains:
   debdeploy-cp-codfw-text:
 value: standard
+varnish_version4: true

-- 
To view, visit https://gerrit.wikimedia.org/r/319351
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I26014ad39fd40b06a356b77ea907a5e2e3908ab0
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: VCL: allow to load test versions of netmapper JSON files

2016-10-28 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: VCL: allow to load test versions of netmapper JSON files
..


VCL: allow to load test versions of netmapper JSON files

Do not hardcode /var/netmapper in wikimedia-frontend.vcl.erb, use a
template variable instead. Specify a different directory for the test
version of wikimedia_text-frontend.vcl to allow loading test versions of
carriers.json and proxies.json.

Change-Id: Icca24978a200ec45638dd4ff03182b3f33f23540
---
M modules/varnish/manifests/instance.pp
M modules/varnish/manifests/wikimedia_vcl.pp
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
3 files changed, 5 insertions(+), 2 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/manifests/instance.pp 
b/modules/varnish/manifests/instance.pp
index 4588343..e02cbf5 100644
--- a/modules/varnish/manifests/instance.pp
+++ b/modules/varnish/manifests/instance.pp
@@ -26,6 +26,8 @@
 
 $websocket_support = hiera('cache::websocket_support', false)
 
+$netmapper_dir = '/var/netmapper'
+
 # $varnish_version4 is used to distinguish between v4 and v3 versions of
 # VCL code, as well as to pass the right parameters to varnishd. See
 # varnish.systemd.erb
diff --git a/modules/varnish/manifests/wikimedia_vcl.pp 
b/modules/varnish/manifests/wikimedia_vcl.pp
index 733c0be..a5af94d 100644
--- a/modules/varnish/manifests/wikimedia_vcl.pp
+++ b/modules/varnish/manifests/wikimedia_vcl.pp
@@ -2,6 +2,7 @@
 if $varnish_testing  {
 $varnish_include_path = '/usr/share/varnish/tests/'
 $dynamic_directors = false
+$netmapper_dir = $varnish_include_path
 }
 
 file { $title:
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index 738490e..a031f19 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -221,8 +221,8 @@
call wm_common_directors_init;
// again, netmapper only used in frontends, for recv_fe_ip_processing
// args here are map-name (for .map()), data file, and seconds between 
mtime checks for reload
-   netmapper.init("proxies", "/var/netmapper/proxies.json", 89);
-   netmapper.init("carriers", "/var/netmapper/carriers.json", 89);
+   netmapper.init("proxies", "<%= @netmapper_dir %>/proxies.json", 89);
+   netmapper.init("carriers", "<%= @netmapper_dir %>/carriers.json", 89);
/* Function vcl_init in <%= @vcl %>.inc.vcl will be appended here */
 }
 

-- 
To view, visit https://gerrit.wikimedia.org/r/318320
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Icca24978a200ec45638dd4ff03182b3f33f23540
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text frontend VCL: backend_fetch vs misspass

2016-10-11 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: cache_text frontend VCL: backend_fetch vs misspass
..


cache_text frontend VCL: backend_fetch vs misspass

In Varnish 3 we call text_common_misspass_restore_cookie from vcl_miss
and vcl_pass to make sure the same code is executed before fetching
objects from the backend.

Varnish 4 introduced vcl_backend_fetch, which is called by both vcl_miss
and vcl_pass.

Somehow similarly to the changes introduced in fc3d2b4, this commit
makes our v3 VCL call text_common_misspass_restore_cookie from vcl_miss
and vcl_pass, while the v4 code only calls it from vcl_backend_fetch.

Bug: T131503
Change-Id: I9de27d0cc862567743009d6fcdf1157a2dbc4d09
---
M modules/varnish/templates/text-frontend.inc.vcl.erb
1 file changed, 7 insertions(+), 1 deletion(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/templates/text-frontend.inc.vcl.erb 
b/modules/varnish/templates/text-frontend.inc.vcl.erb
index e0d7f10..1919693 100644
--- a/modules/varnish/templates/text-frontend.inc.vcl.erb
+++ b/modules/varnish/templates/text-frontend.inc.vcl.erb
@@ -199,15 +199,21 @@
 sub cluster_fe_hit { }
 
 sub cluster_fe_miss {
+<% if not @varnish_version4 -%>
call text_common_misspass_restore_cookie;
+<% end -%>
 }
 
 sub cluster_fe_pass {
+<% if not @varnish_version4 -%>
call text_common_misspass_restore_cookie;
+<% end -%>
 }
 
 <% if @varnish_version4 -%>
-sub cluster_fe_backend_fetch { }
+sub cluster_fe_backend_fetch {
+   call text_common_misspass_restore_cookie;
+}
 <% end -%>
 
 sub cluster_fe_backend_response {

-- 
To view, visit https://gerrit.wikimedia.org/r/315219
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I9de27d0cc862567743009d6fcdf1157a2dbc4d09
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: add varnishstat dstat plugin

2016-10-12 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnish: add varnishstat dstat plugin
..


varnish: add varnishstat dstat plugin

dstat_varnishstat is a plugin for dstat allowing to display varnish
statistics in dstat.

Usage: dstat --varnishstat

Change-Id: Ib6fccf6224b3a6408d5caf46ac7be470e30fb813
---
A modules/varnish/files/dstat_varnishstat.py
M modules/varnish/manifests/common.pp
2 files changed, 125 insertions(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/dstat_varnishstat.py 
b/modules/varnish/files/dstat_varnishstat.py
new file mode 100644
index 000..e4fd716
--- /dev/null
+++ b/modules/varnish/files/dstat_varnishstat.py
@@ -0,0 +1,110 @@
+"""
+Display varnish statistics in dstat
+
+Usage: dstat --varnishstat
+
+  Copyright 2016 Emanuele Rocca 
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+"""
+
+import os
+
+counters = (
+("fe-n_object", "f_nobj"),
+("fe-n_lru_nuked", "f_nlru"),
+("fe-backend_fail", "f_bfail"),
+("be-n_object", "b_nobj"),
+("be-n_lru_nuked", "b_nlru"),
+("be-backend_fail", "b_bfail"),
+("fe-threads", "f_thr"),
+("fe-threads_created", "f_thc"),
+("be-threads", "b_thr"),
+("be-threads_created", "b_thc"),
+("fe-exp-lag", "f_exlag"),
+("be-exp-lag", "b_exlag"),
+)
+
+
+class dstat_plugin(dstat):  # noqa F821 undefined name 'dstat'
+
+def __init__(self):
+global counters
+self.name = "varnishstat"
+self.vars = [i[0] for i in counters]
+self.nick = [i[1] for i in counters]
+self.type = "d"
+
+def check(self):
+if os.system("varnishstat -1 > /dev/null") != 0:
+raise Exception("Non-zero exit code from varnishstat")
+
+def version(self):
+cmd = os.popen("""varnishstat -V 2>&1 |
+  awk 'NR==1 { print $2 }' |
+  tr -d '('
+   """)
+return cmd.readline().rstrip()
+
+def varnishstat(self, frontend=False):
+if "varnish-4" in self.version():
+cmd = ("varnishstat -1 -f MAIN.n_object -f MAIN.n_lru_nuked "
+   "-f MAIN.backend_fail -f MAIN.threads "
+   "-f MAIN.threads_created "
+   "-f MAIN.exp_mailed -f MAIN.exp_received")
+else:
+cmd = ("varnishstat -1 -f n_object -f n_lru_nuked "
+   "-f backend_fail -f n_wrk -f n_wrk_create")
+
+if frontend:
+cmd += " -n frontend"
+label = "fe"
+else:
+label = "be"
+
+data = os.popen(cmd)
+total = {}
+for line in data.readlines():
+row = line.split()
+if not row:
+continue
+
+item = "%s-%s" % (label, row[0].replace("MAIN.", ""))
+value = float(row[1])
+total[item] = value
+
+if "varnish-3" in self.version():
+total["fe-threads_created"] = total.get("fe-n_wrk_create", 0)
+total["be-threads_created"] = total.get("be-n_wrk_create", 0)
+total["fe-threads"] = total.get("fe-n_wrk", 0)
+total["be-threads"] = total.get("be-n_wrk", 0)
+
+# Expiry mailbox lag
+total["fe-exp-lag"] = total.get("fe-exp_mailed", 0) - \
+total.get("fe-exp_received", 0)
+total["be-exp-lag"] = total.get("be-exp_mailed", 0) - \
+total.get("be-exp_received", 0)
+
+return total
+
+def extract(self):
+fields = ("n_object", "n_lru_nuked", "backend_fail", "threads",
+  "threads_created", "exp-lag")
+be_values = self.varnishstat(frontend=False)
+
+fe_values = self.varnishstat(frontend=True)
+
+for field in fields:
+self.val["fe-" + field] = fe_values.get("fe-" + field, 0)
+self.val["be-" + field] = be_values.get("be-" + field, 0)
diff --git a/modules/varnish/manifests/common.pp 
b/modules/varnish/manifests/common.pp
index 7173649..0b37f8d 100644
--- a/modules/varnish/manifests/common.pp
+++ b/modules/varnish/manifests/common.pp
@@ -41,6 +41,21 @@
 source => 'puppet:///modules/varnish/varnish-frontend-restart',
 }
 
+file { '/usr/local/share/dstat':
+ensure => directory,
+owner  => 'root',
+group  => 'root',
+mode   => '0755',
+}
+
+

[MediaWiki-commits] [Gerrit] operations/puppet[production]: dstat_varnishstat: define 'counters' as class variable

2016-10-13 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/315643

Change subject: dstat_varnishstat: define 'counters' as class variable
..

dstat_varnishstat: define 'counters' as class variable

Instead of defining 'counters' global, define it as class variable.

Change-Id: I77c19aff7d8c3301407ea893ab3e0a2b8bff164d
---
M modules/varnish/files/dstat_varnishstat.py
1 file changed, 17 insertions(+), 18 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/43/315643/1

diff --git a/modules/varnish/files/dstat_varnishstat.py 
b/modules/varnish/files/dstat_varnishstat.py
index e4fd716..ecc053b 100644
--- a/modules/varnish/files/dstat_varnishstat.py
+++ b/modules/varnish/files/dstat_varnishstat.py
@@ -21,29 +21,28 @@
 
 import os
 
-counters = (
-("fe-n_object", "f_nobj"),
-("fe-n_lru_nuked", "f_nlru"),
-("fe-backend_fail", "f_bfail"),
-("be-n_object", "b_nobj"),
-("be-n_lru_nuked", "b_nlru"),
-("be-backend_fail", "b_bfail"),
-("fe-threads", "f_thr"),
-("fe-threads_created", "f_thc"),
-("be-threads", "b_thr"),
-("be-threads_created", "b_thc"),
-("fe-exp-lag", "f_exlag"),
-("be-exp-lag", "b_exlag"),
-)
-
 
 class dstat_plugin(dstat):  # noqa F821 undefined name 'dstat'
 
+COUNTERS = (
+("fe-n_object", "f_nobj"),
+("fe-n_lru_nuked", "f_nlru"),
+("fe-backend_fail", "f_bfail"),
+("be-n_object", "b_nobj"),
+("be-n_lru_nuked", "b_nlru"),
+("be-backend_fail", "b_bfail"),
+("fe-threads", "f_thr"),
+("fe-threads_created", "f_thc"),
+("be-threads", "b_thr"),
+("be-threads_created", "b_thc"),
+("fe-exp-lag", "f_exlag"),
+("be-exp-lag", "b_exlag"),
+)
+
 def __init__(self):
-global counters
 self.name = "varnishstat"
-self.vars = [i[0] for i in counters]
-self.nick = [i[1] for i in counters]
+self.vars = [i[0] for i in self.COUNTERS]
+self.nick = [i[1] for i in self.COUNTERS]
 self.type = "d"
 
 def check(self):

-- 
To view, visit https://gerrit.wikimedia.org/r/315643
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I77c19aff7d8c3301407ea893ab3e0a2b8bff164d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: dstat_varnishstat: define 'counters' as class variable

2016-10-13 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: dstat_varnishstat: define 'counters' as class variable
..


dstat_varnishstat: define 'counters' as class variable

Instead of defining 'counters' global, define it as class variable.

Change-Id: I77c19aff7d8c3301407ea893ab3e0a2b8bff164d
---
M modules/varnish/files/dstat_varnishstat.py
1 file changed, 17 insertions(+), 18 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/dstat_varnishstat.py 
b/modules/varnish/files/dstat_varnishstat.py
index e4fd716..ecc053b 100644
--- a/modules/varnish/files/dstat_varnishstat.py
+++ b/modules/varnish/files/dstat_varnishstat.py
@@ -21,29 +21,28 @@
 
 import os
 
-counters = (
-("fe-n_object", "f_nobj"),
-("fe-n_lru_nuked", "f_nlru"),
-("fe-backend_fail", "f_bfail"),
-("be-n_object", "b_nobj"),
-("be-n_lru_nuked", "b_nlru"),
-("be-backend_fail", "b_bfail"),
-("fe-threads", "f_thr"),
-("fe-threads_created", "f_thc"),
-("be-threads", "b_thr"),
-("be-threads_created", "b_thc"),
-("fe-exp-lag", "f_exlag"),
-("be-exp-lag", "b_exlag"),
-)
-
 
 class dstat_plugin(dstat):  # noqa F821 undefined name 'dstat'
 
+COUNTERS = (
+("fe-n_object", "f_nobj"),
+("fe-n_lru_nuked", "f_nlru"),
+("fe-backend_fail", "f_bfail"),
+("be-n_object", "b_nobj"),
+("be-n_lru_nuked", "b_nlru"),
+("be-backend_fail", "b_bfail"),
+("fe-threads", "f_thr"),
+("fe-threads_created", "f_thc"),
+("be-threads", "b_thr"),
+("be-threads_created", "b_thc"),
+("fe-exp-lag", "f_exlag"),
+("be-exp-lag", "b_exlag"),
+)
+
 def __init__(self):
-global counters
 self.name = "varnishstat"
-self.vars = [i[0] for i in counters]
-self.nick = [i[1] for i in counters]
+self.vars = [i[0] for i in self.COUNTERS]
+self.nick = [i[1] for i in self.COUNTERS]
 self.type = "d"
 
 def check(self):

-- 
To view, visit https://gerrit.wikimedia.org/r/315643
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I77c19aff7d8c3301407ea893ab3e0a2b8bff164d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Upgrade cp1008 to Varnish 4

2016-10-13 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: Upgrade cp1008 to Varnish 4
..


Upgrade cp1008 to Varnish 4

Bug: T131503
Change-Id: I696e9fb6aee24b7e296af41d65900542255c9ffb
---
M hieradata/hosts/cp1008.yaml
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/hieradata/hosts/cp1008.yaml b/hieradata/hosts/cp1008.yaml
index 0898300..d93343e 100644
--- a/hieradata/hosts/cp1008.yaml
+++ b/hieradata/hosts/cp1008.yaml
@@ -1,3 +1,4 @@
 debdeploy::grains:
   debdeploy-cp:
 value: canary
+varnish_version4: true

-- 
To view, visit https://gerrit.wikimedia.org/r/315662
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I696e9fb6aee24b7e296af41d65900542255c9ffb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Upgrade cp1008 to Varnish 4

2016-10-13 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/315662

Change subject: Upgrade cp1008 to Varnish 4
..

Upgrade cp1008 to Varnish 4

Bug: T131503
Change-Id: I696e9fb6aee24b7e296af41d65900542255c9ffb
---
M hieradata/hosts/cp1008.yaml
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/62/315662/1

diff --git a/hieradata/hosts/cp1008.yaml b/hieradata/hosts/cp1008.yaml
index 0898300..d93343e 100644
--- a/hieradata/hosts/cp1008.yaml
+++ b/hieradata/hosts/cp1008.yaml
@@ -1,3 +1,4 @@
 debdeploy::grains:
   debdeploy-cp:
 value: canary
+varnish_version4: true

-- 
To view, visit https://gerrit.wikimedia.org/r/315662
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I696e9fb6aee24b7e296af41d65900542255c9ffb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: cache_text frontend VCL: backend_fetch vs misspass

2016-10-11 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/315219

Change subject: cache_text frontend VCL: backend_fetch vs misspass
..

cache_text frontend VCL: backend_fetch vs misspass

In Varnish 3 we call text_common_misspass_restore_cookie from vcl_miss
and vcl_pass to make sure the same code is executed before fetching
objects from the backend.

Varnish 4 introduced vcl_backend_fetch, which is called by both vcl_miss
and vcl_pass.

Somehow similarly to the changes introduced in fc3d2b4, this commit
makes our v3 VCL call text_common_misspass_restore_cookie from vcl_miss
and vcl_pass, while the v4 code only calls it from vcl_backend_fetch.

Bug: T131503
Change-Id: I9de27d0cc862567743009d6fcdf1157a2dbc4d09
---
M modules/varnish/templates/text-frontend.inc.vcl.erb
1 file changed, 7 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/19/315219/1

diff --git a/modules/varnish/templates/text-frontend.inc.vcl.erb 
b/modules/varnish/templates/text-frontend.inc.vcl.erb
index e0d7f10..1919693 100644
--- a/modules/varnish/templates/text-frontend.inc.vcl.erb
+++ b/modules/varnish/templates/text-frontend.inc.vcl.erb
@@ -199,15 +199,21 @@
 sub cluster_fe_hit { }
 
 sub cluster_fe_miss {
+<% if not @varnish_version4 -%>
call text_common_misspass_restore_cookie;
+<% end -%>
 }
 
 sub cluster_fe_pass {
+<% if not @varnish_version4 -%>
call text_common_misspass_restore_cookie;
+<% end -%>
 }
 
 <% if @varnish_version4 -%>
-sub cluster_fe_backend_fetch { }
+sub cluster_fe_backend_fetch {
+   call text_common_misspass_restore_cookie;
+}
 <% end -%>
 
 sub cluster_fe_backend_response {

-- 
To view, visit https://gerrit.wikimedia.org/r/315219
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I9de27d0cc862567743009d6fcdf1157a2dbc4d09
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: Text VCL forward-port to Varnish 4

2016-10-12 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: Text VCL forward-port to Varnish 4
..


Text VCL forward-port to Varnish 4

Bug: T131503
Change-Id: I6789d1129aee77f5eb8b7816afeaac2c1a7a1a41
---
M modules/varnish/templates/geoip.inc.vcl.erb
M modules/varnish/templates/initscripts/varnish.systemd.erb
M modules/varnish/templates/normalize_path.inc.vcl.erb
M modules/varnish/templates/text-backend.inc.vcl.erb
M modules/varnish/templates/text-common.inc.vcl.erb
M modules/varnish/templates/text-frontend.inc.vcl.erb
6 files changed, 145 insertions(+), 33 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/varnish/templates/geoip.inc.vcl.erb 
b/modules/varnish/templates/geoip.inc.vcl.erb
index c8d97f2..72c7914 100644
--- a/modules/varnish/templates/geoip.inc.vcl.erb
+++ b/modules/varnish/templates/geoip.inc.vcl.erb
@@ -54,8 +54,14 @@
  * return value is the zero for success, non-zero for failure (input IP is
  * invalid or the encoding of it is too long).
  */
+<% if @varnish_version4 -%>
+static int geo_get_xcip(const struct vrt_ctx *ctx, char* ip) {
+const struct gethdr_s hdr = { HDR_REQ, "\014X-Client-IP:" };
+const char* client_ip = VRT_GetHdr(ctx, );
+<% else -%>
 static int geo_get_xcip(const struct sess* sp, char* ip) {
 const char* client_ip = VRT_GetHdr(sp, HDR_REQ, "\014X-Client-IP:");
+<% end -%>
 size_t len = 0;
 int rv = -1;
 
@@ -141,13 +147,22 @@
 _GEO_IDX_SIZE   = 5,
 } geo_idx_t;
 
+<% if @varnish_version4 -%>
+static void geo_out_cookie(const struct vrt_ctx *ctx, char** geo) {
+<% else -%>
 static void geo_out_cookie(struct sess* sp, char** geo) {
+<% end -%>
 char host_safe[50];
 
 // We can't set a cookie if we don't know the valid top domain, so this
 // is the case where we emit no Cookie output at all (as before) with
 // the two possible bare "return;" below
+<% if @varnish_version4 -%>
+const struct gethdr_s hdr = { HDR_REQ, "\005host:" };
+const char* host = VRT_GetHdr(ctx, );
+<% else -%>
 const char* host = VRT_GetHdr(sp, HDR_REQ, "\005host:");
+<% end -%>
 if (!host)
 return;
 const char* top_dom = geo_get_top_cookie_domain(host);
@@ -173,9 +188,16 @@
 
 // Use libvmod-header to ensure the Set-Cookie header we are adding
 // does not clobber or manipulate existing cookie headers (if any).
+<% if @varnish_version4 -%>
+const struct gethdr_s hdr_set_cookie = { HDR_RESP, "\013Set-Cookie:" };
+Vmod_header_Func.append(ctx, _set_cookie,
+out, "; Path=/; secure; Domain=.",
+host_safe, vrt_magic_string_end);
+<% else -%>
 Vmod_Func_header.append(sp, HDR_RESP, "\013Set-Cookie:",
 out, "; Path=/; secure; Domain=.",
 host_safe, vrt_magic_string_end);
+<% end -%>
 }
 
 static const char* mm_path[_GEO_IDX_SIZE][4] = {
@@ -186,7 +208,11 @@
 {"location", "longitude", NULL, NULL},
 };
 
+<% if @varnish_version4 -%>
+static void geo_xcip_output(const struct vrt_ctx *ctx) {
+<% else -%>
 static void geo_xcip_output(struct sess* sp) {
+<% end -%>
 int gai_error, mmdb_error;
 char ip[INET6_ADDRSTRLEN];
 char* geo[_GEO_IDX_SIZE];
@@ -199,7 +225,11 @@
 
 if (!mmdb)
 goto out;
+<% if @varnish_version4 -%>
+if (geo_get_xcip(ctx, ip))
+<% else -%>
 if (geo_get_xcip(sp, ip))
+<% end -%>
 goto out;
 MMDB_lookup_result_s result = MMDB_lookup_string(mmdb, ip,
 _error, _error);
@@ -232,11 +262,19 @@
 }
 
 out:
+<% if @varnish_version4 -%>
+geo_out_cookie(ctx, geo);
+<% else -%>
 geo_out_cookie(sp, geo);
+<% end -%>
 }
 }C
 
 // Emits a Set-Cookie
 sub geoip_cookie {
+<% if @varnish_version4 -%>
+C{geo_xcip_output(ctx);}C
+<% else -%>
 C{geo_xcip_output(sp);}C
+<% end -%>
 }
diff --git a/modules/varnish/templates/initscripts/varnish.systemd.erb 
b/modules/varnish/templates/initscripts/varnish.systemd.erb
index 3d2414d..c3397f6 100644
--- a/modules/varnish/templates/initscripts/varnish.systemd.erb
+++ b/modules/varnish/templates/initscripts/varnish.systemd.erb
@@ -27,6 +27,7 @@
 <% if @varnish_version4 -%>
 -p thread_pool_min=250 -p thread_pool_max=<%= @processorcount.to_i * 250 -%> 
-p thread_pool_timeout=120 \
 -p vsl_reclen=2048 \
+-p vcc_allow_inline_c=true \
 <% else -%>
 -w 250,<%= @processorcount.to_i * 250 -%>,120 \
 -p shm_reclen=2048 \
diff --git a/modules/varnish/templates/normalize_path.inc.vcl.erb 
b/modules/varnish/templates/normalize_path.inc.vcl.erb
index e145508..dfe5ed5 100644
---

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: double workspace_backend

2016-11-29 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/324103

Change subject: varnish: double workspace_backend
..

varnish: double workspace_backend

We're seeing varnish assertion errors likely due to workspace_backend
not being large enough.

Bug: T151563
Ref: https://github.com/varnishcache/varnish-cache/issues/1990
Change-Id: I97731a0577af708631be95a80aa0ccb580d4c3b3
---
M modules/varnish/templates/initscripts/varnish.systemd.erb
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/03/324103/1

diff --git a/modules/varnish/templates/initscripts/varnish.systemd.erb 
b/modules/varnish/templates/initscripts/varnish.systemd.erb
index 2fe8243..f0dba3c 100644
--- a/modules/varnish/templates/initscripts/varnish.systemd.erb
+++ b/modules/varnish/templates/initscripts/varnish.systemd.erb
@@ -22,6 +22,7 @@
 -f /etc/varnish/wikimedia_<%= @vcl -%>.vcl \
 -p thread_pool_min=250 -p thread_pool_max=<%= @processorcount.to_i * 250 -%> 
-p thread_pool_timeout=120 \
 -p vsl_reclen=2048 \
+-p workspace_backend=128k \
 -p vcc_allow_inline_c=true \
 -S /etc/varnish/secret \
 <%= @storage -%> \

-- 
To view, visit https://gerrit.wikimedia.org/r/324103
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I97731a0577af708631be95a80aa0ccb580d4c3b3
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: dstat_varnishstat: remove varnish 3 compatibility code

2016-12-10 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/326247 )

Change subject: dstat_varnishstat: remove varnish 3 compatibility code
..

dstat_varnishstat: remove varnish 3 compatibility code

Bug: T150660
Change-Id: I1ef6b555a14352cf0d82daf93877872e62ed7f44
---
M modules/varnish/files/dstat_varnishstat.py
1 file changed, 4 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/47/326247/1

diff --git a/modules/varnish/files/dstat_varnishstat.py 
b/modules/varnish/files/dstat_varnishstat.py
index ecc053b..2bfe269 100644
--- a/modules/varnish/files/dstat_varnishstat.py
+++ b/modules/varnish/files/dstat_varnishstat.py
@@ -49,22 +49,11 @@
 if os.system("varnishstat -1 > /dev/null") != 0:
 raise Exception("Non-zero exit code from varnishstat")
 
-def version(self):
-cmd = os.popen("""varnishstat -V 2>&1 |
-  awk 'NR==1 { print $2 }' |
-  tr -d '('
-   """)
-return cmd.readline().rstrip()
-
 def varnishstat(self, frontend=False):
-if "varnish-4" in self.version():
-cmd = ("varnishstat -1 -f MAIN.n_object -f MAIN.n_lru_nuked "
-   "-f MAIN.backend_fail -f MAIN.threads "
-   "-f MAIN.threads_created "
-   "-f MAIN.exp_mailed -f MAIN.exp_received")
-else:
-cmd = ("varnishstat -1 -f n_object -f n_lru_nuked "
-   "-f backend_fail -f n_wrk -f n_wrk_create")
+cmd = ("varnishstat -1 -f MAIN.n_object -f MAIN.n_lru_nuked "
+   "-f MAIN.backend_fail -f MAIN.threads "
+   "-f MAIN.threads_created "
+   "-f MAIN.exp_mailed -f MAIN.exp_received")
 
 if frontend:
 cmd += " -n frontend"
@@ -82,12 +71,6 @@
 item = "%s-%s" % (label, row[0].replace("MAIN.", ""))
 value = float(row[1])
 total[item] = value
-
-if "varnish-3" in self.version():
-total["fe-threads_created"] = total.get("fe-n_wrk_create", 0)
-total["be-threads_created"] = total.get("be-n_wrk_create", 0)
-total["fe-threads"] = total.get("fe-n_wrk", 0)
-total["be-threads"] = total.get("be-n_wrk", 0)
 
 # Expiry mailbox lag
 total["fe-exp-lag"] = total.get("fe-exp_mailed", 0) - \

-- 
To view, visit https://gerrit.wikimedia.org/r/326247
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1ef6b555a14352cf0d82daf93877872e62ed7f44
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: add varnish hitrate dstat plugin

2016-12-09 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/326132 )

Change subject: varnish: add varnish hitrate dstat plugin
..

varnish: add varnish hitrate dstat plugin

dstat_varnish_hit is a plugin for dstat allowing to display varnish
hitrate statistics in dstat.

Usage: dstat --varnish-hit

Change-Id: I40684205b4b6dd28afe62d5aae8419712fe5c29d
---
A modules/varnish/files/dstat_varnish_hit.py
1 file changed, 86 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/32/326132/1

diff --git a/modules/varnish/files/dstat_varnish_hit.py 
b/modules/varnish/files/dstat_varnish_hit.py
new file mode 100644
index 000..446669a
--- /dev/null
+++ b/modules/varnish/files/dstat_varnish_hit.py
@@ -0,0 +1,86 @@
+"""
+Display varnish hitrate in dstat
+
+Usage: dstat --varnish-hit
+
+  Copyright 2016 Emanuele Rocca 
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+"""
+
+class dstat_plugin(dstat):  # noqa F821 undefined name 'dstat'
+"""
+Display varnish cache miss/hit percentages in dstat
+"""
+def __init__(self):
+self.name = 'hitrate'
+self.vars = ('fe-hitrate', 'be-hitrate')
+self.nick = ('fe%', 'be%')
+self.type = 'd'
+#self.width = 6
+self.scale = 10
+
+# Initial values
+self.values = {
+'be': self.varnishstat(frontend=False),
+'fe': self.varnishstat(frontend=True),
+}
+
+def check(self):
+if os.system("varnishstat -1 > /dev/null") != 0:
+raise Exception, 'Non-zero exit code from varnishstat'
+
+def varnishstat(self, frontend=False):
+"""
+{'be-cache_hit': 185.61, 'be-cache_miss': 52.42, 'be-s_req': 2480.98, 
'be-s_pass': 0.21, 'be-s_synth': 2242.82}
+{'fe-s_pass': 0.71, 'fe-cache_miss': 317.95, 'fe-cache_hit': 556.74, 
'fe-s_synth': 2245.61, 'fe-s_req': 3121.01}
+"""
+cmd = "varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss"
+
+if frontend:
+cmd += " -n frontend"
+
+data = os.popen(cmd)
+total = {}
+for line in data.readlines():
+row = line.split()
+if not row:
+continue
+
+item = row[0].replace("MAIN.", "")
+# Use third column, change per second
+value = float(row[1])
+total[item] = value
+
+return total
+
+def hitrate(self, frontend=False):
+if frontend:
+label = "fe"
+else:
+label = "be"
+
+values = self.varnishstat(frontend)
+hit = values['cache_hit'] - self.values[label]['cache_hit']
+miss = values['cache_miss'] - self.values[label]['cache_miss']
+self.val[label + "-hitrate"] = 100 * (hit / (0.0001 + hit + miss))
+
+if frontend:
+self.values['fe'] = values
+else:
+self.values['be'] = values
+
+def extract(self):
+self.hitrate(frontend=False)
+self.hitrate(frontend=True)

-- 
To view, visit https://gerrit.wikimedia.org/r/326132
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I40684205b4b6dd28afe62d5aae8419712fe5c29d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: add varnish hitrate dstat plugin

2016-12-09 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/326132 )

Change subject: varnish: add varnish hitrate dstat plugin
..


varnish: add varnish hitrate dstat plugin

dstat_varnish_hit is a plugin for dstat allowing to display varnish
hitrate statistics in dstat.

Usage: dstat --varnish-hit

Change-Id: I40684205b4b6dd28afe62d5aae8419712fe5c29d
---
A modules/varnish/files/dstat_varnish_hit.py
M modules/varnish/manifests/common.pp
2 files changed, 93 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/dstat_varnish_hit.py 
b/modules/varnish/files/dstat_varnish_hit.py
new file mode 100644
index 000..94e446e
--- /dev/null
+++ b/modules/varnish/files/dstat_varnish_hit.py
@@ -0,0 +1,85 @@
+"""
+Display varnish hitrate in dstat
+
+Usage: dstat --varnish-hit
+
+  Copyright 2016 Emanuele Rocca 
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+"""
+
+
+import os
+
+
+class dstat_plugin(dstat):  # noqa F821 undefined name 'dstat'
+"""
+Display varnish cache miss/hit percentages in dstat
+"""
+def __init__(self):
+self.name = 'hitrate'
+self.vars = ('fe-hitrate', 'be-hitrate')
+self.nick = ('fe%', 'be%')
+self.type = 'd'
+self.scale = 10
+
+# Initial values
+self.values = {
+'be': self.varnishstat(frontend=False),
+'fe': self.varnishstat(frontend=True),
+}
+
+def check(self):
+if os.system("varnishstat -1 > /dev/null") != 0:
+raise Exception('Non-zero exit code from varnishstat')
+
+def varnishstat(self, frontend=False):
+cmd = "varnishstat -1 -f MAIN.cache_hit -f MAIN.cache_miss"
+
+if frontend:
+cmd += " -n frontend"
+
+data = os.popen(cmd)
+total = {}
+for line in data.readlines():
+row = line.split()
+if not row:
+continue
+
+item = row[0].replace("MAIN.", "")
+# Use third column, change per second
+value = float(row[1])
+total[item] = value
+
+return total
+
+def hitrate(self, frontend=False):
+if frontend:
+label = "fe"
+else:
+label = "be"
+
+values = self.varnishstat(frontend)
+hit = values['cache_hit'] - self.values[label]['cache_hit']
+miss = values['cache_miss'] - self.values[label]['cache_miss']
+self.val[label + "-hitrate"] = 100 * (hit / (0.0001 + hit + miss))
+
+if frontend:
+self.values['fe'] = values
+else:
+self.values['be'] = values
+
+def extract(self):
+self.hitrate(frontend=False)
+self.hitrate(frontend=True)
diff --git a/modules/varnish/manifests/common.pp 
b/modules/varnish/manifests/common.pp
index a7a10b1..033a76c 100644
--- a/modules/varnish/manifests/common.pp
+++ b/modules/varnish/manifests/common.pp
@@ -50,6 +50,14 @@
 require => File['/usr/local/share/dstat'],
 }
 
+file { '/usr/local/share/dstat/dstat_varnish_hit.py':
+owner   => 'root',
+group   => 'root',
+mode=> '0444',
+source  => 'puppet:///modules/varnish/dstat_varnish_hit.py',
+require => File['/usr/local/share/dstat'],
+}
+
 # `vlogdump` is a small tool to filter the output of varnishlog
 # See  for more.
 file { '/usr/local/bin/vlogdump':

-- 
To view, visit https://gerrit.wikimedia.org/r/326132
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I40684205b4b6dd28afe62d5aae8419712fe5c29d
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: package_builder: run shell inside the chroot on build failures

2016-12-15 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/327497 )

Change subject: package_builder: run shell inside the chroot on build failures
..

package_builder: run shell inside the chroot on build failures

By default, pbuilder removes the chroot environment used during the
build process regardless of whether the build has been successful or
not.

On failed builds, running a shell inside the chroot seems like a better
idea, so that the user can investigate the failure before wiping
everything.

Ref: 
https://pbuilder-docs.readthedocs.io/en/latest/faq.html#logging-in-to-pbuilder-to-investigate-build-failure
Change-Id: I875aadcf617b86097297f47a3328fdd18dbd5e5d
---
M modules/package_builder/manifests/pbuilder_hook.pp
A modules/package_builder/templates/C10shell.wikimedia.org.erb
2 files changed, 22 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/97/327497/1

diff --git a/modules/package_builder/manifests/pbuilder_hook.pp 
b/modules/package_builder/manifests/pbuilder_hook.pp
index 2cd78ec..9d0b8c0 100644
--- a/modules/package_builder/manifests/pbuilder_hook.pp
+++ b/modules/package_builder/manifests/pbuilder_hook.pp
@@ -13,6 +13,14 @@
 mode   => '0755',
 }
 
+file { "${basepath}/hooks/${distribution}/C10shell.wikimedia.org":
+ensure  => present,
+owner   => 'root',
+group   => 'root',
+mode=> '0555',
+content => template('package_builder/C10shell.wikimedia.org.erb'),
+}
+
 file { "${basepath}/hooks/${distribution}/D01apt.wikimedia.org":
 ensure  => present,
 owner   => 'root',
@@ -39,6 +47,7 @@
 
 
 # Dependency info
+File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/C10shell.wikimedia.org"]
 File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D01apt.wikimedia.org"]
 File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D02backports"]
 File["${basepath}/hooks/${distribution}"] -> 
File["${basepath}/hooks/${distribution}/D05localsources"]
diff --git a/modules/package_builder/templates/C10shell.wikimedia.org.erb 
b/modules/package_builder/templates/C10shell.wikimedia.org.erb
new file mode 100644
index 000..a93c7b2
--- /dev/null
+++ b/modules/package_builder/templates/C10shell.wikimedia.org.erb
@@ -0,0 +1,13 @@
+#!/bin/bash
+#
+# invoke shell if build fails.
+cd /tmp/buildd/*/debian/..
+
+echo "**"
+echo "The build has failed. We are now running a shell in the chroot"
+echo "used  for building, giving  you the chance  to investigate the"
+echo "failure. Note that leaving this shell means exiting the chroot"
+echo "itself and the chroot will be removed  immediately afterwards."
+echo "**"
+
+/bin/bash < /dev/tty > /dev/tty 2> /dev/tty

-- 
To view, visit https://gerrit.wikimedia.org/r/327497
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I875aadcf617b86097297f47a3328fdd18dbd5e5d
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishrls: port to cachestats.CacheStatsSender

2016-12-16 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/327733 )

Change subject: varnishrls: port to cachestats.CacheStatsSender
..

varnishrls: port to cachestats.CacheStatsSender

Bug: T151643
Change-Id: If791787b9d6901558e9394678c2b08030cda484f
---
M modules/varnish/files/varnishrls
1 file changed, 25 insertions(+), 32 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/33/327733/1

diff --git a/modules/varnish/files/varnishrls b/modules/varnish/files/varnishrls
index cfaf5a1..51bd5cb 100755
--- a/modules/varnish/files/varnishrls
+++ b/modules/varnish/files/varnishrls
@@ -29,53 +29,46 @@
   limitations under the License.
 
 """
-from __future__ import division
 
 import re
-import varnishlog
-import varnishprocessor
+import sys
+
+from cachestats import CacheStatsSender
 
 
-class ResourceLoaderVarnishLogProcessor(varnishprocessor.VarnishLogProcessor):
+class RlsCacheStatsSender(CacheStatsSender):
+
+cmd = ['/usr/bin/varnishncsa', '-n', 'frontend',
+   # VSL query matching ResourceLoader ReqURLs
+   '-q', 'ReqURL ~ "^/w/load.php" and ReqMethod ne "PURGE"',
+   # status code - CC - INM
+   '-F', '%s\t%{Cache-Control}o\t%{If-None-Match}i']
+
 description = 'ResourceLoader Browser Cache Hit Ratio StatsD Reporter'
 key_prefix = 'ResourceLoader'
 
-def process_transaction(self, transaction):
-"""Process a single completed transaction."""
-status_code = transaction['RespStatus']
+def gen_stats(self, record):
+status_code, cache_control_header, inm = record.split('\t')
+
+if not status_code.isdigit():
+return
+
 metric_keys = ['reqs.all', 'resps.' + status_code]
 
-if 'ReqHeader' in transaction:
+if inm != "-":
 metric_keys.append('reqs.if_none_match')
 
-cache_control_header = transaction.get('RespHeader')
 cache_control = 'no'
-if cache_control_header:
-match = re.search(r'(?<=max-age=)\d+', cache_control_header)
-if match:
-cache_control = 'short' if match.group() == '300' else 'long'
+match = re.search(r'(?<=max-age=)\d+', cache_control_header)
+if match:
+cache_control = 'short' if match.group() == '300' else 'long'
 metric_keys.append('responses.%s_cache_control.%s' %
(cache_control, status_code))
 
 for key in metric_keys:
-self.stats[key] = self.stats.get(key, 0) + 1
+s = "{0}.{1}".format(self.args.key_prefix, key)
+self.stats[s] = self.stats.get(s, 0) + 1
 
-if self.stats['reqs.all'] > 1:
-self.flush_stats()
 
-def start(self):
-# VSL query matching ResourceLoader ReqURLs
-query = 'ReqURL ~ "^/w/load.php" and ReqMethod ne "PURGE"'
-
-varnishlog.varnishlog((
-('q', query), # VSL query
-('n', 'frontend'),# Consider the frontend Varnish instance
-('i', 'RespStatus'),  # Get RespStatus for the HTTP status code
-('i', 'ReqURL'),  # Get ReqURL to match /w/load.php
-('i', 'Timestamp'),   # Timestamp lines
-('C', ''),# Use case-insensitive matching
-('I', 'ReqHeader:if-none-match'),
-('I', 'RespHeader:cache-control'),
-), self.handle_log_record)
-
-lp = ResourceLoaderVarnishLogProcessor()
+if __name__ == "__main__":
+RlsCacheStatsSender(sys.argv[1:]).main()

-- 
To view, visit https://gerrit.wikimedia.org/r/327733
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: If791787b9d6901558e9394678c2b08030cda484f
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish cachestats.py: strip final newline

2016-12-15 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/327511 )

Change subject: varnish cachestats.py: strip final newline
..


varnish cachestats.py: strip final newline

Get rid of the trailing newline before passing self.cmd's output
handle_record.

Bug: T151643
Change-Id: I91ba38dcaf6b888f82027d0e4f27d03ee4377529
---
M modules/varnish/files/cachestats.py
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Volans: Looks good to me, but someone else must approve



diff --git a/modules/varnish/files/cachestats.py 
b/modules/varnish/files/cachestats.py
index 3650765..1bf7be8 100644
--- a/modules/varnish/files/cachestats.py
+++ b/modules/varnish/files/cachestats.py
@@ -106,7 +106,7 @@
 
 try:
 while True:
-line = p.stdout.readline()
+line = p.stdout.readline().rstrip('\n')
 self.handle_record(line)
 except KeyboardInterrupt:
 os.waitpid(p.pid, 0)

-- 
To view, visit https://gerrit.wikimedia.org/r/327511
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I91ba38dcaf6b888f82027d0e4f27d03ee4377529
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish cachestats.py: make key_prefix configurable

2016-12-15 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/327504 )

Change subject: varnish cachestats.py: make key_prefix configurable
..


varnish cachestats.py: make key_prefix configurable

Bug: T151643
Change-Id: I2edffc6c766fc30075c7a6848fd5347b78b24e38
---
M modules/varnish/files/cachestats.py
M modules/varnish/files/varnishxcps
2 files changed, 4 insertions(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Volans: Looks good to me, but someone else must approve



diff --git a/modules/varnish/files/cachestats.py 
b/modules/varnish/files/cachestats.py
index fa3ee28..3650765 100644
--- a/modules/varnish/files/cachestats.py
+++ b/modules/varnish/files/cachestats.py
@@ -52,6 +52,7 @@
 
 cmd = []
 description = ''
+key_prefix = ''
 
 def __init__(self, argument_list):
 """Parse CLI arguments, initialize self.stats and statsd socket.
@@ -65,7 +66,7 @@
 ap.add_argument('--statsd-server', help='statsd server',
 type=parse_statsd_server_string, default=None)
 ap.add_argument('--key-prefix', help='metric key prefix',
-type=parse_prefix_string, default='varnish.clients')
+type=parse_prefix_string, default=self.key_prefix)
 ap.add_argument('--interval', help='send interval',
 type=int, default=30)
 self.args = ap.parse_args(argument_list)
diff --git a/modules/varnish/files/varnishxcps 
b/modules/varnish/files/varnishxcps
index c24feed..9ec94a5 100755
--- a/modules/varnish/files/varnishxcps
+++ b/modules/varnish/files/varnishxcps
@@ -42,6 +42,8 @@
 
 description = 'X-Connection-Properties StatsD reporter'
 
+key_prefix = 'varnish.clients'
+
 def __init__(self, argument_list):
 super(XcpsCacheStatsSender, self).__init__(argument_list)
 self.key_value_pairs = re.compile('([A-Z][A-Z0-9]*)=([^;]+)')

-- 
To view, visit https://gerrit.wikimedia.org/r/327504
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I2edffc6c766fc30075c7a6848fd5347b78b24e38
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishmedia: port to cachestats.CacheStatsSender

2016-12-15 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/327513 )

Change subject: varnishmedia: port to cachestats.CacheStatsSender
..

varnishmedia: port to cachestats.CacheStatsSender

Bug: T151643
Change-Id: I1e96dc12e8ab38615dc89d7c0b8b6e894f6ee103
---
M modules/varnish/files/varnishmedia
1 file changed, 18 insertions(+), 25 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/13/327513/1

diff --git a/modules/varnish/files/varnishmedia 
b/modules/varnish/files/varnishmedia
index 89728b0..ad0b0f7 100755
--- a/modules/varnish/files/varnishmedia
+++ b/modules/varnish/files/varnishmedia
@@ -14,6 +14,7 @@
 
   Copyright 2015 Ori Livneh 
   Copyright 2015 Gilles Dubuc 
+  Copyright 2016 Emanuele Rocca 
 
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
@@ -28,40 +29,32 @@
   limitations under the License.
 
 """
-from __future__ import division
 
-import re
+import sys
 
-import varnishlog
-import varnishprocessor
+from cachestats import CacheStatsSender
 
 
-class MediaVarnishLogProcessor(varnishprocessor.VarnishLogProcessor):
+class MediaCacheStatsSender(CacheStatsSender):
+
+cmd = ['/usr/bin/varnishncsa', '-n', 'frontend',
+   '-q', 'ReqMethod ne "PURGE"',
+   '-q', 'ReqURL ~ "/thumb/"',
+   # Status sent to the client
+   '-F', '%s']
+
 description = 'Media Browser Cache Hit Ratio StatsD Reporter'
+
 key_prefix = 'media.thumbnail.varnish'
 
-def process_transaction(self, transaction):
-"""Process a single completed transaction."""
-status_code = transaction.get('RespStatus')
-if status_code is None:
-return
-
+def gen_stats(self, record):
+status_code = record
 metric_keys = ['reqs.all', 'resps.' + status_code]
 
 for key in metric_keys:
-self.stats[key] = self.stats.get(key, 0) + 1
+s = "%s.%s" % (self.args.key_prefix, key)
+self.stats[s] = self.stats.get(s, 0) + 1
 
-if self.stats['reqs.all'] > 1:
-self.flush_stats()
 
-def start(self):
-varnishlog.varnishlog((
-('q', 'ReqMethod ne "PURGE"'),  # Skip PURGE requests
-('q', 'ReqURL ~ "/thumb/"'),# Only look at thumb requests
-('n', 'frontend'),  # Only frontend Varnish instance
-('i', 'RespStatus'),# RespStatus for the HTTP status 
code
-('i', 'ReqURL'),# ReqURL to delimit requests
-('i', 'Timestamp'), # Timestamp to delimit requests
-), self.handle_log_record)
-
-lp = MediaVarnishLogProcessor()
+if __name__ == "__main__":
+MediaCacheStatsSender(sys.argv[1:]).main()

-- 
To view, visit https://gerrit.wikimedia.org/r/327513
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1e96dc12e8ab38615dc89d7c0b8b6e894f6ee103
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishxcps: subscribe to cachestats.py

2016-12-15 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/327516 )

Change subject: varnishxcps: subscribe to cachestats.py
..

varnishxcps: subscribe to cachestats.py

Bug: T151643
Change-Id: I12ffb2ee3f06f49019800c81742e21c8333f8904
---
M modules/varnish/manifests/logging/xcps.pp
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/16/327516/1

diff --git a/modules/varnish/manifests/logging/xcps.pp 
b/modules/varnish/manifests/logging/xcps.pp
index dc6e166..09cbf4f 100644
--- a/modules/varnish/manifests/logging/xcps.pp
+++ b/modules/varnish/manifests/logging/xcps.pp
@@ -33,7 +33,7 @@
 strict => false,
 template_name  => 'varnishxcps',
 require=> File['/usr/local/bin/varnishxcps'],
-subscribe  => 
File['/usr/local/lib/python2.7/dist-packages/varnishlog.py'],
+subscribe  => 
File['/usr/local/lib/python2.7/dist-packages/cachestats.py'],
 service_params => {
 require => Service['varnish-frontend'],
 enable  => true,

-- 
To view, visit https://gerrit.wikimedia.org/r/327516
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I12ffb2ee3f06f49019800c81742e21c8333f8904
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishxcps: subscribe to cachestats.py

2016-12-15 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/327516 )

Change subject: varnishxcps: subscribe to cachestats.py
..


varnishxcps: subscribe to cachestats.py

Bug: T151643
Change-Id: I12ffb2ee3f06f49019800c81742e21c8333f8904
---
M modules/varnish/manifests/logging/xcps.pp
1 file changed, 1 insertion(+), 1 deletion(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/manifests/logging/xcps.pp 
b/modules/varnish/manifests/logging/xcps.pp
index dc6e166..09cbf4f 100644
--- a/modules/varnish/manifests/logging/xcps.pp
+++ b/modules/varnish/manifests/logging/xcps.pp
@@ -33,7 +33,7 @@
 strict => false,
 template_name  => 'varnishxcps',
 require=> File['/usr/local/bin/varnishxcps'],
-subscribe  => 
File['/usr/local/lib/python2.7/dist-packages/varnishlog.py'],
+subscribe  => 
File['/usr/local/lib/python2.7/dist-packages/cachestats.py'],
 service_params => {
 require => Service['varnish-frontend'],
 enable  => true,

-- 
To view, visit https://gerrit.wikimedia.org/r/327516
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I12ffb2ee3f06f49019800c81742e21c8333f8904
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishmedia: port to cachestats.CacheStatsSender

2016-12-16 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/327513 )

Change subject: varnishmedia: port to cachestats.CacheStatsSender
..


varnishmedia: port to cachestats.CacheStatsSender

Bug: T151643
Change-Id: I1e96dc12e8ab38615dc89d7c0b8b6e894f6ee103
---
M modules/varnish/files/varnishmedia
M modules/varnish/manifests/logging/media.pp
2 files changed, 22 insertions(+), 27 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Volans: Looks good to me, but someone else must approve



diff --git a/modules/varnish/files/varnishmedia 
b/modules/varnish/files/varnishmedia
index 89728b0..e443cea 100755
--- a/modules/varnish/files/varnishmedia
+++ b/modules/varnish/files/varnishmedia
@@ -14,6 +14,7 @@
 
   Copyright 2015 Ori Livneh 
   Copyright 2015 Gilles Dubuc 
+  Copyright 2016 Emanuele Rocca 
 
   Licensed under the Apache License, Version 2.0 (the "License");
   you may not use this file except in compliance with the License.
@@ -28,40 +29,34 @@
   limitations under the License.
 
 """
-from __future__ import division
 
-import re
+import sys
 
-import varnishlog
-import varnishprocessor
+from cachestats import CacheStatsSender
 
 
-class MediaVarnishLogProcessor(varnishprocessor.VarnishLogProcessor):
+class MediaCacheStatsSender(CacheStatsSender):
+
+cmd = ['/usr/bin/varnishncsa', '-n', 'frontend',
+   '-q', 'ReqMethod ne "PURGE"',
+   '-q', 'ReqURL ~ "/thumb/"',
+   # Status sent to the client
+   '-F', '%s']
+
 description = 'Media Browser Cache Hit Ratio StatsD Reporter'
+
 key_prefix = 'media.thumbnail.varnish'
 
-def process_transaction(self, transaction):
-"""Process a single completed transaction."""
-status_code = transaction.get('RespStatus')
-if status_code is None:
+def gen_stats(self, record):
+if not record.isdigit():
 return
 
-metric_keys = ['reqs.all', 'resps.' + status_code]
+status_code = record
 
-for key in metric_keys:
-self.stats[key] = self.stats.get(key, 0) + 1
+for key in ('reqs.all', 'resps.' + status_code):
+s = "{0}.{1}".format(self.args.key_prefix, key)
+self.stats[s] = self.stats.get(s, 0) + 1
 
-if self.stats['reqs.all'] > 1:
-self.flush_stats()
 
-def start(self):
-varnishlog.varnishlog((
-('q', 'ReqMethod ne "PURGE"'),  # Skip PURGE requests
-('q', 'ReqURL ~ "/thumb/"'),# Only look at thumb requests
-('n', 'frontend'),  # Only frontend Varnish instance
-('i', 'RespStatus'),# RespStatus for the HTTP status 
code
-('i', 'ReqURL'),# ReqURL to delimit requests
-('i', 'Timestamp'), # Timestamp to delimit requests
-), self.handle_log_record)
-
-lp = MediaVarnishLogProcessor()
+if __name__ == "__main__":
+MediaCacheStatsSender(sys.argv[1:]).main()
diff --git a/modules/varnish/manifests/logging/media.pp 
b/modules/varnish/manifests/logging/media.pp
index f39e0a1..4434f70 100644
--- a/modules/varnish/manifests/logging/media.pp
+++ b/modules/varnish/manifests/logging/media.pp
@@ -23,7 +23,7 @@
 owner   => 'root',
 group   => 'root',
 mode=> '0555',
-require => 
File['/usr/local/lib/python2.7/dist-packages/varnishlog.py'],
+require => 
File['/usr/local/lib/python2.7/dist-packages/cachestats.py'],
 notify  => Service['varnishmedia'],
 }
 
@@ -33,7 +33,7 @@
 strict => false,
 template_name  => 'varnishmedia',
 require=> File['/usr/local/bin/varnishmedia'],
-subscribe  => 
File['/usr/local/lib/python2.7/dist-packages/varnishlog.py'],
+subscribe  => 
File['/usr/local/lib/python2.7/dist-packages/cachestats.py'],
 service_params => {
 enable => true,
 },

-- 
To view, visit https://gerrit.wikimedia.org/r/327513
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1e96dc12e8ab38615dc89d7c0b8b6e894f6ee103
Gerrit-PatchSet: 5
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: Volans 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: dstat_varnishstat: remove varnish 3 compatibility code

2016-12-12 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/326247 )

Change subject: dstat_varnishstat: remove varnish 3 compatibility code
..


dstat_varnishstat: remove varnish 3 compatibility code

Bug: T150660
Change-Id: I1ef6b555a14352cf0d82daf93877872e62ed7f44
---
M modules/varnish/files/dstat_varnishstat.py
1 file changed, 4 insertions(+), 21 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved



diff --git a/modules/varnish/files/dstat_varnishstat.py 
b/modules/varnish/files/dstat_varnishstat.py
index ecc053b..2bfe269 100644
--- a/modules/varnish/files/dstat_varnishstat.py
+++ b/modules/varnish/files/dstat_varnishstat.py
@@ -49,22 +49,11 @@
 if os.system("varnishstat -1 > /dev/null") != 0:
 raise Exception("Non-zero exit code from varnishstat")
 
-def version(self):
-cmd = os.popen("""varnishstat -V 2>&1 |
-  awk 'NR==1 { print $2 }' |
-  tr -d '('
-   """)
-return cmd.readline().rstrip()
-
 def varnishstat(self, frontend=False):
-if "varnish-4" in self.version():
-cmd = ("varnishstat -1 -f MAIN.n_object -f MAIN.n_lru_nuked "
-   "-f MAIN.backend_fail -f MAIN.threads "
-   "-f MAIN.threads_created "
-   "-f MAIN.exp_mailed -f MAIN.exp_received")
-else:
-cmd = ("varnishstat -1 -f n_object -f n_lru_nuked "
-   "-f backend_fail -f n_wrk -f n_wrk_create")
+cmd = ("varnishstat -1 -f MAIN.n_object -f MAIN.n_lru_nuked "
+   "-f MAIN.backend_fail -f MAIN.threads "
+   "-f MAIN.threads_created "
+   "-f MAIN.exp_mailed -f MAIN.exp_received")
 
 if frontend:
 cmd += " -n frontend"
@@ -82,12 +71,6 @@
 item = "%s-%s" % (label, row[0].replace("MAIN.", ""))
 value = float(row[1])
 total[item] = value
-
-if "varnish-3" in self.version():
-total["fe-threads_created"] = total.get("fe-n_wrk_create", 0)
-total["be-threads_created"] = total.get("be-n_wrk_create", 0)
-total["fe-threads"] = total.get("fe-n_wrk", 0)
-total["be-threads"] = total.get("be-n_wrk", 0)
 
 # Expiry mailbox lag
 total["fe-exp-lag"] = total.get("fe-exp_mailed", 0) - \

-- 
To view, visit https://gerrit.wikimedia.org/r/326247
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1ef6b555a14352cf0d82daf93877872e62ed7f44
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishxcps: use varnishncsa to read log entries from the VSM

2016-12-13 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/326965 )

Change subject: varnishxcps: use varnishncsa to read log entries from the VSM
..

varnishxcps: use varnishncsa to read log entries from the VSM

varnsihlog.py and python-varnishapi are not performant enough to read
from the VSM without overruns. As an alternative approach, call
varnishncsa instead.

We define a base class called cachestats.CacheStatsSender responsible
for: i) abstracting away the statsd-related operation, which are common
to all our varnish python stats modules, and ii) calling an external
command (varnishncsa) to read the shared memory log. Another class
called XcpsCacheStatsSender is responsible for dealing with the
xcps-specific details of parsing X-Connection-Properties and generating
the stats.

Bug: T151643
Change-Id: Ied623dfbbd5004e129761287e2e566492a0ed1a7
---
A modules/varnish/files/cachestats.py
M modules/varnish/files/varnishxcps
M modules/varnish/manifests/common.pp
M modules/varnish/manifests/logging/xcps.pp
4 files changed, 148 insertions(+), 71 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/65/326965/1

diff --git a/modules/varnish/files/cachestats.py 
b/modules/varnish/files/cachestats.py
new file mode 100644
index 000..abd5980
--- /dev/null
+++ b/modules/varnish/files/cachestats.py
@@ -0,0 +1,109 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+  cachestats
+  ~~
+  cachestats.CacheStatsSender is responsible for abstracting away any
+  statsd-related operation, common to all our varnish python stats modules, as
+  well as calling an external command (eg: varnishncsa) to read from VSM.
+  Subclasses are responsible for dealing with the details of parsing the log
+  entries and generating stats.
+
+  Copyright 2016 Emanuele Rocca 
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+"""
+
+import io
+import os
+import time
+import socket
+import argparse
+import urlparse
+
+from subprocess import Popen, PIPE
+
+
+def parse_statsd_server_string(server_string):
+"""Convert statsd server string into (hostname, port) tuple."""
+parsed = urlparse.urlparse('//' + server_string)
+return parsed.hostname, parsed.port or 8125
+
+
+def parse_prefix_string(key_prefix):
+key_prefix = key_prefix.strip('.')
+if not key_prefix:
+raise ValueError('Key prefix must not be empty')
+return key_prefix
+
+
+class CacheStatsSender(object):
+
+cmd = []
+description = ''
+
+def __init__(self):
+"""Parse CLI arguments, initialize self.stats and statsd socket"""
+ap = argparse.ArgumentParser(
+description=self.description,
+epilog='If no statsd server is specified, '
+   'prints stats to stdout instead.')
+
+ap.add_argument('--statsd-server', help='statsd server',
+type=parse_statsd_server_string, default=None)
+ap.add_argument('--key-prefix', help='metric key prefix',
+type=parse_prefix_string, default='varnish.clients')
+ap.add_argument('--interval', help='send interval',
+type=int, default=30)
+self.args = ap.parse_args()
+
+self.next_pub = time.time() + self.args.interval
+
+self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+self.stats = {}
+
+def gen_stats(self, record):
+"""Update the self.stats dictionary. Implementation left to the
+subclasses"""
+raise NotImplementedError()
+
+def handle_record(self, record):
+"""Update the self.stats dictionary by calling self.gen_stats. Send
+stats to statsd at regular intervals defined by self.args.interval"""
+self.gen_stats(record)
+
+now = time.time()
+if now >= self.next_pub:
+self.next_pub = now + self.args.interval
+buf = io.BytesIO()
+while self.stats:
+metric = '%s:%s|c\n' % self.stats.popitem()
+buf.write(metric.encode('utf-8'))
+buf.seek(io.SEEK_SET)
+if self.args.statsd_server:
+self.sock.sendto(buf.read(), self.args.statsd_server)
+else:
+print(buf.read().decode('utf-8', errors='replace').rstrip())
+
+def main(self):
+"""Execute the command specified in self.cmd

[MediaWiki-commits] [Gerrit] operations...varnishkafka[debian]: Update default config file

2016-12-12 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/326426 )

Change subject: Update default config file
..

Update default config file

Get rid of configuration options for features that have been removed,
such as format.key and logline.data.copy.

Specify the VSL query to use with varnish.arg.q instead of the old
varnish.arg.m, used by Varnish 3.

Change-Id: I1776524db492af8907359bd14ad653cd6a767b5e
---
M debian/varnishkafka.conf
1 file changed, 2 insertions(+), 26 deletions(-)


  git pull 
ssh://gerrit.wikimedia.org:29418/operations/software/varnish/varnishkafka 
refs/changes/26/326426/1

diff --git a/debian/varnishkafka.conf b/debian/varnishkafka.conf
index 4235af6..474b49c 100644
--- a/debian/varnishkafka.conf
+++ b/debian/varnishkafka.conf
@@ -87,18 +87,6 @@
 #format = %{@host}l %{@seq_num!num}n %{@timestamp}t 
%{Varnish:time_firstbyte@response!num}x %{@ip}h %{Varnish:handling@handling}x 
%{@http_status}s %{@bytes_sent!num}b %{@request_method}m %{Host@host}i %{@uri}U 
%{@query}q %{Content-Type@mime_type}o %{Referer@referer}i 
%{X-Forwarded-For@x_forwarded_for}i %{User-Agent@user_agent}i 
%{Accept-Language@accept_language}i
 
 
-
-# Optional secondary formatting.
-#   'output = kafka':  The rendered 'format.key' will be provided as the
-#  Kafka message Key
-#   'output = string': Print string to stdout.
-# Supports the same formating and type as 'format' and 'format.type'.
-#
-format.key.type = string
-format.key = %{%s}t
-
-
-
 # Where to output varnish log lines:
 #  kafka  - (default) send to kafka broker
 #  stdout - just print to stdout (behave like varnishncsa)
@@ -110,18 +98,6 @@
 # Larger tags will be truncated to this size.
 # Defaults to 2048
 #tag.size.max = 2048
-
-
-# EXPERIMENTAL
-# Indicates if the log tag data read from VSL files should be copied instantly
-# when read (true). If this is set to false the data is assumed to be
-# persistent (for the duration of collecting and formatting a single request)
-# and no copies will be made, thus improving performance.
-#
-# NOTE:
-#   Must be set to true for offline files (-r file..) due to the way
-#   libvarnishapi reads its data.
-logline.data.copy = true
 
 
 # TUNING
@@ -211,8 +187,8 @@
 # #
 ###
 
-# -m tag:regex
-varnish.arg.m = RxRequest:^(?!PURGE$)
+# -q query
+varnish.arg.q = ReqMethod ne "PURGE"
 
 # Examples:
 # -C: ignore case when matching regex

-- 
To view, visit https://gerrit.wikimedia.org/r/326426
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I1776524db492af8907359bd14ad653cd6a767b5e
Gerrit-PatchSet: 1
Gerrit-Project: operations/software/varnish/varnishkafka
Gerrit-Branch: debian
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...varnishkafka[debian]: Update default config file

2016-12-12 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/326426 )

Change subject: Update default config file
..


Update default config file

- Get rid of configuration options for features that have been removed,
  such as format.key and logline.data.copy.

- Specify the VSL query to use with varnish.arg.q instead of the old
  varnish.arg.m, used by Varnish 3.

- Set output to `stdout` in order to have a working-by-default config
  file

Change-Id: I1776524db492af8907359bd14ad653cd6a767b5e
---
M debian/changelog
M debian/varnishkafka.conf
2 files changed, 16 insertions(+), 27 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Elukey: Looks good to me, but someone else must approve



diff --git a/debian/changelog b/debian/changelog
index 71df5da..e5d2cb2 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,15 @@
+varnishkafka (1.0.12-2) jessie-wikimedia; urgency=medium
+
+  * Update default config file
+- Get rid of configuration options for features that have been removed,
+  such as format.key and logline.data.copy
+- Specify the VSL query to use with varnish.arg.q instead of the old
+  varnish.arg.m, used by Varnish 3
+- Set output to `stdout` in order to have a working-by-default config
+  file
+
+ -- Emanuele Rocca   Mon, 12 Dec 2016 14:59:46 +0100
+
 varnishkafka (1.0.12-1) jessie-wikimedia; urgency=medium
 
   * New upstream release: improved resiliency during
diff --git a/debian/varnishkafka.conf b/debian/varnishkafka.conf
index 4235af6..459829a 100644
--- a/debian/varnishkafka.conf
+++ b/debian/varnishkafka.conf
@@ -87,41 +87,18 @@
 #format = %{@host}l %{@seq_num!num}n %{@timestamp}t 
%{Varnish:time_firstbyte@response!num}x %{@ip}h %{Varnish:handling@handling}x 
%{@http_status}s %{@bytes_sent!num}b %{@request_method}m %{Host@host}i %{@uri}U 
%{@query}q %{Content-Type@mime_type}o %{Referer@referer}i 
%{X-Forwarded-For@x_forwarded_for}i %{User-Agent@user_agent}i 
%{Accept-Language@accept_language}i
 
 
-
-# Optional secondary formatting.
-#   'output = kafka':  The rendered 'format.key' will be provided as the
-#  Kafka message Key
-#   'output = string': Print string to stdout.
-# Supports the same formating and type as 'format' and 'format.type'.
-#
-format.key.type = string
-format.key = %{%s}t
-
-
-
 # Where to output varnish log lines:
 #  kafka  - (default) send to kafka broker
 #  stdout - just print to stdout (behave like varnishncsa)
 #  null   - (test) collect all tags specified by format but dont output 
anything
-output = kafka
+output = stdout
+#output = kafka
 
 
 # The maximum accepted log tag size.
 # Larger tags will be truncated to this size.
 # Defaults to 2048
 #tag.size.max = 2048
-
-
-# EXPERIMENTAL
-# Indicates if the log tag data read from VSL files should be copied instantly
-# when read (true). If this is set to false the data is assumed to be
-# persistent (for the duration of collecting and formatting a single request)
-# and no copies will be made, thus improving performance.
-#
-# NOTE:
-#   Must be set to true for offline files (-r file..) due to the way
-#   libvarnishapi reads its data.
-logline.data.copy = true
 
 
 # TUNING
@@ -211,8 +188,8 @@
 # #
 ###
 
-# -m tag:regex
-varnish.arg.m = RxRequest:^(?!PURGE$)
+# -q query
+varnish.arg.q = ReqMethod ne "PURGE"
 
 # Examples:
 # -C: ignore case when matching regex

-- 
To view, visit https://gerrit.wikimedia.org/r/326426
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I1776524db492af8907359bd14ad653cd6a767b5e
Gerrit-PatchSet: 2
Gerrit-Project: operations/software/varnish/varnishkafka
Gerrit-Branch: debian
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishxcps: use varnishncsa to read log entries from the VSM

2016-12-14 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/326965 )

Change subject: varnishxcps: use varnishncsa to read log entries from the VSM
..


varnishxcps: use varnishncsa to read log entries from the VSM

varnsihlog.py and python-varnishapi are not performant enough to read
from the VSM without overruns. As an alternative approach, call
varnishncsa instead.

We define a base class called cachestats.CacheStatsSender responsible
for: i) abstracting away the statsd-related operation, which are common
to all our varnish python stats modules, and ii) calling an external
command (varnishncsa) to read the shared memory log. Another class
called XcpsCacheStatsSender is responsible for dealing with the
xcps-specific details of parsing X-Connection-Properties and generating
the stats.

Bug: T151643
Change-Id: Ied623dfbbd5004e129761287e2e566492a0ed1a7
---
A modules/varnish/files/cachestats.py
M modules/varnish/files/varnishxcps
M modules/varnish/manifests/common.pp
M modules/varnish/manifests/logging/xcps.pp
4 files changed, 151 insertions(+), 71 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/modules/varnish/files/cachestats.py 
b/modules/varnish/files/cachestats.py
new file mode 100644
index 000..fa3ee28
--- /dev/null
+++ b/modules/varnish/files/cachestats.py
@@ -0,0 +1,111 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+"""
+  cachestats
+  ~~
+  cachestats.CacheStatsSender is responsible for abstracting away any
+  statsd-related operation, common to all our varnish python stats modules, as
+  well as calling an external command (eg: varnishncsa) to read from VSM.
+  Subclasses are responsible for dealing with the details of parsing the log
+  entries and generating stats.
+
+  Copyright 2016 Emanuele Rocca 
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+
+"""
+
+import argparse
+import io
+import os
+import socket
+import time
+import urlparse
+
+from subprocess import PIPE, Popen
+
+
+def parse_statsd_server_string(server_string):
+"""Convert statsd server string into (hostname, port) tuple."""
+parsed = urlparse.urlparse('//' + server_string)
+return parsed.hostname, parsed.port or 8125
+
+
+def parse_prefix_string(key_prefix):
+key_prefix = key_prefix.strip('.')
+if not key_prefix:
+raise ValueError('Key prefix must not be empty')
+return key_prefix
+
+
+class CacheStatsSender(object):
+
+cmd = []
+description = ''
+
+def __init__(self, argument_list):
+"""Parse CLI arguments, initialize self.stats and statsd socket.
+
+argument_list is a list such as ['--foo', 'FOO', '--bar', 'BAR']"""
+ap = argparse.ArgumentParser(
+description=self.description,
+epilog='If no statsd server is specified, '
+   'prints stats to stdout instead.')
+
+ap.add_argument('--statsd-server', help='statsd server',
+type=parse_statsd_server_string, default=None)
+ap.add_argument('--key-prefix', help='metric key prefix',
+type=parse_prefix_string, default='varnish.clients')
+ap.add_argument('--interval', help='send interval',
+type=int, default=30)
+self.args = ap.parse_args(argument_list)
+
+self.next_pub = time.time() + self.args.interval
+
+self.sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+self.stats = {}
+
+def gen_stats(self, record):
+"""Update the self.stats dictionary. Implementation left to the
+subclasses"""
+raise NotImplementedError()
+
+def handle_record(self, record):
+"""Update the self.stats dictionary by calling self.gen_stats. Send
+stats to statsd at regular intervals defined by self.args.interval"""
+self.gen_stats(record)
+
+now = time.time()
+if now >= self.next_pub:
+self.next_pub = now + self.args.interval
+buf = io.BytesIO()
+while self.stats:
+metric = '%s:%s|c\n' % self.stats.popitem()
+buf.write(metric.encode('utf-8'))
+buf.seek(io.SEEK_SET)
+if self.args.statsd_server:
+self.sock.sendto(buf.read(), self.args.statsd_server)
+else:
+print(buf.read().decode('utf-8',

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish cachestats.py: make key_prefix configurable

2016-12-15 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/327504 )

Change subject: varnish cachestats.py: make key_prefix configurable
..

varnish cachestats.py: make key_prefix configurable

Bug: T151643
Change-Id: I2edffc6c766fc30075c7a6848fd5347b78b24e38
---
M modules/varnish/files/cachestats.py
M modules/varnish/files/varnishxcps
2 files changed, 4 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/04/327504/1

diff --git a/modules/varnish/files/cachestats.py 
b/modules/varnish/files/cachestats.py
index fa3ee28..3650765 100644
--- a/modules/varnish/files/cachestats.py
+++ b/modules/varnish/files/cachestats.py
@@ -52,6 +52,7 @@
 
 cmd = []
 description = ''
+key_prefix = ''
 
 def __init__(self, argument_list):
 """Parse CLI arguments, initialize self.stats and statsd socket.
@@ -65,7 +66,7 @@
 ap.add_argument('--statsd-server', help='statsd server',
 type=parse_statsd_server_string, default=None)
 ap.add_argument('--key-prefix', help='metric key prefix',
-type=parse_prefix_string, default='varnish.clients')
+type=parse_prefix_string, default=self.key_prefix)
 ap.add_argument('--interval', help='send interval',
 type=int, default=30)
 self.args = ap.parse_args(argument_list)
diff --git a/modules/varnish/files/varnishxcps 
b/modules/varnish/files/varnishxcps
index c24feed..9ec94a5 100755
--- a/modules/varnish/files/varnishxcps
+++ b/modules/varnish/files/varnishxcps
@@ -42,6 +42,8 @@
 
 description = 'X-Connection-Properties StatsD reporter'
 
+key_prefix = 'varnish.clients'
+
 def __init__(self, argument_list):
 super(XcpsCacheStatsSender, self).__init__(argument_list)
 self.key_value_pairs = re.compile('([A-Z][A-Z0-9]*)=([^;]+)')

-- 
To view, visit https://gerrit.wikimedia.org/r/327504
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I2edffc6c766fc30075c7a6848fd5347b78b24e38
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish cachestats.py: strip final newline

2016-12-15 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/327511 )

Change subject: varnish cachestats.py: strip final newline
..

varnish cachestats.py: strip final newline

Get rid of the trailing newline before passing self.cmd's output
handle_record.

Bug: T151643
Change-Id: I91ba38dcaf6b888f82027d0e4f27d03ee4377529
---
M modules/varnish/files/cachestats.py
1 file changed, 1 insertion(+), 1 deletion(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/11/327511/1

diff --git a/modules/varnish/files/cachestats.py 
b/modules/varnish/files/cachestats.py
index 3650765..1bf7be8 100644
--- a/modules/varnish/files/cachestats.py
+++ b/modules/varnish/files/cachestats.py
@@ -106,7 +106,7 @@
 
 try:
 while True:
-line = p.stdout.readline()
+line = p.stdout.readline().rstrip('\n')
 self.handle_record(line)
 except KeyboardInterrupt:
 os.waitpid(p.pid, 0)

-- 
To view, visit https://gerrit.wikimedia.org/r/327511
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I91ba38dcaf6b888f82027d0e4f27d03ee4377529
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: debug_proxy: bump proxy_read_timeout

2016-12-14 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/327204 )

Change subject: debug_proxy: bump proxy_read_timeout
..

debug_proxy: bump proxy_read_timeout

The default proxy_read_timeout of 60s has been a source of confusion
while debugging T152895. Setting a default timeout of 80s in the
application made debug_proxy's nginx return 504 errors. Bump the value
to 180s, which is what we use in tlsproxy.

Bug: T152895
Change-Id: I095ab4f0dcd8ff99ea843237253db75862096811
---
M modules/debug_proxy/templates/debug_proxy.nginx.erb
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/04/327204/1

diff --git a/modules/debug_proxy/templates/debug_proxy.nginx.erb 
b/modules/debug_proxy/templates/debug_proxy.nginx.erb
index 7206571..65822ea 100644
--- a/modules/debug_proxy/templates/debug_proxy.nginx.erb
+++ b/modules/debug_proxy/templates/debug_proxy.nginx.erb
@@ -40,5 +40,6 @@
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $host;
 proxy_pass_header Server;
+proxy_read_timeout 180s;
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/327204
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: I095ab4f0dcd8ff99ea843237253db75862096811
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: debug_proxy: bump proxy_read_timeout

2016-12-14 Thread Ema (Code Review)

Ema has submitted this change and it was merged. ( 
https://gerrit.wikimedia.org/r/327204 )

Change subject: debug_proxy: bump proxy_read_timeout
..


debug_proxy: bump proxy_read_timeout

The default proxy_read_timeout of 60s has been a source of confusion
while debugging T152895. Setting a default timeout of 80s in the
application made debug_proxy's nginx return 504 errors. Bump the value
to 180s, which is what we use in tlsproxy.

Bug: T152895
Change-Id: I095ab4f0dcd8ff99ea843237253db75862096811
---
M modules/debug_proxy/templates/debug_proxy.nginx.erb
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  Elukey: Looks good to me, but someone else must approve



diff --git a/modules/debug_proxy/templates/debug_proxy.nginx.erb 
b/modules/debug_proxy/templates/debug_proxy.nginx.erb
index 7206571..65822ea 100644
--- a/modules/debug_proxy/templates/debug_proxy.nginx.erb
+++ b/modules/debug_proxy/templates/debug_proxy.nginx.erb
@@ -40,5 +40,6 @@
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header Host $host;
 proxy_pass_header Server;
+proxy_read_timeout 180s;
 }
 }

-- 
To view, visit https://gerrit.wikimedia.org/r/327204
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I095ab4f0dcd8ff99ea843237253db75862096811
Gerrit-PatchSet: 3
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: DCausse 
Gerrit-Reviewer: Elukey 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: double workspace_backend

2016-11-29 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnish: double workspace_backend
..


varnish: double workspace_backend

We're seeing varnish assertion errors likely due to workspace_backend
not being large enough.

Bug: T151563
Ref: https://github.com/varnishcache/varnish-cache/issues/1990
Change-Id: I97731a0577af708631be95a80aa0ccb580d4c3b3
---
M modules/varnish/templates/initscripts/varnish.systemd.erb
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Verified; Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve



diff --git a/modules/varnish/templates/initscripts/varnish.systemd.erb 
b/modules/varnish/templates/initscripts/varnish.systemd.erb
index 2fe8243..f0dba3c 100644
--- a/modules/varnish/templates/initscripts/varnish.systemd.erb
+++ b/modules/varnish/templates/initscripts/varnish.systemd.erb
@@ -22,6 +22,7 @@
 -f /etc/varnish/wikimedia_<%= @vcl -%>.vcl \
 -p thread_pool_min=250 -p thread_pool_max=<%= @processorcount.to_i * 250 -%> 
-p thread_pool_timeout=120 \
 -p vsl_reclen=2048 \
+-p workspace_backend=128k \
 -p vcc_allow_inline_c=true \
 -S /etc/varnish/secret \
 <%= @storage -%> \

-- 
To view, visit https://gerrit.wikimedia.org/r/324103
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I97731a0577af708631be95a80aa0ccb580d4c3b3
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: disable extrachance

2016-11-29 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/324229

Change subject: varnish: disable extrachance
..

varnish: disable extrachance

Bug: T150247
Change-Id: Ic6f66e8f02c3d3f3b0a894c49c91af559742eb49
---
M modules/varnish/templates/initscripts/varnish.systemd.erb
1 file changed, 1 insertion(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/29/324229/1

diff --git a/modules/varnish/templates/initscripts/varnish.systemd.erb 
b/modules/varnish/templates/initscripts/varnish.systemd.erb
index f0dba3c..07368bc 100644
--- a/modules/varnish/templates/initscripts/varnish.systemd.erb
+++ b/modules/varnish/templates/initscripts/varnish.systemd.erb
@@ -29,6 +29,7 @@
 -p thread_pool_stack=131072 \
 -p listen_depth=4096 -p session_max=20 -p vcc_err_unref=off \
 -p http_req_size=24576 \
+-p gethdr_extrachance=0 \
 -p gzip_level=8 \
 -p gzip_memlevel=9 \
 <%= @runtime_params -%> \

-- 
To view, visit https://gerrit.wikimedia.org/r/324229
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic6f66e8f02c3d3f3b0a894c49c91af559742eb49
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnish: disable extrachance

2016-11-29 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: varnish: disable extrachance
..


varnish: disable extrachance

Bug: T150247
Change-Id: Ic6f66e8f02c3d3f3b0a894c49c91af559742eb49
---
M modules/varnish/templates/initscripts/varnish.systemd.erb
1 file changed, 1 insertion(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  BBlack: Looks good to me, but someone else must approve
  jenkins-bot: Verified



diff --git a/modules/varnish/templates/initscripts/varnish.systemd.erb 
b/modules/varnish/templates/initscripts/varnish.systemd.erb
index f0dba3c..07368bc 100644
--- a/modules/varnish/templates/initscripts/varnish.systemd.erb
+++ b/modules/varnish/templates/initscripts/varnish.systemd.erb
@@ -29,6 +29,7 @@
 -p thread_pool_stack=131072 \
 -p listen_depth=4096 -p session_max=20 -p vcc_err_unref=off \
 -p http_req_size=24576 \
+-p gethdr_extrachance=0 \
 -p gzip_level=8 \
 -p gzip_memlevel=9 \
 <%= @runtime_params -%> \

-- 
To view, visit https://gerrit.wikimedia.org/r/324229
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Ic6f66e8f02c3d3f3b0a894c49c91af559742eb49
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 
Gerrit-Reviewer: BBlack 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...varnish4[debian-wmf]: Release 4.1.4-1wm1

2016-12-06 Thread Ema (Code Review)

Ema has submitted this change and it was merged.

Change subject: Release 4.1.4-1wm1
..


Release 4.1.4-1wm1

Change-Id: Id9499a10edd97f41b5c7b6dcc486d1509fc3eb17
---
M debian/changelog
1 file changed, 6 insertions(+), 0 deletions(-)

Approvals:
  Ema: Looks good to me, approved
  jenkins-bot: Verified



diff --git a/debian/changelog b/debian/changelog
index 319edd8..ff68cd6 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+varnish (4.1.4-1wm1) jessie-wikimedia; urgency=medium
+
+  * New upstream release.
+
+ -- Emanuele Rocca   Mon, 05 Dec 2016 13:19:31 +0100
+
 varnish (4.1.3-1wm4) jessie-wikimedia; urgency=medium
 
   * Make extrachance upper bound configurable.

-- 
To view, visit https://gerrit.wikimedia.org/r/325289
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: Id9499a10edd97f41b5c7b6dcc486d1509fc3eb17
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/varnish4
Gerrit-Branch: debian-wmf
Gerrit-Owner: Ema 
Gerrit-Reviewer: Ema 
Gerrit-Reviewer: jenkins-bot <>

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations...varnish4[debian-wmf]: Release 4.1.4-1wm1

2016-12-05 Thread Ema (Code Review)

Ema has uploaded a new change for review.

  https://gerrit.wikimedia.org/r/325289

Change subject: Release 4.1.4-1wm1
..

Release 4.1.4-1wm1

Change-Id: Id9499a10edd97f41b5c7b6dcc486d1509fc3eb17
---
M debian/changelog
1 file changed, 6 insertions(+), 0 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/debs/varnish4 
refs/changes/89/325289/1

diff --git a/debian/changelog b/debian/changelog
index 319edd8..ff68cd6 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+varnish (4.1.4-1wm1) jessie-wikimedia; urgency=medium
+
+  * New upstream release.
+
+ -- Emanuele Rocca   Mon, 05 Dec 2016 13:19:31 +0100
+
 varnish (4.1.3-1wm4) jessie-wikimedia; urgency=medium
 
   * Make extrachance upper bound configurable.

-- 
To view, visit https://gerrit.wikimedia.org/r/325289
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Id9499a10edd97f41b5c7b6dcc486d1509fc3eb17
Gerrit-PatchSet: 1
Gerrit-Project: operations/debs/varnish4
Gerrit-Branch: debian-wmf
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: systemd-timesyncd: fix config file

2017-01-06 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/330863 )

Change subject: systemd-timesyncd: fix config file
..

systemd-timesyncd: fix config file

The proper option to specify NTP servers is 'Servers', not 'NTP'.

Also avoid trailing whitespace before the first server name:
Servers= foo bar -> Servers=foo bar

Bug: T150257
Change-Id: Icc0d0ebe04da1107a100208928f5626e9815c9a6
---
M modules/base/templates/timesyncd.conf.erb
1 file changed, 1 insertion(+), 2 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/63/330863/1

diff --git a/modules/base/templates/timesyncd.conf.erb 
b/modules/base/templates/timesyncd.conf.erb
index 6f3c9a2..6b70325 100644
--- a/modules/base/templates/timesyncd.conf.erb
+++ b/modules/base/templates/timesyncd.conf.erb
@@ -1,5 +1,4 @@
 ## THIS FILE IS MANAGED BY PUPPET
 
 [Time]
-NTP=<%@ntp_servers[@site].sort.each do |ntpserver| -%> <%= ntpserver -%>
-<% end %>
+Servers=<%@ntp_servers[@site].sort.each do |ntpserver| -%><%= ntpserver -%> <% 
end %>

-- 
To view, visit https://gerrit.wikimedia.org/r/330863
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icc0d0ebe04da1107a100208928f5626e9815c9a6
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema 

___
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits

[MediaWiki-commits] [Gerrit] operations/puppet[production]: varnishxcache: port to cachestats.CacheStatsSender

2017-01-02 Thread Ema (Code Review)

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/330111 )

Change subject: varnishxcache: port to cachestats.CacheStatsSender
..

varnishxcache: port to cachestats.CacheStatsSender

Bug: T151643
This reverts commit 2a6ffdb2a8253e40b894efc332457e8df9893e4e.

Change-Id: I82eedf0fb7581064ba2d8d392021411eda3bd53c
---
M modules/varnish/files/varnishxcache
1 file changed, 46 insertions(+), 86 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/11/330111/1

diff --git a/modules/varnish/files/varnishxcache 
b/modules/varnish/files/varnishxcache
index ebbcead..0b5058b 100755
--- a/modules/varnish/files/varnishxcache
+++ b/modules/varnish/files/varnishxcache
@@ -2,7 +2,7 @@
 # -*- coding: utf-8 -*-
 """
   varnishxcache
-  ~~~
+  ~
 
   Accumulate X-Cache stats and report them to StatsD.
 
@@ -12,6 +12,7 @@
 --key-prefix PREFIX metric key prefix (default: varnish.xcache)
 
   Copyright 2016 Brandon Black 
+  Copyright 2016 Emanuele Rocca 
   Copyright 2015 Ori Livneh 
 
   Licensed under the Apache License, Version 2.0 (the "License");
@@ -27,100 +28,59 @@
   limitations under the License.
 
 """
-import argparse
-import io
 import re
-import socket
-import threading
-import urlparse
-import time
+import sys
 
-import varnishlog
+from cachestats import CacheStatsSender
 
 
-def parse_statsd_server_string(server_string):
-parsed = urlparse.urlparse('//' + server_string)
-return parsed.hostname, parsed.port or 8125
+class XCacheStatsSender(CacheStatsSender):
+cmd = ['/usr/bin/varnishncsa', '-n', 'frontend',
+   '-q', 'ReqMethod ne "PURGE"',
+   '-F', '%{X-Cache}o']
 
+description = 'X-Cache StatsD reporter'
+key_prefix = 'varnish.xcache'
+default_keys = (
+'unknown',
+'hit-front',
+'hit-local',
+'hit-remote',
+'int-front',
+'int-local',
+'int-remote',
+'pass',
+'miss',
+)
 
-def parse_prefix_string(key_prefix):
-key_prefix = key_prefix.strip('.')
-if not key_prefix:
-raise ValueError('Key prefix must not be empty')
-return key_prefix
+def __init__(self, argument_list):
+super(XCacheStatsSender, self).__init__(argument_list)
 
+self.key_value_pairs = re.compile(r'([A-Z][A-Z0-9]*)=([^;]+)')
 
-ap = argparse.ArgumentParser(
-description='X-Cache StatsD reporter',
-epilog='If no statsd server is specified, prints stats to stdout instead.'
-)
-ap.add_argument('--statsd-server', help='statsd server',
-type=parse_statsd_server_string, default=None)
-ap.add_argument('--key-prefix', help='metric key prefix',
-type=parse_prefix_string, default='test.varnish.xcache')
-ap.add_argument('--interval', help='send interval',
-type=int, default=30)
-args = ap.parse_args()
+self.re_simplify = re.compile(
+'cp[0-9]{4} (hit|miss|pass|int)(?:/[0-9]+)?')
 
-sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
-key_value_pairs = re.compile(r'([A-Z][A-Z0-9]*)=([^;]+)')
-stats = {
-'unknown': 0,
-'hit-front': 0,
-'hit-local': 0,
-'hit-remote': 0,
-'int-front': 0,
-'int-local': 0,
-'int-remote': 0,
-'pass': 0,
-'miss': 0,
-}
+self.state_finder = [
+[re.compile(r'hit$'),'hit-front'],
+[re.compile(r'hit,[^,]+$'),  'hit-local'],
+[re.compile(r'hit'), 'hit-remote'],
+[re.compile(r'int$'),'int-front'],
+[re.compile(r'int,[^,]+$'),  'int-local'],
+[re.compile(r'int'), 'int-remote'],
+[re.compile(r'pass,[^,]+$'), 'pass'],
+[re.compile(r'miss'),'miss'],
+]
 
-re_simplify = re.compile('cp[0-9]{4} (hit|miss|pass|int)(?:/[0-9]+)?')
-state_finder = [
-[re.compile(r'hit$'),'hit-front'],
-[re.compile(r'hit,[^,]+$'),  'hit-local'],
-[re.compile(r'hit'), 'hit-remote'],
-[re.compile(r'int$'),'int-front'],
-[re.compile(r'int,[^,]+$'),  'int-local'],
-[re.compile(r'int'), 'int-remote'],
-[re.compile(r'pass,[^,]+$'), 'pass'],
-[re.compile(r'miss'),'miss'],
-]
+def gen_stats(self, record):
+record = self.re_simplify.sub(r'\1', record)
+state = 'unknown'
+for sf in self.state_finder:
+if sf[0].search(record):
+state = sf[1]
+break
 
+self.stats[state] += 1
 
-def vsl_callback(transaction_id, tag, record, remote_party):
-record = record.split(': ')[1]
-
-record = re_simplify.sub(r'\1', record)
-state = 'unknown'
-for sf in state_finder:
-if sf[0].search(record):
-state = sf[1]
-break
-stats[state] += 1
-
-now = time.time()
-

< 1 2 3 4 5 6 7 8 9 10 >

501 - 600 of 1208 matches

Mail list logo