[MediaWiki-commits] [Gerrit] operations/puppet[production]: VCL: rate limit wikiScrape with vsthrottle

[Ema (Code Review)]

Ema has uploaded a new change for review. ( 
https://gerrit.wikimedia.org/r/357995 )

Change subject: VCL: rate limit wikiScrape with vsthrottle
......................................................................

VCL: rate limit wikiScrape with vsthrottle

Bug: T163233
Change-Id: Ib7decc6d1f9ab77be009862bfb6861bfd3be173e
---
M modules/varnish/templates/text-frontend.inc.vcl.erb
M modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
2 files changed, 8 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.wikimedia.org:29418/operations/puppet 
refs/changes/95/357995/1

diff --git a/modules/varnish/templates/text-frontend.inc.vcl.erb 
b/modules/varnish/templates/text-frontend.inc.vcl.erb
index d6e50bc..91f680f 100644
--- a/modules/varnish/templates/text-frontend.inc.vcl.erb
+++ b/modules/varnish/templates/text-frontend.inc.vcl.erb
@@ -150,10 +150,6 @@
                return (synth(741, "Buggy request, please report at 
https://phabricator.wikimedia.org/T141786";));
        }
 
-       if (req.http.User-Agent ~ "wikiScrape/0.0.0") {
-               return (synth(742, "Too many requests"));
-       }
-
        call mobile_redirect;
 
        # normalize all /static to the same hostname for caching
@@ -206,6 +202,11 @@
 sub cluster_fe_hit { }
 
 sub cluster_fe_miss {
+       if (req.http.User-Agent ~ "^wikiScrape/[0-9]+\.[0-9]+\.[0-9]+$") {
+               if (vsthrottle.is_denied(client.identity, 1, 1s)) {
+                       return (synth(742, "Too Many Requests"));
+               }
+       }
 }
 
 sub cluster_fe_pass {
diff --git a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb 
b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
index 014e85f..33d528e 100644
--- a/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-common.inc.vcl.erb
@@ -14,6 +14,9 @@
 # this import conditional on existing vslp backend definitions.
 import vslp;
 
+# rate-limiting
+import vsthrottle;
+
 include "errorpage.inc.vcl";
 
 

-- 
To view, visit https://gerrit.wikimedia.org/r/357995
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib7decc6d1f9ab77be009862bfb6861bfd3be173e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: Ema <e...@wikimedia.org>

_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits