BBlack has uploaded a new change for review. (
https://gerrit.wikimedia.org/r/376751 )
Change subject: [WIP] stabilize backend storage patterns
......................................................................
[WIP] stabilize backend storage patterns
As discussed on IRC, this may help reduce the mailbox lag problem
and improve the backend hitrates (for local fe traffic) at the
core DCs. The basic idea is that for be->be fetches, the
receiving backend should operate in a hit-or-pass mode: it will
serve existing object hits, but will not bring new misses into its
cached set. A hieradata setting should control the
"@backend_warming" flag in the VCL, for the operationally-rare
case where we want to disable this behavior when warming an inner
cache using an outer's misses.
Note this change also modifies the calculation of X-Cache-Status
wrt miss-vs-pass. We previously only had known cases of
layer-differential pass behavior in a single direction (where the
FE might pass but the BE would attempt real hit/miss), but now we
have layer-differential pass possible in both directions (an outer
be might miss, while the inner be's all pass). In light of this,
it probably makes more sense to consider X-Cache lines containing
any miss at any layer to be a true miss, and only count a pass
when every layer passes the traffic. This could cause small
sudden jumps in the existing miss/pass stats due to the
re-interpration.
TODO: hook up the @backend_warming flag, modulo bikeshedding about
various puppet coding rules and ongoing refactors.
Change-Id: I17f02801f1d18ee0b27521f182fd4e041c6dd8fb
---
M modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
2 files changed, 28 insertions(+), 10 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/51/376751/1
diff --git a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
index 5f63062..f745a5d 100644
--- a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb
@@ -80,8 +80,25 @@
return (synth(403, "Access denied"));
}
- // Used to signal to other logic whether this request jumps next to
another cache or the applayer
- unset req.http.X-Next-Is-Cache;
+ // X-Next-Is-Cache is used to signal to other logic whether this
+ // request jumps next to another cache or the applayer. We also
+ // convert such a signal sent by a previous backend cache to
+ // X-MISS2PASS, which converts misses to passes (we're only looking for
+ // opporunistic hits on these, but don't want to disturb the local
+ // backend cache's object hotness/distribution). hash_ignore_busy is
+ // to avoid stalling on those passes (like a cheaper hfp). The
+ // backend_warming setting is meant to be set temporarily from
+ // hieradata to disable this behavior (when we want to use the misses
+ // from an outer-layer backend to pre-warm some data in the next layer
+ // deep of backend).
+ unset req.http.X-MISS2PASS;
+ if (req.http.X-Next-Is-Cache) {
+ unset req.http.X-Next-Is-Cache;
+ <% if not @backend_warming -%> // XXX no implementation of
setting this flag yet
+ set req.http.X-MISS2PASS = 1;
+ set req.hash_ignore_busy = true;
+ <% end -%>
+ }
call wm_common_recv_early;
@@ -133,6 +150,10 @@
sub vcl_miss {
call wm_common_miss;
call cluster_be_miss;
+ if (req.http.X-MISS2PASS) {
+ unset req.http.X-MISS2PASS;
+ return (pass);
+ }
return (fetch); // no default VCL (which is just "return (fetch)"
anyways)
}
diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
index 1e17089..83b5a06 100644
--- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
+++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb
@@ -229,7 +229,9 @@
error 403 "This server has been disabled, you probably ended up here
because of hardcoded IPs or broken DNS caching";
<% end -%>
- unset req.http.X-DCPath; // no injection from outside our stack
+ // no injection from outside our stack allowed for these
+ unset req.http.X-DCPath;
+ unset req.http.X-Next-Is-Cache;
unset req.http.Proxy; // https://httpoxy.org/
if (req.restarts == 0) {
@@ -367,15 +369,10 @@
set resp.http.X-Cache-Status = "hit";
} elsif (resp.http.X-Cache ~ "int") {
set resp.http.X-Cache-Status = "int";
- } elsif (resp.http.X-Cache ~ "pass,[^,]+$") {
- // The special pass regex above is because we have
- // layer-differential pass rules where a frontend may
- // intentionally pass traffic that a backend does not,
- // which should still count as a true 'miss' if the
- // backend misses it.
- set resp.http.X-Cache-Status = "pass";
} elsif (resp.http.X-Cache ~ "miss") {
set resp.http.X-Cache-Status = "miss";
+ } elsif (resp.http.X-Cache ~ "pass") {
+ set resp.http.X-Cache-Status = "pass";
} else {
set resp.http.X-Cache-Status = "bug";
}
--
To view, visit https://gerrit.wikimedia.org/r/376751
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I17f02801f1d18ee0b27521f182fd4e041c6dd8fb
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: BBlack <bbl...@wikimedia.org>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
