BBlack has uploaded a new change for review. ( https://gerrit.wikimedia.org/r/376751 )
Change subject: [WIP] stabilize backend storage patterns ...................................................................... [WIP] stabilize backend storage patterns As discussed on IRC, this may help reduce the mailbox lag problem and improve the backend hitrates (for local fe traffic) at the core DCs. The basic idea is that for be->be fetches, the receiving backend should operate in a hit-or-pass mode: it will serve existing object hits, but will not bring new misses into its cached set. A hieradata setting should control the "@backend_warming" flag in the VCL, for the operationally-rare case where we want to disable this behavior when warming an inner cache using an outer's misses. Note this change also modifies the calculation of X-Cache-Status wrt miss-vs-pass. We previously only had known cases of layer-differential pass behavior in a single direction (where the FE might pass but the BE would attempt real hit/miss), but now we have layer-differential pass possible in both directions (an outer be might miss, while the inner be's all pass). In light of this, it probably makes more sense to consider X-Cache lines containing any miss at any layer to be a true miss, and only count a pass when every layer passes the traffic. This could cause small sudden jumps in the existing miss/pass stats due to the re-interpration. TODO: hook up the @backend_warming flag, modulo bikeshedding about various puppet coding rules and ongoing refactors. Change-Id: I17f02801f1d18ee0b27521f182fd4e041c6dd8fb --- M modules/varnish/templates/vcl/wikimedia-backend.vcl.erb M modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb 2 files changed, 28 insertions(+), 10 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/puppet refs/changes/51/376751/1 diff --git a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb index 5f63062..f745a5d 100644 --- a/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb +++ b/modules/varnish/templates/vcl/wikimedia-backend.vcl.erb @@ -80,8 +80,25 @@ return (synth(403, "Access denied")); } - // Used to signal to other logic whether this request jumps next to another cache or the applayer - unset req.http.X-Next-Is-Cache; + // X-Next-Is-Cache is used to signal to other logic whether this + // request jumps next to another cache or the applayer. We also + // convert such a signal sent by a previous backend cache to + // X-MISS2PASS, which converts misses to passes (we're only looking for + // opporunistic hits on these, but don't want to disturb the local + // backend cache's object hotness/distribution). hash_ignore_busy is + // to avoid stalling on those passes (like a cheaper hfp). The + // backend_warming setting is meant to be set temporarily from + // hieradata to disable this behavior (when we want to use the misses + // from an outer-layer backend to pre-warm some data in the next layer + // deep of backend). + unset req.http.X-MISS2PASS; + if (req.http.X-Next-Is-Cache) { + unset req.http.X-Next-Is-Cache; + <% if not @backend_warming -%> // XXX no implementation of setting this flag yet + set req.http.X-MISS2PASS = 1; + set req.hash_ignore_busy = true; + <% end -%> + } call wm_common_recv_early; @@ -133,6 +150,10 @@ sub vcl_miss { call wm_common_miss; call cluster_be_miss; + if (req.http.X-MISS2PASS) { + unset req.http.X-MISS2PASS; + return (pass); + } return (fetch); // no default VCL (which is just "return (fetch)" anyways) } diff --git a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb index 1e17089..83b5a06 100644 --- a/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb +++ b/modules/varnish/templates/vcl/wikimedia-frontend.vcl.erb @@ -229,7 +229,9 @@ error 403 "This server has been disabled, you probably ended up here because of hardcoded IPs or broken DNS caching"; <% end -%> - unset req.http.X-DCPath; // no injection from outside our stack + // no injection from outside our stack allowed for these + unset req.http.X-DCPath; + unset req.http.X-Next-Is-Cache; unset req.http.Proxy; // https://httpoxy.org/ if (req.restarts == 0) { @@ -367,15 +369,10 @@ set resp.http.X-Cache-Status = "hit"; } elsif (resp.http.X-Cache ~ "int") { set resp.http.X-Cache-Status = "int"; - } elsif (resp.http.X-Cache ~ "pass,[^,]+$") { - // The special pass regex above is because we have - // layer-differential pass rules where a frontend may - // intentionally pass traffic that a backend does not, - // which should still count as a true 'miss' if the - // backend misses it. - set resp.http.X-Cache-Status = "pass"; } elsif (resp.http.X-Cache ~ "miss") { set resp.http.X-Cache-Status = "miss"; + } elsif (resp.http.X-Cache ~ "pass") { + set resp.http.X-Cache-Status = "pass"; } else { set resp.http.X-Cache-Status = "bug"; } -- To view, visit https://gerrit.wikimedia.org/r/376751 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I17f02801f1d18ee0b27521f182fd4e041c6dd8fb Gerrit-PatchSet: 1 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: BBlack <bbl...@wikimedia.org> _______________________________________________ MediaWiki-commits mailing list MediaWiki-commits@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits