BBlack has submitted this change and it was merged. ( https://gerrit.wikimedia.org/r/385415 )
Change subject: htcppurger: per-dc/cluster delay data ...................................................................... htcppurger: per-dc/cluster delay data This sets queueing delays on all of the purge queues to greatly reduce the odds of layer-race conditions: We use 500ms as our base value at the backend-most backend (eqiad), add 500ms more per DC tier, and then use a fixed 1s for the frontend at all sites (which occurs serially after the local backend has gone through its delay+purge). Backends only pull from other backends, and the current routing config is that codfw and esams pull from eqiad, and ulsfo pulls from codfw. Thus the timeline will look something like this if one only looks at the queueing delay plus expected actual network delays (half-RTT, as they're unidirectional!), but no local PURGE execution delay in varnish (which can add a few more ms between BE->FE within a site, especially in the case of execution backlog due to rate spikes): T0.000 - A multicast PURGE is emitted from somewhere in EQIAD T0.000 - Multicast arrives at EQIAD backends T0.018 - Multicast arrives at CODFW backends T0.039 - Multicast arrives at ULSFO backends T0.042 - Multicast arrives at ESAMS backends T0.500 - EQIAD backends execute PURGE T1.018 - CODFW backends execute PURGE T1.042 - ESAMS backends execute PURGE T1.500 - EQIAD frontends execute PURGE T1.539 - ULSFO backends execute PURGE T2.018 - CODFW frontends execute PURGE T2.042 - ESAMS frontends execute PURGE T2.539 - ULSFO frontends execute PURGE Bug: T133821 Change-Id: Id909f54cfc126dcd1c933e2f7cef12782cc7ab6f --- A hieradata/role/codfw/cache/misc.yaml M hieradata/role/codfw/cache/text.yaml M hieradata/role/codfw/cache/upload.yaml A hieradata/role/eqiad/cache/misc.yaml M hieradata/role/eqiad/cache/text.yaml M hieradata/role/eqiad/cache/upload.yaml A hieradata/role/esams/cache/misc.yaml A hieradata/role/esams/cache/text.yaml A hieradata/role/esams/cache/upload.yaml A hieradata/role/ulsfo/cache/misc.yaml A hieradata/role/ulsfo/cache/text.yaml A hieradata/role/ulsfo/cache/upload.yaml 12 files changed, 12 insertions(+), 0 deletions(-) Approvals: BBlack: Looks good to me, approved jenkins-bot: Verified diff --git a/hieradata/role/codfw/cache/misc.yaml b/hieradata/role/codfw/cache/misc.yaml new file mode 100644 index 0000000..f6e702a --- /dev/null +++ b/hieradata/role/codfw/cache/misc.yaml @@ -0,0 +1 @@ +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.0', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/codfw/cache/text.yaml b/hieradata/role/codfw/cache/text.yaml index 6a2fc86..708814d 100644 --- a/hieradata/role/codfw/cache/text.yaml +++ b/hieradata/role/codfw/cache/text.yaml @@ -1 +1,2 @@ cache::backend_warming: false +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.0', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/codfw/cache/upload.yaml b/hieradata/role/codfw/cache/upload.yaml index 6a2fc86..708814d 100644 --- a/hieradata/role/codfw/cache/upload.yaml +++ b/hieradata/role/codfw/cache/upload.yaml @@ -1 +1,2 @@ cache::backend_warming: false +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.0', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/eqiad/cache/misc.yaml b/hieradata/role/eqiad/cache/misc.yaml new file mode 100644 index 0000000..86578ae --- /dev/null +++ b/hieradata/role/eqiad/cache/misc.yaml @@ -0,0 +1 @@ +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,0.5', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/eqiad/cache/text.yaml b/hieradata/role/eqiad/cache/text.yaml index 6a2fc86..bb27810 100644 --- a/hieradata/role/eqiad/cache/text.yaml +++ b/hieradata/role/eqiad/cache/text.yaml @@ -1 +1,2 @@ cache::backend_warming: false +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,0.5', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/eqiad/cache/upload.yaml b/hieradata/role/eqiad/cache/upload.yaml index 6a2fc86..bb27810 100644 --- a/hieradata/role/eqiad/cache/upload.yaml +++ b/hieradata/role/eqiad/cache/upload.yaml @@ -1 +1,2 @@ cache::backend_warming: false +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,0.5', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/esams/cache/misc.yaml b/hieradata/role/esams/cache/misc.yaml new file mode 100644 index 0000000..f6e702a --- /dev/null +++ b/hieradata/role/esams/cache/misc.yaml @@ -0,0 +1 @@ +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.0', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/esams/cache/text.yaml b/hieradata/role/esams/cache/text.yaml new file mode 100644 index 0000000..f6e702a --- /dev/null +++ b/hieradata/role/esams/cache/text.yaml @@ -0,0 +1 @@ +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.0', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/esams/cache/upload.yaml b/hieradata/role/esams/cache/upload.yaml new file mode 100644 index 0000000..f6e702a --- /dev/null +++ b/hieradata/role/esams/cache/upload.yaml @@ -0,0 +1 @@ +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.0', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/ulsfo/cache/misc.yaml b/hieradata/role/ulsfo/cache/misc.yaml new file mode 100644 index 0000000..a3d63bf --- /dev/null +++ b/hieradata/role/ulsfo/cache/misc.yaml @@ -0,0 +1 @@ +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.5', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/ulsfo/cache/text.yaml b/hieradata/role/ulsfo/cache/text.yaml new file mode 100644 index 0000000..a3d63bf --- /dev/null +++ b/hieradata/role/ulsfo/cache/text.yaml @@ -0,0 +1 @@ +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.5', '127.0.0.1:3127,1.0'] diff --git a/hieradata/role/ulsfo/cache/upload.yaml b/hieradata/role/ulsfo/cache/upload.yaml new file mode 100644 index 0000000..a3d63bf --- /dev/null +++ b/hieradata/role/ulsfo/cache/upload.yaml @@ -0,0 +1 @@ +profile::cache::base::purge_varnishes: ['127.0.0.1:3128,1.5', '127.0.0.1:3127,1.0'] -- To view, visit https://gerrit.wikimedia.org/r/385415 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id909f54cfc126dcd1c933e2f7cef12782cc7ab6f Gerrit-PatchSet: 3 Gerrit-Project: operations/puppet Gerrit-Branch: production Gerrit-Owner: BBlack <[email protected]> Gerrit-Reviewer: BBlack <[email protected]> Gerrit-Reviewer: jenkins-bot <> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
