This is incredible, thank you so much!
-- Sammy Tarling Software Engineer Wikimedia Foundation On Mon, 24 Oct 2022, 15:51 Krinkle, <krin...@fastmail.com> wrote: > I've done a major update to a number of diagrams on Wikitech. > > Usually, I don't mention an update here, but I'm highlighting it now as > it's been a while since we mentioned them on-list and the community and > foundation have grown a lot so some of these may be new to you. > > Given how much has changed in recent changes, I also included a changelog > and a link to where in the docs you'd normally discover this diagram > on-wiki: > > *== 1. File:Wikipedia_webrequest_2022.png > <https://wikitech.wikimedia.org/wiki/File:Wikipedia_webrequest_2022.png> > (Updated) ==* > > This is a highly simplified diagram, covering the general shape of our > stack through the example of a typical Wikipedia webrequest. > > Previous: > https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia_webrequest_flow_2020.png > New: > https://upload.wikimedia.org/wikipedia/commons/4/4d/Wikipedia_webrequest_2022.png > Documentation: wikitech:MediaWiki_at_WMF > <https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF> and > wikitech:Caching_overview > <https://wikitech.wikimedia.org/wiki/Caching_overview>. > Notable changes: > * Change edge TLS termination ("HTTPS") from ats-tls to HAProxy. I wrote a > "Caching overview § History > <https://wikitech.wikimedia.org/wiki/Caching_overview>" section. > * Change appserver TLS from Nginx- to Envoy. > * Add new MainStash DB. > * Include storage ExternalStore DB, ParserCache DB, and Swift media. > * Include services Shellbox, Mathoid, and Kask. > > *== 2. File:WMF_infrastructure_2022.png > <https://wikitech.wikimedia.org/wiki/File:WMF_infrastructure_2022.png> > (Updated) ==* > > This is a continous attempt at an overview of tier-1/user-facing > infrastructure. It will likely never be complete from all POV, but.. it is > more accurate and complete than it has been. Thanks to all that contributed > by entertaining my many questions over the years. > > Previous (2016 by Elukey): > https://upload.wikimedia.org/wikipedia/labs/4/4d/Infrastructure_overview.png > New: > https://upload.wikimedia.org/wikipedia/commons/4/48/WMF_infrastructure_2022.png > Documentation: wikitech:Wikimedia_infrastructure > <https://wikitech.wikimedia.org/wiki/Wikimedia_infrastructure> and > wikitech:Purged <https://wikitech.wikimedia.org/wiki/Purged> > Notable changes: > * Add new Drmrs data center in Marseille, France. > * Add new services: purged.go, EventStreams, Thumbor, mcrouter, Envoy, > etcd. > * Add new distinction for Multi-DC between primary and secondary data > center. > * Change sessionstore from Redis to Kask/Cassandra. > * Change jobqueue from Redis to EventGate/Kafka. > * Include distinct MediaWiki server roles and clusters. > * Include high-level MediaWiki platform components. > * Include example flow for "JobQueue job" and "CDN purge". > > *== 3. File:MediaWiki_infrastructure_2022.png > <https://wikitech.wikimedia.org/wiki/File:MediaWiki_infrastructure_2022.png> > (New) ==* > > Similar to WMF Infra diagram, but more abstract around DC and services, > and more detailed within the platform. Including more core services, and > recognising extensions as their own layer. > > New: > https://upload.wikimedia.org/wikipedia/commons/e/ee/MediaWiki_infrastructure_2022.png > Documentation: wikitech:MediaWiki_at_WMF > <https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF> > > *== 4. File:Wikipedia_Memcached_flow_2022.png > <https://wikitech.wikimedia.org/wiki/File:Wikipedia_Memcached_flow_2022.png> > (Updated)* > > Previous: > https://upload.wikimedia.org/wikipedia/commons/d/db/Wikipedia_Memcached_flow_2020.png > New: > https://upload.wikimedia.org/wikipedia/commons/4/45/Wikipedia_Memcached_flow_2022.png > Documentation: wikitech:Memcached_for_MediaWiki > <https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki> > Notable changes: > * Include the three tiers of ParserCache. > * Add WANCache legend to explain different keytypes you may encounter on > the network. > * Add full name of the mcrouter-with-onhost-tier service for greppability. > * Add new WRStats service (T310662 > <https://phabricator.wikimedia.org/T310662>). This was part of Multi-DC > work > <https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Multi-DC_MediaWiki> > to reduce primary DB writes and (not bi-di replicated) Redis use in > AbuseFilter. This service also replaces the old "User ping limiter" in core > and is now able to serve both use cases. > * Remove "on-host: soon" labels. Adopting on-host memc for WANCache was > considered not worth the added runtime complexity (T264604 > <https://phabricator.wikimedia.org/T264604>). Note that SRE's work on > adding 10G network links for memcached hosts, and the addition of > mcrouter-managed gutter pools take care of the general usecase that we were > exploring on-host for. We kept it for ParserCache however (T244340 > <https://phabricator.wikimedia.org/T244340>. > > *== Edit link ==* > > As before, each diagram file page has an "Edit" link in the description > that takes you directly to the open-source Diagrams.net web app (loading > file read-only from Google Drive). You can fork by using "Save as" in the > web app. See also: wikitech:Performance/Runbook/diagrams.net_conventions > <https://wikitech.wikimedia.org/wiki/Performance/Runbook/diagrams.net_conventions> > > -- Timo > > > _______________________________________________ > Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org > To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org > https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
_______________________________________________ Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org To unsubscribe send an email to wikitech-l-le...@lists.wikimedia.org https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
