As promised, I started to dig in to the Maps documentation [1] and started to write some diagrams. The source of those diagram is at the end of this email, you can process that source online if you want the pretty pictures [2] (well, not that pretty).
I should probably post those diagrams somewhere for discussion, but not sure where it make sense... [1] https://wikitech.wikimedia.org/wiki/Maps [2] http://www.planttext.com/planttext **Maps component diagram:** @startuml [Varnish] [Kartotherian] [Tilerator] [TileratorUI] [OSM] database Cassandra database Redis database Postgres OSM -> Postgres: import maps data Tilerator -left-> Redis: consume job queue Tilerator --> Postgres: get data to pre-generate\nvector tiles Tilerator -> Cassandra: stores pre-generated\nvector tiles TileratorUI ..> Redis: schedule jobs to\npre-generate tiles Kartotherian --> Cassandra: serve tiles\nto end user Varnish --> Kartotherian @enduml **Maps deployment diagram:** @startuml () "Maps\n(public)" as mapsP package codfw { package "Maps cache cluster" as cache { node cp2003 { [varnish-frontend] as vfe2003 [varnish-backend] as vbe2003 } node cp2009 { [varnish-frontend] as vfe2009 [varnish-backend] as vbe2009 } node cp2015 { [varnish-frontend] as vfe2015 [varnish-backend] as vbe2015 } node cp2021 { [varnish-frontend] as vfe2021 [varnish-backend] as vbe2021 } } () "Maps\n(internal)" as mapsI node "maps-test2001\n(master)" as maps2001 { [Kartotherian] as Kartotherian2001 [Tilerator] as Tilerator2001 [TileratorUI] as TileratorUI2001 database Cassandra as Cassandra2001 database Redis as Redis2001 database "Postgres\nmaster" as Postgres2001 Tilerator2001 -left-> Redis2001 Tilerator2001 --> Postgres2001 Tilerator2001 -> Cassandra2001 TileratorUI2001 --> Redis2001 Kartotherian2001 --> Cassandra2001 } node "maps-test2002-4\n(slaves)" as maps2002 { [Kartotherian] as Kartotherian20xx [Tilerator] as Tilerator20xx [TileratorUI] as TileratorUI20xx database Cassandra as Cassandra20xx database Redis as Redis20xx database "Postgres\nslaves" as Postgres20xx Tilerator20xx -left-> Redis20xx Tilerator20xx --> Postgres20xx Tilerator20xx --> Cassandra20xx TileratorUI20xx -> Redis20xx Kartotherian20xx --> Cassandra20xx } mapsI - Kartotherian2001 mapsI - Kartotherian20xx vbe2003 -> mapsI vbe2009 -> mapsI vbe2015 -> mapsI vbe2021 -> mapsI vfe2003 --> vbe2003 vfe2009 --> vbe2009 vfe2015 --> vbe2015 vfe2021 --> vbe2021 ' un comment the block below to have the mostly complete Varnish connections ' vfe2003 --> vbe2003 ' vfe2003 --> vbe2009 ' vfe2003 --> vbe2015 ' vfe2003 --> vbe2021 ' ' vfe2009 --> vbe2003 ' vfe2009 --> vbe2009 ' vfe2009 --> vbe2015 ' vfe2009 --> vbe2021 ' ' vfe2015 --> vbe2003 ' vfe2015 --> vbe2009 ' vfe2015 --> vbe2015 ' vfe2015 --> vbe2021 ' ' vfe2021 --> vbe2003 ' vfe2021 --> vbe2009 ' vfe2021 --> vbe2015 ' vfe2021 --> vbe2021 mapsP -- vfe2003 mapsP -- vfe2009 mapsP -- vfe2015 mapsP -- vfe2021 Postgres20xx <- Postgres2001 Cassandra20xx <-> Cassandra2001 Redis20xx <-> Redis2001 note right of vfe2003 interconnections between Varnish frontend and backend are more complex, not showing all this here. end note note right of mapsI Need to check if this is a LVS endpoint or if Varnish talks directly to Mapsf end note } note as n1 - unsure about what communication there is between maps-test nodes end note @enduml -- Guillaume Lederrey Operations Engineer, Discovery Wikimedia Foundation _______________________________________________ discovery mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/discovery
