https://bugzilla.wikimedia.org/show_bug.cgi?id=41130
--- Comment #86 from Bawolff (Brian Wolff) <bawolff...@gmail.com> --- Just to clarify, Here's a summary of this bug: There are 3 main issues mentioned in this bug, corresponding to three stages of an HTCP packet's life cycle (Generation, in transit, destination) 1) The generation of purge packets. Sometimes MW doesn't realize that it needs to send a purge for certain thumb sizes. This is what (I believe) TMg is referring to by a "deadlock" situation in comment 9 (Not to be confused with the varnishhtcpd deadlock which is entirely separate). This is also the issue that Robla refers to as "Squid/varnish cache contains images that aren't in Swift". This is tracked by bug 44428. The symptoms of this bug is that some (most) thumbnails are regenerated perfectly fine, however there are a few that are "stuck". If all new thumbnails are failing, then it is not this bug. action=purge 'ing the image description pages does not fix the stuck thumbnails. However the work arounds (either comment 18 's work around, or mine from comment 43. Both work the same way) will fix stuck thumbnails that are suffering from this issue. Note that these work arounds do not work if the other bugs are present. 2) (All) the purge packets get lost in transits going to certain caches. Its not 100% clear if this is fixed or not. Something is wrong with the program (or hardware, etc) that "tunnels" purge packets generated in North America to the cache servers in Amsterdam. In affect, MediaWiki yells to the world that "Image foo has been updated", but only the North America cache servers can hear mediawiki. The symptoms of this bug are: Only people accessing via Europe see the issue. All thumbs (along with full size versions of recently updated images) are outdated for Europe people. Additionally Annonoymous users accessing via Europe will see outdated versions of pages (Since the htcp tunnel also tunnels non-image purges). bug 44391 was specificly about that. 3) Issues at the destination. This is the fixed issue. Varnish servers (and only Varnish. This does not include the Squid servers. The Amsterdam caching servers are squids, where the North America caching servers are Varnish, at least last I heard) use a program called varnishhtcpd. When mediawiki sends a "Image foo has been updated" message to the caching servers, varnishhtcpd translates the message from the format MediaWiki uses (htcp) to the format that varnish uses (http PURGE). There was some problems with this program, that caused it to freeze. If the program freezed, the varnish servers could no longer understand the purge messages MediaWiki was sending. The symptoms of this bug was that only people accessing via North America, since only North America had the varnishes (The Squid servers in Europe could understand HTCP purges by themselves). This bug was tracked by bug 43448. Note: there were some brief periods where after (3) was fixed before (2) cropped up, where people were mostly happy. Additionally there is bug 43449, to add monitoring so that warning bells (other than a horde of angry users) go off when cache purging fails. There is is also bug 44269 which seems to not have anything to do with the caching servers as far as I can tell. ---------- Current situation - Just testing now. Purging an image seems to result in the cache being cleared both in europe and North America. This suggests that problem 2 is indeed fixed (yay Leslie and anyone else involved!), which leaves us just with problem 1. If that is the case, the work arounds should generally work. To re-iterate, the work around is as follows. If File:Example.svg was not updating for the 200px size, you would do the following: 1. Go to http://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Example.svg/200px-Example.svg.png?RandomNumberHere1234 (replacing RandomNumberHere1234 with something random) 2. Go to http://commons.wikimedia.org/wiki/File:Example.svg?action=purge -- You are receiving this mail because: You are watching all bug changes. _______________________________________________ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
