[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Ryan Kaldari rkald...@wikimedia.org changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=48927 -- You are receiving this mail because: You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 bennylin bknl...@yahoo.com changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=44872 -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Priority|Immediate |Normal Status|REOPENED|RESOLVED Resolution|--- |INVALID Assignee|ro...@wikimedia.org |aklap...@wikimedia.org Severity|critical|major --- Comment #90 from Andre Klapper aklap...@wikimedia.org --- This report mixes various numbers of issues which are now handled in separate reports, with one problem per report (see comment 86 and comment 89 for links). In case that cache purging issues come up that are not handled by one the existing reports, please file a new report. General recommendations are available at https://www.mediawiki.org/wiki/How_to_report_a_bug It has become rather impossible to define at which state (after some actions or partial fixes) this report could be considered FIXED due to several issues reported here. Hence I am closing this report as INVALID. This just refers to this report, it does not mean that the described problems are INVALID - they are just handled in separate tickets now. I'd like to thank everybody for the helpful input which led to identifying and separating several problems in Wikimedia's infrastructure, some of them having received fixes already. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #91 from Andre Klapper aklap...@wikimedia.org --- (In reply to comment #89) I'm splitting comment 87 - 88 off into bug 44508. Thanks a lot for doing that! Bug 44508 has been handled and three testers state that it is fixed now. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Nemo federicol...@tiscali.it changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=44428 -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #88 from TMg mr.h...@gmx.de --- (In reply to comment #86) the work around is as follows. If File:Example.svg was not updating for the 200px size, you would do the following: 1. Go to http://upload.wikimedia.org/wikipedia/commons/thumb/8/84/ Example.svg/200px-Example.svg.png?RandomNumberHere1234 (replacing RandomNumberHere1234 with something random) 2. Go to http://commons.wikimedia.org/wiki/File:Example.svg?action=purge Tried again. Does not work. The files http://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/Wappen_Landkreis_Aurich.svg/140px-Wappen_Landkreis_Aurich.svg.png http://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/Wappen_Landkreis_Aurich.svg/423px-Wappen_Landkreis_Aurich.svg.png still show the old and the new version randomly when pressing F5. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Bawolff (Brian Wolff) bawolff...@gmail.com changed: What|Removed |Added Blocks||44508 -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #89 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to comment #88) (In reply to comment #86) the work around is as follows. If File:Example.svg was not updating for the 200px size, you would do the following: 1. Go to http://upload.wikimedia.org/wikipedia/commons/thumb/8/84/ Example.svg/200px-Example.svg.png?RandomNumberHere1234 (replacing RandomNumberHere1234 with something random) 2. Go to http://commons.wikimedia.org/wiki/File:Example.svg?action=purge Tried again. Does not work. The files http://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/ Wappen_Landkreis_Aurich.svg/140px-Wappen_Landkreis_Aurich.svg.png http://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/ Wappen_Landkreis_Aurich.svg/423px-Wappen_Landkreis_Aurich.svg.png still show the old and the new version randomly when pressing F5. Confirmed, I'm seeing the same thing. I think this issue was only fixed for a couple hours and then broke again. I'm splitting comment 87 - 88 off into bug 44508. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #84 from TMg mr.h...@gmx.de --- (In reply to comment #79) Not sending a purge at all is completely different from the purge being lost along the way. Doesn't reuploading a file sends a purge? I have the feeling you are talking about a different issue. Please consider creating a separate report for this. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #85 from TMg mr.h...@gmx.de --- I'm pretty sure this won't be helpful. However the following is new to me and explains a lot (like people calling me an idiot because they see the right image but I don't). Currently when I load these thumbnails: http://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/Wappen_Landkreis_Aurich.svg/140px-Wappen_Landkreis_Aurich.svg.png http://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/Wappen_Landkreis_Aurich.svg/423px-Wappen_Landkreis_Aurich.svg.png and press F5 multiple times, sometimes it shows the old version and sometimes the new. Obviously this is an effect of the load balancer. I would be happy to give you more information but tracert always outputs the same (see comment 29). You are right, there are a lot of things to do: a) Stop all this from happening. If there are multiple causes it's *your* job to split this into multiple reports. We (the users) can't do this. All we see is random garbage. b) Fix the broken thumbnails. c) Fix the broken original size images. d) Give us one or more tools to fix this ourself. None of the ideas above works, not adding a random parameter to the thumbnail URL, not the /thumb.php?... call. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #86 from Bawolff (Brian Wolff) bawolff...@gmail.com --- Just to clarify, Here's a summary of this bug: There are 3 main issues mentioned in this bug, corresponding to three stages of an HTCP packet's life cycle (Generation, in transit, destination) 1) The generation of purge packets. Sometimes MW doesn't realize that it needs to send a purge for certain thumb sizes. This is what (I believe) TMg is referring to by a deadlock situation in comment 9 (Not to be confused with the varnishhtcpd deadlock which is entirely separate). This is also the issue that Robla refers to as Squid/varnish cache contains images that aren't in Swift. This is tracked by bug 44428. The symptoms of this bug is that some (most) thumbnails are regenerated perfectly fine, however there are a few that are stuck. If all new thumbnails are failing, then it is not this bug. action=purge 'ing the image description pages does not fix the stuck thumbnails. However the work arounds (either comment 18 's work around, or mine from comment 43. Both work the same way) will fix stuck thumbnails that are suffering from this issue. Note that these work arounds do not work if the other bugs are present. 2) (All) the purge packets get lost in transits going to certain caches. Its not 100% clear if this is fixed or not. Something is wrong with the program (or hardware, etc) that tunnels purge packets generated in North America to the cache servers in Amsterdam. In affect, MediaWiki yells to the world that Image foo has been updated, but only the North America cache servers can hear mediawiki. The symptoms of this bug are: Only people accessing via Europe see the issue. All thumbs (along with full size versions of recently updated images) are outdated for Europe people. Additionally Annonoymous users accessing via Europe will see outdated versions of pages (Since the htcp tunnel also tunnels non-image purges). bug 44391 was specificly about that. 3) Issues at the destination. This is the fixed issue. Varnish servers (and only Varnish. This does not include the Squid servers. The Amsterdam caching servers are squids, where the North America caching servers are Varnish, at least last I heard) use a program called varnishhtcpd. When mediawiki sends a Image foo has been updated message to the caching servers, varnishhtcpd translates the message from the format MediaWiki uses (htcp) to the format that varnish uses (http PURGE). There was some problems with this program, that caused it to freeze. If the program freezed, the varnish servers could no longer understand the purge messages MediaWiki was sending. The symptoms of this bug was that only people accessing via North America, since only North America had the varnishes (The Squid servers in Europe could understand HTCP purges by themselves). This bug was tracked by bug 43448. Note: there were some brief periods where after (3) was fixed before (2) cropped up, where people were mostly happy. Additionally there is bug 43449, to add monitoring so that warning bells (other than a horde of angry users) go off when cache purging fails. There is is also bug 44269 which seems to not have anything to do with the caching servers as far as I can tell. -- Current situation - Just testing now. Purging an image seems to result in the cache being cleared both in europe and North America. This suggests that problem 2 is indeed fixed (yay Leslie and anyone else involved!), which leaves us just with problem 1. If that is the case, the work arounds should generally work. To re-iterate, the work around is as follows. If File:Example.svg was not updating for the 200px size, you would do the following: 1. Go to http://upload.wikimedia.org/wikipedia/commons/thumb/8/84/Example.svg/200px-Example.svg.png?RandomNumberHere1234 (replacing RandomNumberHere1234 with something random) 2. Go to http://commons.wikimedia.org/wiki/File:Example.svg?action=purge -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #87 from Bawolff (Brian Wolff) bawolff...@gmail.com --- Current situation - Just testing now. Purging an image seems to result in the cache being cleared both in europe and North America. This suggests that problem 2 is indeed fixed (yay Leslie and anyone else involved!), which leaves us just with problem 1. If that is the case, the work arounds should generally work. After testing a bunch more, it seems the workingness is a bit intermittent. In one test, I did https://commons.wikimedia.org/wiki/File:Moscow_metro_map_ru_sb_future.svg?action=purge . I then looked at the age header. When accessing via caches in north america, the age header was reset as expected (yay!). However, when accessing via the europe caching servers [1] there was a rather unexpected result. Sometimes a varnish server responded (specifically the response had the header X-Cache: cp1033 hit (1), cp3010 miss (0), cp3009 frontend hit (2) ). When this happened the age header had been recently reset as expected. This goes beyond my knowledge of WMF's network setup, but I'm guessing that sometimes cache requests gets forwarded from esams to eqiad(?) since cp1033 is the cache server that seems to respond from eqiad too (Then again, I could be totally confused here). When I got a response from a squid server from esams, the age header was not reset. (It was 57479 = 15 hours, so nothing was horrendously old, hence htcp purges were getting there recently, but they aren't at the precise moment of me writing this). At the same time, tests I did of purging articles resulted in the cache being cleared both for the squids at esams, and for the varnish in eqiad, so it seems like htcp purges are being delievered properly. The conclusion I draw from this is: *I really have no idea :s Wild guesses include: Only the upload squid servers are for some reason not getting the htcp multicast purges, and only sometimes? The squid servers are overloaded? (However timing seems too coincidental for that to happen, also I would expect varnish to get overloaded first as it has the extra overhead of converting htcp - http purge request). It would be nice if ops (or other powers to be) could comment on what they think the status of multicast htcp purges working is. In various places there have been comments of we think this is fixed now, but no one has explicitly said any of the following: *The issue is 100% fixed and we're not worrying about it any longer *We managed to get things sort of working, but there's still some issues, and we're looking into them *Things are horribly horribly broken, and we're doing the best we can to sort things out *The issue has some other status. It would be really nice if we could have such a comment about this issue of the aforementioned nature. - [1] To simulate accessing from europe, I used commands of the form: wget -U bawolff -S --header 'host: upload.wikimedia.org' --no-check-certificate 'https://upload-lb.esams.wikimedia.org/wikipedia/commons/3/3d/Moscow_metro_map_ru_sb_future.svg' -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Summary|Invalid originals and |Invalidation of Varnish |thumbnails after|thumbnail cache sometimes |reupload/purge: |doesn't work |Invalidation of Varnish | |thumbnail cache sometimes | |doesn't work? | --- Comment #76 from Andre Klapper aklap...@wikimedia.org --- Mid-air collision with RobLa, anyway: (In reply to comment #69) The server admin log says multicast issues fixed (yesterday). So are they fixed or not? See https://en.wikipedia.org/w/index.php?title=Wikipedia:Village_pump_%28technical%29oldid=535338671#Latest_update_from_Operations (In reply to comment #61) This may be a cross post (see bug 31680) but to be honest I have no idea why this was split into multiple bugs. This was already explained in comment 2: The original posting (comment 0) covered three different issues. In general, bug reports should only be about ONE issue, otherwise they become messy and unfixable. Like this report now. The numbers of the already existing bug reports for the other two issues that you face(d) were provided in comment 2 and 3. The remaining issue (1) that this report was left to be about became ?action=purge on the file description page no longer regenerates thumbnails Comment 18 mentions a workaround for this specific problem. Issue (2) about reuploading isn't properly purging thumbnails is supposed to be bug 31680, but recent postings mixed this all together in this report again. Please use bug 31680. Plus comments 67 and 71 then added all reuploads failing, not only thumbnails. Different issue, unrelated to thumbnails. This needs a separate report. Reverting today's summary change which broadened the topic. Side note speculation: When the scope of a report becomes very blurry I can understand if developers unassign themselves. Please try to file separate bug reports for problems differing from the issue that a bug report is about, otherwise reports like this one become impossible fo fix. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #77 from TMg mr.h...@gmx.de --- (In reply to comment #76) The original posting (comment 0) covered three different issues. Two. The 404 issue was split into a separate bug and fixed. Everything else is the same issue. A purge is a purge no matter if it is triggered by an ?action=purge or by reuploading a file. From what I know both do the same. Both fail randomly for over a year now. Comment 75 explains this very well (thank you very much). His point 1 (Squid/varnish cache contains images that aren't in Swift) is not a separate issue. It is an effect of point 2 (Multicast purge requests don't make it to all of the Varnish Squid caches). Bug 41130 was created because bug 31680 was assigned to the wrong developers. It became clear that this is not a MediaWiki bug. However bug 31680 was not closed or reassigned because there *is* something the MediaWiki developers should do. They should implement a workaround to help us break out of that deadlock situation. See comment 16 and several others. No one ever replied to that (maybe because it should be moved to bug 31680). Five days ago bug 31680 was reassigned. Now it is a duplicate. needs a separate report. I will not create more reports with the same issue. In the past they were all closed as duplicates. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #78 from Nemo federicol...@tiscali.it --- (In reply to comment #77) needs a separate report. I will not create more reports with the same issue. In the past they were all closed as duplicates. I think this: (In comment #75) I'm going to leave this issue open for now, but at this point, but once the other issues get filed, I plan to close this issue and use the other bug reports to track this. means Rob or others at WMF will split this bug in more bugs as they feel needed/best to track and address the specific issues. Probably, given comment 75, it's something like: A) a bug for comment 34 (migration to varnish) if one doesn't exist, B) one to ask and investigate mass workarounds/fixes as in comment 18, C) for multicast purge requests failures etc., one or more bugs for what's deemed viable in a-d options comment 9 + the last two paragraphs of it (Leslie is working on the last one, if I understand correctly). Indeed this is a bit too confused for random users like us to do. :-) -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #79 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to comment #77) (In reply to comment #76) The original posting (comment 0) covered three different issues. Two. The 404 issue was split into a separate bug and fixed. Everything else is the same issue. A purge is a purge no matter if it is triggered by an ?action=purge or by reuploading a file. From what I know both do the same. Both fail randomly for over a year now. Comment 75 explains this very well (thank you very much). His point 1 (Squid/varnish cache contains images that aren't in Swift) is not a separate issue. It is an effect of point 2 (Multicast purge requests don't make it to all of the Varnish Squid caches). What? No. Not sending a purge at all is completely different from the purge being lost along the way. Different causes and different symptoms. Of course without the multicast issue the local file cache would not be so out of sync and as a result the squid/varnish cache has things not in swift bug would appear very rarely to the point where it probably wouldn't be noticed. Bug 41130 was created because bug 31680 was assigned to the wrong developers. It became clear that this is not a MediaWiki bug. However bug 31680 was not closed or reassigned because there *is* something the MediaWiki developers should do. They should implement a workaround to help us break out of that deadlock situation. See comment 16 and several others. No one ever replied to that (maybe because it should be moved to bug 31680). There is a work around (albeit an unintuitive one). The reason it didnt work (at the time comment 16 was made) was because (in my not entirely substantiated opinion) this bug was misdiagonosed as being about files in swift being out of sync with files in varnish when actually the bug was due to varnishhtcpd being borked. Five days ago bug 31680 was reassigned. Now it is a duplicate. needs a separate report. I will not create more reports with the same issue. In the past they were all closed as duplicates. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #80 from Rrius rrius...@gmail.com --- For some reason the file referred to in bug 44269 is mostly corrected, but the 2000px PNG is showing a version of the file from July 2012. Ten uploads have been made since then: three in September; one each in October, November, and December; and four in January. I'm not sure whether that fits the borked varnishhtcpd theory because I don't know what that means. File: https://commons.wikimedia.org/wiki/File:41st_Can_Senate.svg 2000px version: http://upload.wikimedia.org/wikipedia/commons/thumb/4/41/41st_Can_Senate.svg/2000px-41st_Can_Senate.svg.png -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #81 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to comment #80) For some reason the file referred to in bug 44269 is mostly corrected, but the 2000px PNG is showing a version of the file from July 2012. Ten uploads have been made since then: three in September; one each in October, November, and December; and four in January. I'm not sure whether that fits the borked varnishhtcpd theory because I don't know what that means. That's the old issue. The current issue is something else (something to do with udp multicast probably involving the tunnel between data centers. Its kind of unclear the precise cause afaik) However at first glance the thumb seems consistent with neither (because adding ?Randomstring to the end of the url doesnt seem to show most recent which should happen for all presented theories) which is intetesting. Need to test with something better than my cell phone ;) File: https://commons.wikimedia.org/wiki/File:41st_Can_Senate.svg 2000px version: http://upload.wikimedia.org/wikipedia/commons/thumb/4/41/41st_Can_Senate.svg/ 2000px-41st_Can_Senate.svg.png -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #82 from MZMcBride b...@mzmcbride.com --- (In reply to comment #75) The workaround for this problem is in comment 18. The rest of your post looked fine, but this part (referring to Derk-Jan's suggestion to use thumb.php)... God help me, are we really going to encourage this thumb.php hack? This doesn't seem remotely sane. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Nemo federicol...@tiscali.it changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=44269 --- Comment #83 from Nemo federicol...@tiscali.it --- (In reply to comment #82) (In reply to comment #75) The workaround for this problem is in comment 18. The rest of your post looked fine, but this part (referring to Derk-Jan's suggestion to use thumb.php)... God help me, are we really going to encourage this thumb.php hack? This doesn't seem remotely sane. At least it helps debugging: bawolff has just de-duped bug 44269, and in that case thumb.php doesn't work; it also doesn't work in some cases where there are size rounding problems, anyway (or was that fixed in MediaWiki?). -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #66 from TMg mr.h...@gmx.de --- (In reply to comment #62) Sounds like different root cause (since esams still uses squid instead of varnish afaik). I don't know why the title of the bug was changed to something with Varnish. Originally the bug was about invalidation not working when re-uploading files from *Germany*. If there are multiple problems (which is obviously the case) you should split this into multiple bugs. Here is an other example from today: File description page: http://commons.wikimedia.org/wiki/File:Laser_mirror_reflection.jpg Broken garbage including the original size (!): http://upload.wikimedia.org/wikipedia/commons/8/82/Laser_mirror_reflection.jpg http://upload.wikimedia.org/wikipedia/commons/thumb/8/82/Laser_mirror_reflection.jpg/170px-Laser_mirror_reflection.jpg How it should look: http://upload.wikimedia.org/wikipedia/commons/thumb/8/82/Laser_mirror_reflection.jpg/120px-Laser_mirror_reflection.jpg Tracert is the same as above. This is so damn annoying. What's the problem with fixing this bug? There must be *thousands* and *thousands* of examples. It's reported by dozens of users in almost every discussion page in every project: http://commons.wikimedia.org/wiki/Commons:Forum#Falsche_Bildversion http://de.wikipedia.org/wiki/Wikipedia_Diskussion:Fotowerkstatt http://de.wikipedia.org/wiki/Wikipedia:Fragen_zur_Wikipedia#Bildaktualisierung_schl.C3.A4gt_fehl_trotz_.E2.80.9E.3Faction.3Dpurge.E2.80.9C -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #67 from Marco maic...@yahoo.com --- All(!) reuploads failing. Please fix this because it is impossible to update the latest version of any file. * https://commons.wikimedia.org/wiki/File:Muelle_Monroe,_Chicago,_Illinois,_Estados_Unidos,_2012-10-20,_DD_05.jpg * ... -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 TMg mr.h...@gmx.de changed: What|Removed |Added Priority|High|Immediate Severity|critical|blocker -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added Priority|Immediate |Highest CC||lc...@wikimedia.org Severity|blocker |critical --- Comment #68 from Andre Klapper aklap...@wikimedia.org --- TMg: This is neither immediate nor a blocker as it does not block reading articles (or such). See http://www.mediawiki.org/wiki/Bugzilla/Fields#Priority for some information. Reverting. (In reply to comment #66) I don't know why the title of the bug was changed to something with Varnish. Likely because it describes the bug better. Varnish is an HTTP accelerator which routes web site requests to appropriate clusters (load balancing). It is used by Wikimedia. See http://en.wikipedia.org/wiki/Varnish_%28software%29 for more information. What's the problem with fixing this bug? See comment 64. It was explained there. There must be *thousands* and *thousands* of examples. The problem here is not missing examples, the problem is finding reasons. I am going to quote some more info from Leslie here in case that comment 64 wasn't sufficient. I've added a few explanatory words for some terms to hopefully make this more understandable: The problem is about multicast requests to send HTCP purge requests to group 239.128.0.112 port 4827. The requests arrive at a machine in Tampa (dobson) where they are put through a multicast to unicast relay to send them to our European AS for cache purges. Until earlier this week, these notices were sourced from the old Tampa datacenter as well, so if this problem existed, it was masked. This week, we switched them to be sourced from the new eqiad datacenter and then discovered that the traffic for group 239.128.0.112 (and ONLY group 239.128.0.112) was not being delivered. [...] In addition, if we look at the multicast routing on cr1-sdtpa.wikimedia.org and cr2-pmtpa.wikimedia.org, it does not seem to be seeing the sources of traffic for 239.128.0.112, even though it is fine for all of the other groups. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #69 from Bawolff (Brian Wolff) bawolff...@gmail.com --- The server admin log says multicast issues fixed (yesterday). So are they fixed or not? Your quote from Leslie is unclear in that regard ( furthermore, out of curiosity where is that quote from? Somewhere public or secret ops mailing list?) In any case the continued reports suggest there are still outstanding issues. In regards to priority - while it doesnt prevent reading I would point out that wiki is hawiain for quick, not for thirty days or so until entries fall out of cache. The frustration building in regards to this bug is understandable. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Richard Guk richardg...@yahoo.com changed: What|Removed |Added CC||richardg...@yahoo.com -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Jonathan Dodd jonathand...@gmail.com changed: What|Removed |Added CC||jonathand...@gmail.com --- Comment #70 from Jonathan Dodd jonathand...@gmail.com --- I'm afraid I can't offer any insight into solutions, but having run into this today it was requested on IRC that I post this example, with screenshots (to make it real for y'all? :) http:// i48.tinypic.com/2rnkh89.jpg http://pruebita.com/commons-thumbnail-looks-fine-for-me-2013-01-27.png -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #71 from Jonathan Dodd jonathand...@gmail.com --- (In reply to comment #70) Sorry, in case it isn't clear: the screenshots show the file simultaneously not working for me yet working for somebody else. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #64 from Nemo federicol...@tiscali.it --- (In reply to comment #63) If all the new reuploads (upload of a new version) are failing to be visible when accessing in europe (via esams) my initial guess would be that the udp multicast tunnel was forgotten about when moving data centers ( that's pure speculation though. Ive done no investigation and didnt even look at the docs to see if htcp multicast at wmf works like I think it does). LeslieCarr was investigating on it when you wrote this. * 01:21 LeslieCarr: htcp purging across datacenters now works. dobson is now receiving purge requests on multicast group 239.128.0.112 port 4827 and transmitting them via udpmcast.py (started by rc.local) to hooft in esams * 01:14 LeslieCarr: deactivating multicast for 1 minute in order to try and flush the multicast forwarding table https://wikitech.wikimedia.org/index.php?title=Server_admin_logdiff=55822oldid=55819diffonly=yes The most recent reported reuploads seem to work now? https://commons.wikimedia.org/w/index.php?title=Commons:Village_pumpoldid=88876606#Trouble_with_uploading_new_versions -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #65 from TMg mr.h...@gmx.de --- Reuploads from today, January 26, 12:00 o'clock still failing. http://commons.wikimedia.org/wiki/File:Wappen_Landkreis_Aurich.svg -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #61 from TMg mr.h...@gmx.de --- This may be a cross post (see bug 31680) but to be honest I have no idea why this was split into multiple bugs. It's the same problem over and over again. Here is a fresh example from the German Wikipedia: File description page: http://de.wikipedia.org/wiki/Datei:Radiobuttons.gif Broken garbage: http://upload.wikimedia.org/wikipedia/de/d/dc/Radiobuttons.gif http://upload.wikimedia.org/wikipedia/de/thumb/d/dc/Radiobuttons.gif/120px-Radiobuttons.gif How it should look: http://upload.wikimedia.org/wikipedia/de/d/dc/Radiobuttons.gif?dummy Tracert: Routenverfolgung zu upload-lb.esams.wikimedia.org [91.198.174.234] über maximal 30 Abschnitte: [I removed the first few steps] 853 ms52 ms51 ms ge0-1-0-cr0.ixf.de.as6908.net [80.81.192.244] 958 ms57 ms56 ms te3-4-3502-cr0.nik.nl.as6908.net [78.41.154.17] 1056 ms54 ms56 ms ge-2-2.br1-knams.wikimedia.org [78.41.155.38] 1156 ms56 ms56 ms ve7.te-8-1.csw1-esams.wikimedia.org [91.198.174.250] 1256 ms58 ms64 ms upload-lb.esams.wikimedia.org [91.198.174.234] -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #62 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to comment #61) This may be a cross post (see bug 31680) but to be honest I have no idea why this was split into multiple bugs. It's the same problem over and over again. Here is a fresh example from the German Wikipedia: File description page: http://de.wikipedia.org/wiki/Datei:Radiobuttons.gif Broken garbage: http://upload.wikimedia.org/wikipedia/de/d/dc/Radiobuttons.gif http://upload.wikimedia.org/wikipedia/de/thumb/d/dc/Radiobuttons.gif/120px- Radiobuttons.gif How it should look: http://upload.wikimedia.org/wikipedia/de/d/dc/Radiobuttons.gif?dummy Tracert: Routenverfolgung zu upload-lb.esams.wikimedia.org [91.198.174.234] über maximal 30 Abschnitte: [I removed the first few steps] 853 ms52 ms51 ms ge0-1-0-cr0.ixf.de.as6908.net [80.81.192.244] 958 ms57 ms56 ms te3-4-3502-cr0.nik.nl.as6908.net [78.41.154.17] 1056 ms54 ms56 ms ge-2-2.br1-knams.wikimedia.org [78.41.155.38] 1156 ms56 ms56 ms ve7.te-8-1.csw1-esams.wikimedia.org [91.198.174.250] 1256 ms58 ms64 ms upload-lb.esams.wikimedia.org [91.198.174.234] Sounds like different root cause (since esams still uses squid instead of varnish afaik). -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #63 from Bawolff (Brian Wolff) bawolff...@gmail.com --- If all the new reuploads (upload of a new version) are failing to be visible when accessing in europe (via esams) my initial guess would be that the udp multicast tunnel was forgotten about when moving data centers ( that's pure speculation though. Ive done no investigation and didnt even look at the docs to see if htcp multicast at wmf works like I think it does). -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #60 from Douglas Johnson johnson487...@excite.com --- Thanks Brian--the workaround works great! -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Bug 41130 depends on bug 43448, which changed state. Bug 43448 Summary: varnishhtcpd occasionally stops responding to HTCP requests https://bugzilla.wikimedia.org/show_bug.cgi?id=43448 What|Removed |Added Status|NEW |RESOLVED Resolution|--- |FIXED -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #55 from Tim Starling tstarl...@wikimedia.org --- I wrote: (In reply to comment #42) I confirmed that HTCP delivery is working using tcpdump on all Varnish servers in the cp1022-1036 range, and I checked one of them with purgeList.php to ensure that the HTCP CLR message was correctly acted on. I'm not sure how this is possible, since from my analysis and testing of varnishhtcpd today, it appears to have been completely broken since October 25. It should work better now, although there may be some packet loss. I changed varnishhtcpd to be single-threaded, but it's threatening to exhaust a single CPU. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #56 from Micki mailmi...@gmail.com --- Problem on sr.wp is now fixed. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Nemo federicol...@tiscali.it changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=31680 -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #57 from Douglas Johnson johnson487...@excite.com --- The problem with http://en.wikipedia.org/wiki/Wikipedia:Graphics_Lab/Map_workshop#Map_thumbnail_problem still remains, but it's been slowly correcting itself. When I view the thumbnails on the page http://en.wikipedia.org/wiki/File:USA_Wisconsin_GSUSA_council_boundaries.png, they all render correctly now except the 585px thumbnail. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #58 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to comment #57) The problem with http://en.wikipedia.org/wiki/Wikipedia:Graphics_Lab/ Map_workshop#Map_thumbnail_problem still remains, but it's been slowly correcting itself. When I view the thumbnails on the page http://en.wikipedia.org/wiki/File:USA_Wisconsin_GSUSA_council_boundaries.png, they all render correctly now except the 585px thumbnail. The 585px thumb should be fixed now (I used the workaround of going to the thumb's url, adding some junk to the end of the url to bypass varnish (ex http://upload.wikimedia.org/wikipedia/en/thumb/5/53/USA_Wisconsin_GSUSA_council_boundaries.png/585px-USA_Wisconsin_GSUSA_council_boundaries.png?bypasscache )[in order to ensure there's a copy on the server so that mediawiki knows to send purge for that size] and then did ?action=purge on http://en.wikipedia.org/wiki/File:USA_Wisconsin_GSUSA_council_boundaries.png?action=purge . -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #59 from Rrius rrius...@gmail.com --- Bawolff marked 44269 as a duplicate of this one, but I don't see why. This bug is about thumbs not displaying properly. Bug 44269 is about the main image not updating. Thumbs for the file that prompted 44269 actually work fine, both on the file version list and when the image is used with the thumb parameter on the English Wikipedia. Unfortunately, the file is mostly used in infoboxes, so the fact thumbs work properly doesn't help. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Bawolff (Brian Wolff) bawolff...@gmail.com changed: What|Removed |Added CC||rrius...@gmail.com --- Comment #54 from Bawolff (Brian Wolff) bawolff...@gmail.com --- *** Bug 44269 has been marked as a duplicate of this bug. *** -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Nemo federicol...@tiscali.it changed: What|Removed |Added CC||federicol...@tiscali.it See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=43449 Depends on||43448 --- Comment #53 from Nemo federicol...@tiscali.it --- (In reply to comment #51) (In reply to comment #50) I'm not sure is this a right place. We have a problem on sr.wp to restore our official logo after holidays ( https://sr.wikipedia.org/wiki/File:Wiki.png ). I tried everything, but it didn't work. Any ideas? Sounds like this bug. If its the same cause as last time somebody with access to the servers just has to restart the program (varnishhtcpd) that makes sure old versions of images go away. I don't know about restarting, but the problem was split to bug 43448. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Micki mailmi...@gmail.com changed: What|Removed |Added CC||mailmi...@gmail.com --- Comment #50 from Micki mailmi...@gmail.com --- I'm not sure is this a right place. We have a problem on sr.wp to restore our official logo after holidays ( https://sr.wikipedia.org/wiki/File:Wiki.png ). I tried everything, but it didn't work. Any ideas? -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130
--- Comment #51 from Bawolff (Brian Wolff) bawolff...@gmail.com ---
(In reply to comment #50)
I'm not sure is this a right place. We have a problem on sr.wp to restore our
official logo after holidays ( https://sr.wikipedia.org/wiki/File:Wiki.png
). I
tried everything, but it didn't work. Any ideas?
Sounds like this bug. If its the same cause as last time somebody with access
to the servers just has to restart the program (varnishhtcpd) that makes sure
old versions of images go away.
If its critical for your wiki to make the logo go back to normal now, you could
put some css in mediawiki:common.css (important remember to remove after this
bug is fixed as it could cause problems with cache clearing in the future. Note
havent tested this but it should work)
#p-logo a {background-image:
url(//upload.wikimedia.org/wikipedia/sr/b/bc/Wiki.png?1); }
--
You are receiving this mail because:
You are watching all bug changes.
___
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #52 from Micki mailmi...@gmail.com --- Thanks Brian. It's not so critical. We can wait a few days. But I'll try that css customization if the problem remains. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Douglas Johnson johnson487...@excite.com changed: What|Removed |Added CC||johnson487...@excite.com --- Comment #49 from Douglas Johnson johnson487...@excite.com --- This seems to be another instance of this problem: http://en.wikipedia.org/wiki/Wikipedia:Graphics_Lab/Map_workshop#Map_thumbnail_problem (or maybe bug 31680?) -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #47 from Bawolff (Brian Wolff) bawolff...@gmail.com --- reports at commons of http://upload.wikimedia.org/wikipedia/commons/8/85/Complexe_sonore.png (as well as its thumbs) not purging. Notes should be a darker green colour ( compare with http://upload.wikimedia.org/wikipedia/commons/8/85/Complexe_sonore.png?nocache ). Tested from North America. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #48 from Marco maic...@yahoo.com --- (In reply to comment #47) Tested from North America. works fine in Germany -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #46 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to comment #44) It seems in retrospect like bug 42963 should not have been closed as a duplicate of this one, since the time that bug 42963 was filed could have been when varnishhtcpd stopped responding to HTCP requests. The problem that we're tracking with this bug is any problem related to the fact that we currently don't have a mechanism for purging *all* thumbnails associated with a given original. We may need to close this issue and open a new one to avoid having the same problem again. Reading over this bug - comment 2 suggests this was always the varnishhtcpd issue imho. While the cant purge all thumbs is a potential issue, without the other issue I imagine it is rather rare and even rarer that it would get noticed, although I have no evidence to back that assertion up. Furthermore appending action=purge to thumb url and then to image desc page would probably fix it, which users may try on their own. Obviously it should be fixed if possible but I don't think it is as bad as it has been made out to be -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=43345 -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #45 from TMg mr.h...@gmx.de --- (In reply to comment #42) There is no reason to think the problem is limited to North America. It is not. Apparently the problem is not that purging doesn't work, but that on specific occasions at some time in the past, it didn't work. Debugging that is a very different kind of problem to debugging a complete and current failure. I know that and I'm sorry. I found comment #15 very helpful. (In reply to comment #44) we currently don't have a mechanism for purging *all* thumbnails associated with a given original. As suggested above several times (e.g. in comment #16) I would like to add ?action=purge to a specific thumbnail URL. In my opinion this would be a very helpful workaround to purge such broken thumbnails. From what I understand this is easy to implement. Possibly two lines of code in thumb.php (one being a redirect) and maybe a [qsappend] in an Apache configuration. Could you please take a look and tell us if it's possible to add this feature? Thank you. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Tim Starling tstarl...@wikimedia.org changed: What|Removed |Added Summary|North america upload caches |Invalidation of Varnish |aren't responding to squid |thumbnail cache sometimes |purges |doesn't work --- Comment #42 from Tim Starling tstarl...@wikimedia.org --- (In reply to comment #9) This isn't really easy to prevent. One can do a few things: a) Change SquidPurgeClientPool [...] We don't use SquidPurgeClientPool, we use HTCP. The bug summary is North america upload caches aren't responding to squid purges. This is confusing since: * Upload uses Varnish, not Squid. * The Varnish caches in question are in fact receiving and responding to HTCP CLR messages. * There is no reason to think the problem is limited to North America. I confirmed that HTCP delivery is working using tcpdump on all Varnish servers in the cp1022-1036 range, and I checked one of them with purgeList.php to ensure that the HTCP CLR message was correctly acted on. Apparently the problem is not that purging doesn't work, but that on specific occasions at some time in the past, it didn't work. Debugging that is a very different kind of problem to debugging a complete and current failure. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 Betacommand phoenixoverr...@gmail.com changed: What|Removed |Added Blocks|41371 | -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #43 from Bawolff (Brian Wolff) bawolff...@gmail.com --- (In reply to comment #42) The bug summary is North america upload caches aren't responding to squid purges. This is confusing since: You're right, that was a poor choice for bug title. Sorry about that. Apparently the problem is not that purging doesn't work, but that on specific occasions at some time in the past, it didn't work. Debugging that is a very different kind of problem to debugging a complete and current failure. I can confirm that previous cases I tested doing ?action=purge on image description page which had no effect (if accessing upload.wikimedia.org via 208.80.154.235) now work correctly. I had forgotten to access this anonymously. It shouldn't matter if you're anonymous for images. (And furthermore login cookies aren't sent to upload.wikimedia.org so the servers wouldn't even know if you're an anon) -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 41130] Invalidation of Varnish thumbnail cache sometimes doesn't work
https://bugzilla.wikimedia.org/show_bug.cgi?id=41130 --- Comment #44 from Rob Lanphier ro...@wikimedia.org --- Asher restarted varnishhtcpd on all of Eqiad (North American caching center) machines about 30 minutes ago (simultaneously with Tim investigating this problem). Thus, probably the most urgent of the problems associated with this issue has been solved. Asher also spotted and solved another problem, where 404'd images would get cached for 30 days. He fixed that problem as well with Gerrit #40762. There's another problem that he spotted that isn't fixed yet, which I've filed as bug 43448. Until that bug is fixed, we may have to occasionally kick varnishhtcpd. It seems in retrospect like bug 42963 should not have been closed as a duplicate of this one, since the time that bug 42963 was filed could have been when varnishhtcpd stopped responding to HTCP requests. The problem that we're tracking with this bug is any problem related to the fact that we currently don't have a mechanism for purging *all* thumbnails associated with a given original. We may need to close this issue and open a new one to avoid having the same problem again. -- You are receiving this mail because: You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
