Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On 24/05/11 23:32, Thomas Morton wrote: So, just a quick thought for future reference - during maintenance is it possible in future to update the error message to explain that maintenance is ongoing? Seeing as how widely WMF projects are used by a non-technical project the current MySQL connection error I am seeing on Commons is just going to cause confusion :) And the standard error page WIkipedia was showing a minute ago is not particularly helpful/explanatory in this specific situation. Database connection errors were not an anticipated consequence of the scheduled router upgrades. The people who might have been able to change the error message were busy diagnosing and fixing the problem. When we have a lengthy period of downtime, more sysadmins arrive online, and a wider perspective on the problem develops, including attention to community impact and communication. But since the downtime in this case was only half an hour, there was not enough time for this to happen. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
I don't get this. Would it be possible in future, if the sites are unresponsive, or will be unresponsive due to planned maintenance, to establish a fallback that simply displays an explanatory status message to the public? FT2 On Wed, May 25, 2011 at 8:15 AM, Tim Starling tstarl...@wikimedia.orgwrote: (snip) The people who might have been able to change the error message were busy diagnosing and fixing the problem. When we have a lengthy period of downtime, more sysadmins arrive online, and a wider perspective on the problem develops, including attention to community impact and communication. But since the downtime in this case was only half an hour, there was not enough time for this to happen. ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On 25/05/11 17:32, FT2 wrote: I don't get this. Would it be possible in future, if the sites are unresponsive, or will be unresponsive due to planned maintenance, to establish a fallback that simply displays an explanatory status message to the public? You mean replace the entire site with an error page? But only part of the site was down. More and more things became accessible as each database server was fixed. I'm not sure how this could work. Even if we did prepare an error message saying Wikipedia will be down for 2 minutes while a router restarts, I don't think that could be called explanatory if it were displayed for half an hour. Writing informative error messages and displaying them in appropriate places is necessarily a low-priority task during downtime, the higher priority task being to get the site working again. Maybe at some time in the future, we will have enough 24/7 sysadmin manpower that we can respond to any unplanned downtime in the way you suggest. But we don't have that capability just yet. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
I think it's reasonable (and indeed standard) to deploy some sort of downtime maintenance error message. If that requires improving the error handling code to catch a wider variety of errors and push people to the error message page then I understand the time issues :). If the short term solution is that the error page that kept appearing gets tweaked (before the maintenance is started) to explain what is happening, then that seems fine. IRC was flooded with people who didn't understand what was going on. And many didn't believe/understand that it was maintenance... so this is definitely an area worth improving. I'm not trying to criticise; just passing on some ideas based on the issues raised. Tom On 25 May 2011 08:56, Tim Starling tstarl...@wikimedia.org wrote: On 25/05/11 17:32, FT2 wrote: I don't get this. Would it be possible in future, if the sites are unresponsive, or will be unresponsive due to planned maintenance, to establish a fallback that simply displays an explanatory status message to the public? You mean replace the entire site with an error page? But only part of the site was down. More and more things became accessible as each database server was fixed. I'm not sure how this could work. Even if we did prepare an error message saying Wikipedia will be down for 2 minutes while a router restarts, I don't think that could be called explanatory if it were displayed for half an hour. Writing informative error messages and displaying them in appropriate places is necessarily a low-priority task during downtime, the higher priority task being to get the site working again. Maybe at some time in the future, we will have enough 24/7 sysadmin manpower that we can respond to any unplanned downtime in the way you suggest. But we don't have that capability just yet. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
priority task being to get the site working again. Maybe at some time in the future, we will have enough 24/7 sysadmin manpower that we can respond to any unplanned downtime in the way you suggest. But we don't have that capability just yet. In future we will have five nines availability and no downtimes will happen. Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
In future can I have vanilla and strawberry with that? :) FT2 On Wed, May 25, 2011 at 9:16 AM, Domas Mituzas midom.li...@gmail.comwrote: In future we will have five nines availability and no downtimes will happen. ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On Wed, May 25, 2011 at 9:32 AM, FT2 ft2.w...@gmail.com wrote: I don't get this. Would it be possible in future, if the sites are unresponsive, or will be unresponsive due to planned maintenance, to establish a fallback that simply displays an explanatory status message to the public? Would it have changed anything for you? I tried to load Wikipedia a few times during the downtime, and a maintenance error actually did appear most of the time. I did get a few database errors, but I assumed that I wasn't the first to notice and that someone was diligently working on it. Regardless, my action was the same as it would have been in any case: try back later. Austin ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Austin, That's interesting, what was the wording for the maintenance message? I only ever saw the default our servers are experiencing a technical problem error page. Tom On 25 May 2011 10:53, Austin Hair adh...@gmail.com wrote: On Wed, May 25, 2011 at 9:32 AM, FT2 ft2.w...@gmail.com wrote: I don't get this. Would it be possible in future, if the sites are unresponsive, or will be unresponsive due to planned maintenance, to establish a fallback that simply displays an explanatory status message to the public? Would it have changed anything for you? I tried to load Wikipedia a few times during the downtime, and a maintenance error actually did appear most of the time. I did get a few database errors, but I assumed that I wasn't the first to notice and that someone was diligently working on it. Regardless, my action was the same as it would have been in any case: try back later. Austin ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On Wed, May 25, 2011 at 11:57 AM, Thomas Morton morton.tho...@googlemail.com wrote: That's interesting, what was the wording for the maintenance message? I only ever saw the default our servers are experiencing a technical problem error page. I could be misremembering, because I honestly didn't care that much, but I do believe I saw the word maintenance in there somewhere. Either way, it was as informative as any message could be under the circumstances—unless, as Tim already addressed, you wanted a developer assigned to updating the message in real time. Austin ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
unless, as Tim already addressed, you wanted a developer assigned to updating the message in real time. No, definitely not what was being suggested. This is the error message that appeared for me (and apparently others): http://nomulous.com/blog/wp-content/uploads/2009/09/wikipedia_error.png As you can see it refers to some unknown error. In this case the maintentance was known and* pre-planned* for several days. A lot of people were confused by the outage and the error page was unhelpful to them. This could have been mitigated simply by editing that page temporarily to say Our servers are undergoing scheduled maintenance, which has resulted in some downtime. This should be concluded by 14:00 UTC, please be patient whilst the maintenance progesses. And this is the extent of my suggestion to improve our communication with readers. Tom On 25 May 2011 11:02, Austin Hair adh...@gmail.com wrote: On Wed, May 25, 2011 at 11:57 AM, Thomas Morton morton.tho...@googlemail.com wrote: That's interesting, what was the wording for the maintenance message? I only ever saw the default our servers are experiencing a technical problem error page. I could be misremembering, because I honestly didn't care that much, but I do believe I saw the word maintenance in there somewhere. Either way, it was as informative as any message could be under the circumstances—unless, as Tim already addressed, you wanted a developer assigned to updating the message in real time. Austin ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
As you can see it refers to some unknown error. In this case the maintentance was known and* pre-planned* for several days. technically this was unknown problem :) A lot of people were confused by the outage and the error page was unhelpful to them. This could have been mitigated simply by editing that page temporarily to say Our servers are undergoing scheduled maintenance, which has resulted in some downtime. This should be concluded by 14:00 UTC, please be patient whilst the maintenance progesses. We did not really know when we will fix it :) And this is the extent of my suggestion to improve our communication with readers. IMO we're discussing completely wrong things here. Site was down, doesn't really matter in what way ;-) I'm sure we'd look much more professional if our downtime message would always say planned maintenance in process! ;-) Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Huh? The downtime was expected during 13:00 and 14:00 UTC, or at least there was an email warning of such things the day before... hardly unplanned or unknown. Tom On 25 May 2011 11:12, Domas Mituzas midom.li...@gmail.com wrote: As you can see it refers to some unknown error. In this case the maintentance was known and* pre-planned* for several days. technically this was unknown problem :) A lot of people were confused by the outage and the error page was unhelpful to them. This could have been mitigated simply by editing that page temporarily to say Our servers are undergoing scheduled maintenance, which has resulted in some downtime. This should be concluded by 14:00 UTC, please be patient whilst the maintenance progesses. We did not really know when we will fix it :) And this is the extent of my suggestion to improve our communication with readers. IMO we're discussing completely wrong things here. Site was down, doesn't really matter in what way ;-) I'm sure we'd look much more professional if our downtime message would always say planned maintenance in process! ;-) Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On Wed, May 25, 2011 at 12:09 PM, Thomas Morton morton.tho...@googlemail.com wrote: This is the error message that appeared for me (and apparently others): http://nomulous.com/blog/wp-content/uploads/2009/09/wikipedia_error.png I won't continue arguing about whether or not it should say planned, but I do have to say that I love probably temporary. (That, or Wikipedia has gone offline FOREVER.) Austin ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
It might be more worthwhile to put downtime status updates on status.wikimedia.org as a logical page to display the status of the servers, and link to it from the default error messages. Given that status.wm.org is an external service, it would hopefully not be affected by any outages and the Watchmouse service probably should have the functionality to host informational messages like this and explanations for outages (like appstatus.google.com does) even if only after the fact when the ops team has time to write down what is happening. Best regards, Bence ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Hi! Huh? The downtime was expected during 13:00 and 14:00 UTC, or at least there was an email warning of such things the day before... hardly unplanned or unknown. there's a bit of a difference between maintenance window and expected downtime during it. Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
The maintenance was planned, downtime was noted as possible. An error message that reflects that seems, frankly, a good idea. The response to what I thought to be a helpful suggestion in improving communication with readership has been... incredibly disappointing. I wish I hadn't bothered. :( I was just passing on comments from people who came to IRC and basically said oh, well why didn't the site just say that then. Of course; if we are ignoring our readers' concerns now, then fine. Tom On 25 May 2011 11:20, Domas Mituzas midom.li...@gmail.com wrote: Hi! Huh? The downtime was expected during 13:00 and 14:00 UTC, or at least there was an email warning of such things the day before... hardly unplanned or unknown. there's a bit of a difference between maintenance window and expected downtime during it. Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Hi! The maintenance was planned, downtime was noted as possible. An error message that reflects that seems, frankly, a good idea. There're lots of great ideas around the world, feeding the hungry and curing the cancer among them. The response to what I thought to be a helpful suggestion in improving communication with readership has been... incredibly disappointing. Well, you were complaining about confusion at first, probably we indeed should not show any technical details about anything. Site is down, bye! might be better choice, I guess. I wish I hadn't bothered. :( I was just passing on comments from people who came to IRC and basically said oh, well why didn't the site just say that then. If we knew what would fail to put an appropriate error message there, we'd probably fix the problem beforehand. :-) Of course; if we are ignoring our readers' concerns now, then fine. Nobody is ignoring any concerns, they are carefully weighted, hehehe, unlike your negativism. Cheers, Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
If we knew what would fail to put an appropriate error message there, we'd probably fix the problem beforehand. :-) That's... completely missing the point. Yes the specific errors faced were unexpected or unforseen, BUT they were a* direct result* of the maintenance between 13:00 and 14:00. I am simply passing on the feeling of our readership; which was that the situation was badly communicated to them. I am trying to share my experience here as a sysadmin and website operator; users hate downtime/maintenance, and will complain about it endlessly. Improving our communication of planned maintenance is definitely a good idea. Nobody is ignoring any concerns, they are carefully weighted, hehehe, unlike your negativism. I'm trying to be positive, but it seems to simply be dismissed (incorrectly) as well we didn't know what was going to happen. Tom On 25 May 2011 11:33, Domas Mituzas midom.li...@gmail.com wrote: Hi! The maintenance was planned, downtime was noted as possible. An error message that reflects that seems, frankly, a good idea. There're lots of great ideas around the world, feeding the hungry and curing the cancer among them. The response to what I thought to be a helpful suggestion in improving communication with readership has been... incredibly disappointing. Well, you were complaining about confusion at first, probably we indeed should not show any technical details about anything. Site is down, bye! might be better choice, I guess. I wish I hadn't bothered. :( I was just passing on comments from people who came to IRC and basically said oh, well why didn't the site just say that then. If we knew what would fail to put an appropriate error message there, we'd probably fix the problem beforehand. :-) Of course; if we are ignoring our readers' concerns now, then fine. Nobody is ignoring any concerns, they are carefully weighted, hehehe, unlike your negativism. Cheers, Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Tim, When I originally wrote: during maintenance is it possible in future to update the error message to explain that maintenance is ongoing? That was a bit of a silly moment from me :) I see how that implies in-maintenance updates. In fact my suggestion was to update the error message to mention the planned maintenance and the timeframes. Sorry for the confusion! Tom On 25 May 2011 08:15, Tim Starling tstarl...@wikimedia.org wrote: On 24/05/11 23:32, Thomas Morton wrote: So, just a quick thought for future reference - during maintenance is it possible in future to update the error message to explain that maintenance is ongoing? Seeing as how widely WMF projects are used by a non-technical project the current MySQL connection error I am seeing on Commons is just going to cause confusion :) And the standard error page WIkipedia was showing a minute ago is not particularly helpful/explanatory in this specific situation. Database connection errors were not an anticipated consequence of the scheduled router upgrades. The people who might have been able to change the error message were busy diagnosing and fixing the problem. When we have a lengthy period of downtime, more sysadmins arrive online, and a wider perspective on the problem develops, including attention to community impact and communication. But since the downtime in this case was only half an hour, there was not enough time for this to happen. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Domas, what are you trying to achieve with your comments on Tom's suggestions? He just said that if we know that maintenance is done and could cause outages we should put up an error message that informs the reader about the maintenance work and tells him not to worry. That's obviously a good thing. The sensible reaction (from a person who is involved in the maintenance) would be: Oh, sorry, we were so much occupied with making the maintenance work as smooth and uninterruptive as possible that we totally didn't think about that. We will integrate it into our flow charts so we won't forget it the next time we need to do maintenance that could cause outages. Everything else is not very goal-oriented. Marcus Buck User:Slomox ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] No rights to participate
That does not mean that there are not isolated cases of injustice. Such users need to patiently and persistently bring their situation to the attention of the community. Fred Could some consensus be reached on this matter? 6) Fred Bauder might also be willing to fill you with the details of how extremely helpful he has been to me. Permission is granted to make public all my e-mail messages to him, showing my appreciation for his good deeds. Any further questions? Sincerely, Virgilio A. P. Machado The underlying dispute is on the Portuguese Wikipedia. Fred ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Hi! That's... completely missing the point. Yes the specific errors faced were unexpected or unforseen, BUT they were a* direct result* of the maintenance between 13:00 and 14:00. I am simply passing on the feeling of our readership; which was that the situation was badly communicated to them. As majority of our users are anons, who visit us once a day or two, we should probably have started a communication campaign at least two months before the maintenance. We practice a lot during fundraisers :-) OTOH, if there's no downtime, maybe we're causing quite some frustration with superfluous communication? :-) I am trying to share my experience here as a sysadmin and website operator; Oh, finally we got some sysadmins and website operators here. As a sysadmin you sure understand that in larger distributed systems which are not all built on a set of SPOFs there can be various failure modes, happening at various layers and various fuzziness. As a website operator you sure know that it is lots of effort to prepare boilerplates for every possible situation :-) users hate downtime/maintenance, and will complain about it endlessly. You have some annoying users, our users are awesome and don't complain endlessly! Improving our communication of planned maintenance is definitely a good idea. So is curing cancer. Marcus Buck wrote: Domas, what are you trying to achieve with your comments on Tom's suggestions? Put some clue in? The sensible reaction (from a person who is involved in the maintenance) would be: I know nobody likes this, but sensible reaction is to work on good operation rather than standing in front of a mirror and trying five hundred different I'm sorry phrases. You look too much from that single position, that communication is good, without weighting costs or other options. Cheers, Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On 25/05/11 18:14, Thomas Morton wrote: IRC was flooded with people who didn't understand what was going on. And many didn't believe/understand that it was maintenance... so this is definitely an area worth improving. Maybe we can replace the IRC link in the Squid error message with a link to the WatchMouse page (status.wikimedia.org). That would reduce the IRC flood. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Tim Starling wrote: Maybe we can replace the IRC link in the Squid error message with a link to the WatchMouse page (status.wikimedia.org). That would reduce the IRC flood. * https://bugzilla.wikimedia.org/show_bug.cgi?id=16043 * https://bugzilla.wikimedia.org/show_bug.cgi?id=20079 MZMcBride ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
m...@marcusbuck.org wrote: The sensible reaction (from a person who is involved in the maintenance) would be: Oh, sorry, we were so much occupied with making the maintenance work as smooth and uninterruptive as possible that we totally didn't think about that. We will integrate it into our flow charts so we won't forget it the next time we need to do maintenance that could cause outages. I'm kind of surprised that you think Wikimedia has flow charts for this kind of thing. MZMcBride ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Maybe we can replace the IRC link in the Squid error message with a link to the WatchMouse page @Tim; that seems a good idea. @Domas, I'm afraid you don't seem to have understood the premise of my suggestion.. which is fine. But one fallacy is worth responding to: You have some annoying users, our users are awesome and don't complain endlessly! The first rule of a website people use regularly is: users will complain endlessly One of my business mentors has a good maxim about this: Just because you can't see them complaining, don't simply assume they are not. Because they are. Twitter, Facebook, IRC and all sorts of other websites had people complaining about the down time. That is just a fact of life :) To wit: If that static error page cannot easily be changed prior to a maintenance, then fine :) no worries Tom On 25 May 2011 12:10, Domas Mituzas midom.li...@gmail.com wrote: Hi! That's... completely missing the point. Yes the specific errors faced were unexpected or unforseen, BUT they were a* direct result* of the maintenance between 13:00 and 14:00. I am simply passing on the feeling of our readership; which was that the situation was badly communicated to them. As majority of our users are anons, who visit us once a day or two, we should probably have started a communication campaign at least two months before the maintenance. We practice a lot during fundraisers :-) OTOH, if there's no downtime, maybe we're causing quite some frustration with superfluous communication? :-) I am trying to share my experience here as a sysadmin and website operator; Oh, finally we got some sysadmins and website operators here. As a sysadmin you sure understand that in larger distributed systems which are not all built on a set of SPOFs there can be various failure modes, happening at various layers and various fuzziness. As a website operator you sure know that it is lots of effort to prepare boilerplates for every possible situation :-) users hate downtime/maintenance, and will complain about it endlessly. You have some annoying users, our users are awesome and don't complain endlessly! Improving our communication of planned maintenance is definitely a good idea. So is curing cancer. Marcus Buck wrote: Domas, what are you trying to achieve with your comments on Tom's suggestions? Put some clue in? The sensible reaction (from a person who is involved in the maintenance) would be: I know nobody likes this, but sensible reaction is to work on good operation rather than standing in front of a mirror and trying five hundred different I'm sorry phrases. You look too much from that single position, that communication is good, without weighting costs or other options. Cheers, Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] No rights to participate
Only a info: V.A.P. Machado was not banned from pt.wiki because people does not like him. He was at frist banned from edit project and talk pages[1] due a ArbCom decision[2], but that could be revise each year[2]. Instead of change his behaivor he created a meat/sock puppet[3] and, since the ArbCom decision says that if he created a sock he would be banned for real, this time he was banned[4]. _ *Béria Lima* http://wikimedia.pt/(351) 925 171 484 *Imagine um mundo onde é dada a qualquer pessoa a possibilidade de ter livre acesso ao somatório de todo o conhecimento humano. É isso o que estamos a fazer.*** [1]: http://pt.wikipedia.org/wiki/Especial:AbuseFilter/history/29/diff/prev/262http://pt.wikipedia.org/wiki/Especial:AbuseFilter/history/29/item/160 [2]: http://pt.wikipedia.org/wiki/Wikip%C3%A9dia:Conselho_de_arbitragem/Casos/2009-09-01_Virg%C3%ADlio_A._P._Machado [3]: http://pt.wikipedia.org/wiki/Wikip%C3%A9dia:Pedidos_a_verificadores/Arquivo/2010/04#Carmo_Cunha [4]: http://pt.wikipedia.org/w/index.php?title=Especial:Registotype=blockpage=Usuário%3AVapmachado 2011/5/25 Fred Bauder fredb...@fairpoint.net That does not mean that there are not isolated cases of injustice. Such users need to patiently and persistently bring their situation to the attention of the community. Fred Could some consensus be reached on this matter? 6) Fred Bauder might also be willing to fill you with the details of how extremely helpful he has been to me. Permission is granted to make public all my e-mail messages to him, showing my appreciation for his good deeds. Any further questions? Sincerely, Virgilio A. P. Machado The underlying dispute is on the Portuguese Wikipedia. Fred ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On Wed, May 25, 2011 at 4:40 PM, Domas Mituzas midom.li...@gmail.comwrote: Hi! That's... completely missing the point. Yes the specific errors faced were unexpected or unforseen, BUT they were a* direct result* of the maintenance between 13:00 and 14:00. I am simply passing on the feeling of our readership; which was that the situation was badly communicated to them. As majority of our users are anons, who visit us once a day or two, we should probably have started a communication campaign at least two months before the maintenance. We practice a lot during fundraisers :-) OTOH, if there's no downtime, maybe we're causing quite some frustration with superfluous communication? :-) I am trying to share my experience here as a sysadmin and website operator; Oh, finally we got some sysadmins and website operators here. As a sysadmin you sure understand that in larger distributed systems which are not all built on a set of SPOFs there can be various failure modes, happening at various layers and various fuzziness. As a website operator you sure know that it is lots of effort to prepare boilerplates for every possible situation :-) users hate downtime/maintenance, and will complain about it endlessly. You have some annoying users, our users are awesome and don't complain endlessly! Improving our communication of planned maintenance is definitely a good idea. So is curing cancer. Marcus Buck wrote: Domas, what are you trying to achieve with your comments on Tom's suggestions? Put some clue in? The sensible reaction (from a person who is involved in the maintenance) would be: I know nobody likes this, but sensible reaction is to work on good operation rather than standing in front of a mirror and trying five hundred different I'm sorry phrases. You look too much from that single position, that communication is good, without weighting costs or other options. Cheers, Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l I have no idea what Domas is trying to say. I agree with Thomas that there should be a better option to communicate with users about downtime and possible performance issues. I don't know how one would expect a user to discern between a planned downtime for maintenance vs. actual performance issues. There has been several issues earlier this year with performance and even temporary outages, not to mention there might have been more pronounced performance issues in certain locations. Instead of diverting users to IRC, how about an outage/error page with a twitter/identi.ca feed with updates from the tech team, or at least a page with customized message in case of previously planned outage. Most of the tech staff already use Twitter/Identi.ca to update users, maybe we can look for a way to incorporate that feed in the outage page itself or point them to it. How would someone who is not on any of the mailing lists, or has suppressed the banners supposed to find out about the difference between these issues? Theo User:Theo10011 ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Theo10011 wrote: Instead of diverting users to IRC, how about an outage/error page with a twitter/identi.ca feed with updates from the tech team, or at least a page with customized message in case of previously planned outage. Most of the tech staff already use Twitter/Identi.ca to update users, maybe we can look for a way to incorporate that feed in the outage page itself or point them to it. Is it so much to ask that you read the mailing list thread before replying? Nobody's asking you to memorize every word, but having some general idea of what has been discussed would make your replies less redundant and/or seemingly obtuse. MZMcBride ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On Wed, May 25, 2011 at 5:31 PM, MZMcBride z...@mzmcbride.com wrote: Theo10011 wrote: Instead of diverting users to IRC, how about an outage/error page with a twitter/identi.ca feed with updates from the tech team, or at least a page with customized message in case of previously planned outage. Most of the tech staff already use Twitter/Identi.ca to update users, maybe we can look for a way to incorporate that feed in the outage page itself or point them to it. Is it so much to ask that you read the mailing list thread before replying? Yes! hehyou expect me to read Bugzilla? Nobody's asking you to memorize every word, but having some general idea of what has been discussed would make your replies less redundant and/or seemingly obtuse. More Noise. MZMcBride ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On Wed, May 25, 2011 at 10:09 PM, Theo10011 de10...@gmail.com wrote: On Wed, May 25, 2011 at 5:31 PM, MZMcBride z...@mzmcbride.com wrote: Theo10011 wrote: Instead of diverting users to IRC, how about an outage/error page with a twitter/identi.ca feed with updates from the tech team, or at least a page with customized message in case of previously planned outage. Most of the tech staff already use Twitter/Identi.ca to update users, maybe we can look for a way to incorporate that feed in the outage page itself or point them to it. Is it so much to ask that you read the mailing list thread before replying? Yes! hehyou expect me to read Bugzilla? Where did Mz ever suggest to read Bz... I only see mention to reading what was already suggested in this email thread. -Peachey ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
What I understood from this thread is: if you have a planned maintenance windows between 13 and 14 GMT, it would be appreciated if you could: - create a simple page that says: We are working on our servers between 13 and 14 GMT and Wikipedia might be unavailable during that time - replace the usual error message with the newly created page as close as possible to 12:59 - reinstate the usual error message at 14:01 (or whenever the maintenance ends) Nobody (of the millions of anonymous users) really cares about whether a certain db server is down or up at 13:49, or some router is rebooting at 13:23. They just wanna know when they can come back to read about spark plugs (sic!). AFAIK, this is the way big websites like Yahoo do it. It seems like a simple thing to do, so perhaps you could explain calmly and without ironies where is the difficulty? Strainu 2011/5/25 Domas Mituzas midom.li...@gmail.com: Hi! That's... completely missing the point. Yes the specific errors faced were unexpected or unforseen, BUT they were a* direct result* of the maintenance between 13:00 and 14:00. I am simply passing on the feeling of our readership; which was that the situation was badly communicated to them. As majority of our users are anons, who visit us once a day or two, we should probably have started a communication campaign at least two months before the maintenance. We practice a lot during fundraisers :-) OTOH, if there's no downtime, maybe we're causing quite some frustration with superfluous communication? :-) I am trying to share my experience here as a sysadmin and website operator; Oh, finally we got some sysadmins and website operators here. As a sysadmin you sure understand that in larger distributed systems which are not all built on a set of SPOFs there can be various failure modes, happening at various layers and various fuzziness. As a website operator you sure know that it is lots of effort to prepare boilerplates for every possible situation :-) users hate downtime/maintenance, and will complain about it endlessly. You have some annoying users, our users are awesome and don't complain endlessly! Improving our communication of planned maintenance is definitely a good idea. So is curing cancer. Marcus Buck wrote: Domas, what are you trying to achieve with your comments on Tom's suggestions? Put some clue in? The sensible reaction (from a person who is involved in the maintenance) would be: I know nobody likes this, but sensible reaction is to work on good operation rather than standing in front of a mirror and trying five hundred different I'm sorry phrases. You look too much from that single position, that communication is good, without weighting costs or other options. Cheers, Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On 25/05/11 21:19, MZMcBride wrote: Tim Starling wrote: Maybe we can replace the IRC link in the Squid error message with a link to the WatchMouse page (status.wikimedia.org). That would reduce the IRC flood. * https://bugzilla.wikimedia.org/show_bug.cgi?id=16043 * https://bugzilla.wikimedia.org/show_bug.cgi?id=20079 Maybe it's time to completely rewrite it. I noticed Austin joking about the awkward probably temporary phrasing, and the idea that we need to buy new hardware to avoid downtime is a bit dated. The source is in Subversion, at /trunk/debs/squid/debian/errors. Maybe if someone proposed some new text on meta.wikimedia.org, I could see that it gets included in the next Squid update. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Tim, Great, thanks for that. Seeing as it was me that raise this ;) I guess it's only right I take up the gauntlet, so will try and find time later to propose something. Tom On 25 May 2011 13:48, Tim Starling tstarl...@wikimedia.org wrote: On 25/05/11 21:19, MZMcBride wrote: Tim Starling wrote: Maybe we can replace the IRC link in the Squid error message with a link to the WatchMouse page (status.wikimedia.org). That would reduce the IRC flood. * https://bugzilla.wikimedia.org/show_bug.cgi?id=16043 * https://bugzilla.wikimedia.org/show_bug.cgi?id=20079 Maybe it's time to completely rewrite it. I noticed Austin joking about the awkward probably temporary phrasing, and the idea that we need to buy new hardware to avoid downtime is a bit dated. The source is in Subversion, at /trunk/debs/squid/debian/errors. Maybe if someone proposed some new text on meta.wikimedia.org, I could see that it gets included in the next Squid update. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On 25/05/11 22:27, Strainu wrote: What I understood from this thread is: if you have a planned maintenance windows between 13 and 14 GMT, it would be appreciated if you could: - create a simple page that says: We are working on our servers between 13 and 14 GMT and Wikipedia might be unavailable during that time - replace the usual error message with the newly created page as close as possible to 12:59 - reinstate the usual error message at 14:01 (or whenever the maintenance ends) There are dozens of places where error messages are generated. It's not trivial to replace them all. Some of them are hard-coded in compiled binaries, some are on the client side. The error message in question comes from DBConnectionError in Database.php in the MediaWiki source. It's hard-coded and the source would have had to have been patched. Since no database problems were anticipated, even if we had tried to implement your plan, we wouldn't have thought to patch Database.php, and the result would have been the same. Nobody (of the millions of anonymous users) really cares about whether a certain db server is down or up at 13:49, or some router is rebooting at 13:23. They just wanna know when they can come back to read about spark plugs (sic!). There was no way to tell when the site was going to be back up, except perhaps after the problem was isolated and the fix was halfway through being implemented. But by that time there was only a few minutes of downtime left. The maintenance window was 13:00 to 14:00, but after things went wrong, there was no guarantee that all problems would be fixed by 14:00. Indeed, if it wasn't for Domas's help as a volunteer sysadmin, the problem may have lasted much longer. Then there would have been plenty of time for messaging and maybe we wouldn't be having this conversation. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Me - no. Readers who didn't know - yes. Wikipedia going down without a temporary explanation page is roughly of the same scale as apple.com going down with no explanation, google.com going down with no explanation, microsoft.com going down with no explanation, and so on. Top 5 website means we have that kind of use, perception, stature -- and a similar scale of response within the general public if it suddenly doesn't work. Most members of the public do not have the insight you or I would. FT2 On Wed, May 25, 2011 at 10:53 AM, Austin Hair adh...@gmail.com wrote: On Wed, May 25, 2011 at 9:32 AM, FT2 ft2.w...@gmail.com wrote: I don't get this. Would it be possible in future, if the sites are unresponsive, or will be unresponsive due to planned maintenance, to establish a fallback that simply displays an explanatory status message to the public? Would it have changed anything for you? ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
2011/5/25 Tim Starling tstarl...@wikimedia.org: On 25/05/11 22:27, Strainu wrote: What I understood from this thread is: if you have a planned maintenance windows between 13 and 14 GMT, it would be appreciated if you could: - create a simple page that says: We are working on our servers between 13 and 14 GMT and Wikipedia might be unavailable during that time - replace the usual error message with the newly created page as close as possible to 12:59 - reinstate the usual error message at 14:01 (or whenever the maintenance ends) There are dozens of places where error messages are generated. It's not trivial to replace them all. Some of them are hard-coded in compiled binaries, some are on the client side. [...] I kind of anticipated that response, but it's nice to have it written somewhere. I think it is now clear for everybody why there is a need for more sysadmins and/or developers to handle such issues. I believe much of this thread could have been avoided (and this means less time wasted writing emails for you guys) if you had stated that in your first email. Strainu ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
As a non-tech, don't all reads (at least) pass through the squids, so we can identify and report in a nice way a lot of connection errors at that point? /ignoreifnaive FT2 On Wed, May 25, 2011 at 2:18 PM, Tim Starling tstarl...@wikimedia.orgwrote: There are dozens of places where error messages are generated. It's not trivial to replace them all. Some of them are hard-coded in compiled binaries, some are on the client side. The error message in question comes from DBConnectionError in Database.php in the MediaWiki source. It's hard-coded and the source would have had to have been patched. Since no database problems were anticipated, even if we had tried to implement your plan, we wouldn't have thought to patch Database.php, and the result would have been the same. ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Just conceptualising... I haven't played with Squid for a while (so am rusty) but the simplest solution would probably be to catch all PHP errors somewhere in the Mediawiki code and return a 500 status error code. Then get Squid to map that to the static error page. On the other hand throwing a catch any sort of error into an application isn't good practice. As Tim points out, errors can generate from all over the place and it is better to catch them explicitly. So that would be a non-trivial process. Tom On 25 May 2011 14:41, FT2 ft2.w...@gmail.com wrote: As a non-tech, don't all reads (at least) pass through the squids, so we can identify and report in a nice way a lot of connection errors at that point? /ignoreifnaive FT2 On Wed, May 25, 2011 at 2:18 PM, Tim Starling tstarl...@wikimedia.org wrote: There are dozens of places where error messages are generated. It's not trivial to replace them all. Some of them are hard-coded in compiled binaries, some are on the client side. The error message in question comes from DBConnectionError in Database.php in the MediaWiki source. It's hard-coded and the source would have had to have been patched. Since no database problems were anticipated, even if we had tried to implement your plan, we wouldn't have thought to patch Database.php, and the result would have been the same. ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Wikipedia going down without a temporary explanation page is roughly of the same scale as apple.com going down with no explanation, google.com going down with no explanation, microsoft.com going down with no explanation, and so on. WHOAH THERE IS QUITE SOME SELF ENTITLEMENT THERE. Microsoft revenue: $62B (though you should look at their internet division losses) Google revenue: $29B Apple revenue: $62B Wikimedia revenue: ??? Tech staffing and such is somewhat proportional :) Oh, by the way, I don't know where you look, but I somewhat missed communication about maintenance events ongoing in Google or Microsoft or Apple - you think they have none? Did you get lots of clarification why your gmail was unreachable? Did you get explanation/information why search index was outdated? Do they use site-wide sitenotices for that or what? Top 5 website means we have that kind of use, perception, stature -- and a similar scale of response within the general public if it suddenly doesn't work. Most members of the public do not have the insight you or I would. *shrug*, would be interesting if anyone would actually explain policies of other website incident handling. Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On 25/05/11 23:41, FT2 wrote: As a non-tech, don't all reads (at least) pass through the squids, so we can identify and report in a nice way a lot of connection errors at that point? /ignoreifnaive Maybe it would be possible to identify error messages by their HTTP response code, and replace the body with some other text, presumably with the original text embedded somehow for debugging purposes. But I don't think Squid has such a feature, and we have very little development time to spend on this sort of thing. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Domas, why so defensive? No one accused you of anything or blamed you for the downtime. The comments suggesting more finely-tuned error messages weren't critical of you or Tim or the developers in general, they were just (reasonable) suggestions. Maybe adjusting all the various error messages in anticipation of possible downtime is totally unfeasible because of the work involved, but you can probably say that without all the combative snark. ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Is the Squid configuration the foundation employs available publicly somewhere (I'm scanning the SVN and not seeing it..)? Because I don't mind having a look and filing a specific bugzilla correction with various bits of code changes. It's about time I refreshed my Squid knowledge :) Tom On 25 May 2011 14:58, Tim Starling tstarl...@wikimedia.org wrote: On 25/05/11 23:41, FT2 wrote: As a non-tech, don't all reads (at least) pass through the squids, so we can identify and report in a nice way a lot of connection errors at that point? /ignoreifnaive Maybe it would be possible to identify error messages by their HTTP response code, and replace the body with some other text, presumably with the original text embedded somehow for debugging purposes. But I don't think Squid has such a feature, and we have very little development time to spend on this sort of thing. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On 25 May 2011 09:50, Domas Mituzas midom.li...@gmail.com wrote: Oh, by the way, I don't know where you look, but I somewhat missed communication about maintenance events ongoing in Google or Microsoft or Apple - you think they have none? Did you get lots of clarification why your gmail was unreachable? Did you get explanation/information why search index was outdated? Do they use site-wide sitenotices for that or what? Ummyes, actually. My Gmail produces an error code or gives me advance notice when there is scheduled maintenance, as does my hotmail (Microsoft), and Google fairly frequently explains its technical problems (though sometimes one has to look for it). Apple - I know nothing. And I'm realistic enough not to expect that level of service from Wikimedia; there's simply not the personnel to do it. I think we all appreciate, Domas, that notifying customers is not the #1 priority when our excellent team of paid and volunteer developers are fighting a pitched battle with wayward squids - all of us know getting the system working is the top priority, and anyone who's sat back and watched wikimedia-tech during a serious problem knows how incredibly diligent and focused you all are. Wiki(p)(m)edians who forget what collaborative work means should watch you folks when you're taking care of the serious business for a free lesson. It would be worthwhile, however, during a relatively quiet period to tweak the error messages (perhaps make them more generic and all purpose?). There are some useful ideas, particularly Tim's, in this thread, and it appears Thomas has volunteered to do much of the heavy lifting on it. Thanks to you and to all of the team who worked to address this situation yesterday, you did a good job. I know we don't say that nearly often enough. Risker/Anne ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Hi! Domas, why so defensive? I'm contrarian in this case :) unfeasible because of the work involved, but you can probably say that without all the combative snark. Well, as with every downtime, there are way more issues* that end up uncovered and have to be looked at, and yet largest email threads are about nicer error messages :-) This will be my constructive contribution to the thread: FAIL WHALE! W W W WW W W '. W .--._ \ \.--| / -..__) .-' | _ / \'-.__, .__.,' `''._\--' V Domas * buggy forcedeth behavior, european DNS server was hanging before maintenance started, loadbalancer likes to throw errors on first slave failure and doesn't go to others, no auto-fallback to read-only mode, too long connect timeouts, etc ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
@Tim: Understood, I'll make sure I know this will work first so as not to generate work for you. My initial idea might not be so workable given the architecture used (and how Squid handles error codes). I'll roll up some servers here at work and run some tests. @Domos; echoing what Risker said... The intent wasn't to criticise your work :) just to try constructively suggest improvements to something of minor importance. Sorry if that intent got lost somewhere in my messages. As you say; more critical issues appear to have cropped up internally in the ops team. Don't hold our ignorance of these things against us! We're just trying to contribute where we can. :) Tom On 25 May 2011 15:26, Tim Starling tstarl...@wikimedia.org wrote: On 26/05/11 00:05, Thomas Morton wrote: Is the Squid configuration the foundation employs available publicly somewhere (I'm scanning the SVN and not seeing it..)? Because I don't mind having a look and filing a specific bugzilla correction with various bits of code changes. It's about time I refreshed my Squid knowledge :) No, it's private. I can give it to you if you want it, just give me your assurance offlist that it'll be worth my time to clean any private data out of it and tar it up. -- Tim Starling ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Zitat von MZMcBride z...@mzmcbride.com: m...@marcusbuck.org wrote: The sensible reaction (from a person who is involved in the maintenance) would be: Oh, sorry, we were so much occupied with making the maintenance work as smooth and uninterruptive as possible that we totally didn't think about that. We will integrate it into our flow charts so we won't forget it the next time we need to do maintenance that could cause outages. I'm kind of surprised that you think Wikimedia has flow charts for this kind of thing. I'm not a native speaker of English and I don't know whether it was the right word. I meant a documentation that tells you which steps need to be taken before, while and after you touch a critical system. Marcus Buck User:Slomox ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects ...
In a message dated 5/25/2011 3:33:57 AM Pacific Daylight Time, midom.li...@gmail.com writes: There're lots of great ideas around the world, feeding the hungry and curing the cancer among them. Domas your responses are not helpful at all. You are simply stirring the pot to no point. Please stop. ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects ...
Hi! Domas your responses are not helpful at all. You are simply stirring the pot to no point. Please stop. You forgot to tell if all of my responses or just some, and if there's really no point at all, or there might be some. Anyway, thanks for this helpful contribution! Domas ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects ...
In a message dated 5/25/2011 11:01:24 AM Pacific Daylight Time, midom.li...@gmail.com writes: You forgot to tell if all of my responses or just some, and if there's really no point at all, or there might be some. Anyway, thanks for this helpful contribution! Refactoring my comments : Domas some or at least one of your comments are not helpful at all. (Although some or even most of your other comments are helpful.) Will ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
On Tue, May 24, 2011 at 6:32 AM, Thomas Morton morton.tho...@googlemail.com wrote: So, just a quick thought for future reference - during maintenance is it possible in future to update the error message to explain that maintenance is ongoing? I work with lots of (library) databases, and standard practice for these services is to display messages across the top or some other visible space warning of scheduled maintenance ahead of time. I guess we could do that with centralnotice, but /goes back to reading centralnotice thread :-) -- phoebe ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] No rights to participate
I'm sorry for not being as brilliant as you are, but I have read my message over and over and can't find any always there. I haven't made any mention of the number of active editors. I don't know what you mean when you say that obviously there are some cases where we can see things went badly. Nobody ever said that about what you and others like you have done to me. You can't miss the step from that to if someone... can turn things from bad to worse for you, they will. Just retrace your own steps. Please point an occasion, a message you haven't use to take another stab at me. For what purpose? I know. Doesn't anybody here? I never made any racist comments, and it saddened me very deeply that you found it appropriate to use that as an example, therefore associating me with that kind of behavior. I do not use the kind of language that you so proudly display, again in a despicable attempt to associate me with the kind of people who do. I wonder how such a nice person such as yourself can resort to that kind of behavior and be so welcome and so highly regarded. That's why you can't be my buddy, pal, friend and why you do have so much trouble feeling any kind of empathy. Nor do most people here. That is [...] why other people aren't agreeing with you very much either, or [standing up] to your support here. My apologies to the list for the personal tone of this message, but I believe things were getting a bit out of hand. Sincerely, Virgilio A. P. Machado At 09:47 25-05-2011, you wrote: Yeah. It's news for me. You really need to be careful not to replace sometimes by always. We have a few hundreds of thousands of active editors over time. So obviously there are some cases where we can see things went badly. What I'm missing is the step from that to if someone... can turn things from bad to worse for you, they will. A bit of a difference. This is the kind of logic that (in racists) goes: A dark skinned man nudged me on the street and didn't say sorry all dark skinned people are rude Dark skinned people will fuck you over if they can. Sorry, but I don't buy it. Nor do most people here. That is possibly why other people aren't agreeing with you very much either, or jumping to your support here. FT2 On Wed, May 25, 2011 at 6:30 AM, Virgilio A. P. Machado v...@fct.unl.ptwrote: Oh my! That's news for you? Let's see. Just a sample from firsthand experiences. 1) From Meta: http://meta.wikimedia.org/wiki/User_talk:Vapmachado#Updated_request_for_assistance The work on Meta was being done in an orderly manner until the disruption provoked and caused by those same people mentioned above. The user is the same. Trouble only started after the interference of the same people from the Portuguese Wikipedia on Meta. Their votes can be seen popping up on the RfA. There has never been a single block on any other Wikimedia project where these editors do not have any influence. The obvious conclusion is that the hostile behavior stays with that people, not this user. 2) From Wikimedia Outreach: http://human-rights-in-cyberspace.wikia.com/wiki/The_Outreach_debacle Please edit those pages as though they were your own wiki. Make yourself at home on the Outreach wiki. Wrote Lennart. Wow! I was in awe. This project and/or these guys had the right stuff. When I revisited Wikimedia Outreach, my user page had been deleted, my own name suppressed from my message and replaced by (Redacted). Later, my user page was restored with this quite amazing summary: restoring per request, it appears this user intended to out himself, removing personal address. It was decided that Apartado 313, 2826-801 Caparica is my personal address. Well I regret to have to let you know that Apartado 313, 2826-801 Caparica is not my personal address. It's one of my many mailing addresses. Apartado is the Portuguese word for Post Office or P.O. Box, and I can assure you that I never lived there. From the Portuguese Wikipedia: http://human-rights-in-cyberspace.wikia.com/wiki/The_crusaders_against_education_I 3) http://human-rights-in-cyberspace.wikia.com/wiki/The_crusaders_against_education_I#Melhoria_de_predefini.C3.A7.C3.A3o Em engenharia, quando a obra não é executada de acordo com o projecto é um caso sério. Na Wikipédia chama-se «ajudar». «Em Roma, sê Romano.» É assim. Uma pessoa põe o seu «espírito criativo» a funcionar, é «ajudada» e pronto, lá se foi o que planeou para o «galheiro». Já se tinha agradecido, portanto, é tocar para diante e esperar por melhores dias ou que o «ajudante» vá de férias. Não há nada que um vulgar editor faça que um atento administrador, burocrata e membro do conselho de arbitragem não possa desfazer. Nesta Wikipédia o que não falta são ajudas. Se alguém encontrar por aí a definição de «período para discutir o mérito da página», agradece-se desde já. 4) http://human-rights-in-cyberspace.wikia.com/wiki/The_crusaders_against_education_I#O_Emplastro_I With all the
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects ...
Let's just drop it :) I'm not sure where things went so south but I take full responsibility. I've pinged Tim off-list about contributing my own time to work on the error page matter - which I think is only fair enough given that I raised it. And sorry for any offence caused to the ops team by my suggestion. Tom p.s. I appreciate Tim is busy so maybe someone else can answer: if I have a substantial technical proposal to re-work the error page process where is the best place to post it for comment? On 25 May 2011 19:08, wjhon...@aol.com wrote: In a message dated 5/25/2011 11:01:24 AM Pacific Daylight Time, midom.li...@gmail.com writes: You forgot to tell if all of my responses or just some, and if there's really no point at all, or there might be some. Anyway, thanks for this helpful contribution! Refactoring my comments : Domas some or at least one of your comments are not helpful at all. (Although some or even most of your other comments are helpful.) Will ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
[Foundation-l] Open Decentralized Society
Open Decentralized Society is a project that aims at creating a Wikipedia that takes trust relations between users into consideration. It wants to create a wiki that will have different articles for each subject, articles that would be ranked based on the person who is viewing them and his trust relations. Each trust link will also have semantic meaning. Each person will try to certificate which person is a doctor, which person is a software engineer, and for each semantic meaning there will be a different graph. It is important to note that there will not be a global ranking. You could think of it as if everyone is a seed of trust for himself. This project is in its infancy. My intention is to create a community that will implement those features on mediawiki. If you are a freedom proponent. If you believe that the flow of information should be free, join me in creating the next Wikipedia. The project needs funding, peer-reviewing, programmers, security experts, lawyers. Don't hesitate to contact me: xekoukou#gmail.com http://opensociety.referata.com -- This message and any attachments (the message) are confidential, intended solely for the addressee(s), and may contain legally privileged information. Any unauthorised use or dissemination is prohibited. E-mails are susceptible to alteration. I shall not be liable for the message if altered, changed or falsified. Sincerely yours, Apostolis Xekoukoulotakis ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
Re: [Foundation-l] Scheduled intermittent downtime on all Wikimedia projects on May 24
Domas Mituzas wrote: FAIL WHALE! W W W WW W W '. W .--._ \ \.--| / -..__) .-' | _ / \'-.__, .__.,' `''._\--' V http://en.wikipedia.org/wiki/User:MZMcBride/Blame_wheel 3 MZMcBride ___ foundation-l mailing list foundation-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l