Re: [Toolserver-l] Redirects for toolserver.org moved to WMF
On 14-12-11 04:39 AM, Silke Meyer wrote: Coren, am I right in assuming that people contact you directly if there are any changes needed? That is correct. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] What will happen with the Toolserver domain?
On 01/18/2014 08:43 PM, Marc A. Pelletier wrote: > or Confluence on our infrastructure For some reason, I was convinced that Confluence was used as wiki software on Toolserver. Given that this is in fact Mediawiki, keeping a historical copy for historical reasons is relatively simple and quite okay. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] What will happen with the Toolserver domain?
On 01/18/2014 03:29 PM, Dr. Trigon wrote: > I would vote strongly to keep the wiki and also JIRA somewhere > accessible. Both contain a serious amount of history and documentation. There is an issue about both living on non-free platforms we have to address before that's possible (I very much doubt that we can reasonably maintain either Jira or Confluence on our infrastructure). I don't know how /useful/ they would be, but static copies might be appropriate; or we might need to find some other method by which to keep those for history. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] What will happen with the Toolserver domain?
On 12/25/2013 04:57 PM, Manuel Schneider wrote: > We may either keep one simple webserver running and point the old domain > there in order to maintain redirects of the old URLs to the new scripts. > We could also move this late to another server, eg. of Wikimedia CH > which I could provide, so we can decomission the toolserver completely > and save costs and management time while still maintain the redirects. Either way, the WMF with collaborate with whatever is decided in the end. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] What will happen with the Toolserver domain?
On 12/25/2013 03:05 PM, Emilio J. Rodríguez-Posada wrote: > I'm not sure if ALL servers are going to be removed or a basic Apache is > going to run in the domain the next years. Last I heard, the plan is to decommission and donate all the hardware. If WMDE wants, the Foundation can take the domain and keep the redirects up indefinitely so that old links remain valid. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Merry Christmas
On 12/24/2013 10:03 AM, DaB. wrote: > I like to wish you a Merry Christmas. And a happy and prosperous new year to you, DaB. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Redacted database content
On 12/07/2013 10:21 AM, Platonides wrote: > In which way were they non-conformant? Toolserver data was hiding (most) > non-public fields, being very similar to what labs hid (actually, I > remember fields hidden in toolserver available in labs, not the other > way around). What did you remove? Amongst other things, several revision deleted fields were available, as well as a number of suppressed entries. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Redacted database content
On 12/06/2013 10:48 AM, Christian Thiele wrote: > is it possible to have a list of the effected fields? The new views have been modeled after those of the Labs' replicated tables, and should be the same as found in: https://git.wikimedia.org/blob/operations%2Fsoftware/HEAD/maintain-replicas%2Fmaintain-replicas.pl (Look at the definitions starting line 99). IIRC, however, the table /names/ are slightly different on the toolserver; TS's "revision" table matches our "revision_userindex" table, while our "revision" is on the TS as "revision_alternate" or so. Either way, this should give you a good idea of what columns are altered. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] List of the default license of each Toolserver user
On 10/14/2013 05:44 PM, Federico Leva (Nemo) wrote: > With one of my usual silly scripts, here it is: > https://toolserver.org/~nemobis/tsusers/licenses.txt Thanks, this is useful. I see a vast majority of suitably open source things, which is good; also a fair number of erroneous or ambiguous statements that can't be interpreted as being authoritative (less good). A question for legal remains; is the statement to which you've just enumerated the answers to sufficient to apply licensing to things that aren't otherwise clearly indicated? Does anyone have the text of the question handy? -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Need Tool Labs migration support?
On 10/10/2013 12:32 PM, Silke Meyer wrote: > ok, I consulted Nosy, DaB. and Mette about this proposal and we will not > make the home directories accessible to others. They can contain all > sorts of private data, user names, e-mail addresses, ssh keys, who knows > what else - maybe even gpg keys, so handing the complete package over is > no option. I wouldn't want to "intrude" into the people's stuff. And > we'd get in trouble with the law here, as the German data protection act > is binding for us. I agree with this as well, for those very reasons. This is the reason why Tool Labs insists that any "real" tool is run from multi-maintainer project equivalents, and with clear licensing. I would not allow user's own home directories to be turned over to a third party under any circumstances, this is why we want no code running there we might want to salvage in the future. And, likewise, I'm pretty sure the foundation wouldn't /want/ to be given custody over data that may or may not be private from contributors who have not given explicit leave to do so. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Need Tool Labs migration support?
On 10/07/2013 06:27 PM, Magnus Manske wrote: > * Forcibly port the most important, unmaintained tools to Labs Sadly, that's only possible when the code is unambiguously licensed as Open Source, which is often not the case (much of the code running on the Toolserver has no licensing information at all). -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [Toolserver-announce] Security issue in Wikipedia projects
On 10/04/2013 05:34 PM, DaB. wrote: > Now two question: Why does WMF didn’t notice the bounce and why did WMF > not use my SUL-mail-address? And following question 1: How many other > bounces happened without notice? Your second question is easy: the mail was sent to the email address associated with the exposed account. I expect you have that email address still on the project that was on the list, so this is where the email was sent. For your first question: we would notice mail being rejected by the MTA, but not a bounce that came in after the fact. gmx.de did accept the mail for delivery, but sent a bounce asynchronously. Since the from: of the email points to OTRS, and OTRS rejects bounces to avoid starting bounce loops, it got lost. Sadly, we were under severe time pressure to warn as many users as possible as quickly as possible, and it was not practical to construct a mail system that was robust enough to handle edge cases. Since there was a second layer of protection (ending sessions and forcing password changes) that would come into play even for editors that had invalid or no email set, this was viewed as the right compromise to avoid delaying warning users by days. It's of course preferable if editors get the email before they wonder why their session timed out (because, as you yourself experienced, it's a little confusing to end up being forced to change your password without warning) -- but safeguarding the security of users quickly has priority. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [Toolserver-announce] Security issue in Wikipedia projects
On 10/04/2013 04:59 PM, K. Peachey wrote: > Wait, You just released information on a email account that is attached > to a user profile… I have, and it was a goof. This is a known email for DaB, and one which is attached to his public PGP key, so it didn't flag any warnings in my head despite how the association was made. DaB; please accept my apologies if that bugged you -- my intent was obviously to help you debug the issue you had. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [Toolserver-announce] Security issue in Wikipedia projects
On 10/04/2013 12:15 PM, DaB. wrote: > I got no mail, but MediaWiki logged me out and forced me to change my > password (so I guess that I’m affected). Well, the email was indeed sent to you: 2013-10-03 06:57:40 1VRcqy-fo-78 => d...@gmx.de R=wiki_mail T=remote_smtp S=3309 H=wiki-mail.wikimedia.org [208.80.152.133] C="250 OK id=1VRcqy-00086y-Gf" DT=0s 2013-10-03 06:57:40 1VRcqy-fo-78 Completed so I guess it ended up in your spam trap or something? -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [Toolserver-announce] Security issue in Wikipedia projects
On 10/04/2013 10:53 AM, Jeremy Baron wrote: > (not all users on a given DB were actually affected AIUI; some even > had no affected users) Indeed, the fraction of affected users is low even on the affected databases; every account that was affected was sent an email. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] TS and/vs. Labs
On 09/16/2013 04:13 PM, Platonides wrote: labs db server do contain the non-public data. Just to clarify things, here, that's a necessary artifact of the way mediawiki works: many of the bits of data that should be made unavailable are done so conditionally on /live data/. It is not possible to replicate the visible contents of the views and have it not break as column values may be nulled or *come back* depending on actions on the projects (supression, deletion, etc). While some of the more sensitive data never hits the replica (like IP addresses), It is not /possible/ to create a replica without transferring data that can be unsupressed or undeleted -- but might never be. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Setting crontab in toollabs
On 07/27/2013 11:55 AM, Avocato wrote: > In toolserver, we use /cronie -e/ to set a crontab. How could I do the > same in toollabs? 'crontab -e' That said, tool labs related questions are probably best left to lab...@lists.wikimedia.org to avoid spamming this toolserver-specific mailing list. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] My fading out
On 05/30/2013 05:31 PM, DaB. wrote: > that WMF has already won I did not wish to intervene in this thread, DaB, out of respect and admiration for your dedication; but this I cannot let by uncommented. The WMF has not "won" anything, because there was never a contest or battle, nor was there an adversarial position to begin with. My concern - and that of the Foundation - align exactly with yours: provide a good and stable environment for community developers to do their work with the least possible fuss. You sincerely believe that the Toolserver was and is the best solution towards that objective. I disagree, and think that the Foundation has more resources to set up and upkeep that environment and to insure its future. Either way, it's the developer community that "wins", regardless of where the actual environment ends up being. This does not, in any way, diminish the value of what you have done, or of the effort you have expended in doing it. The Toolserver served its purpose very well for a number of years! We have simply reached a point where the continued maintenance of such a critical service living outside the infrastructure remains rational. I am saddened that you felt that the Foundation was an enemy to protect against when we are plainly working towards the same ends. That we are in a position to support the developer community with more resources should be cause to celebrate, not bemoan. The Toolserver deserves a retirement with honors, not a bitter parting. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Status of the toolserver
On 05/13/2013 05:01 PM, DaB. wrote: > The problem is that both ha-nodes run Solaris and all roots are no Solaris- > experts what makes it hard for us to find errors or in this case impossible. There is a former colleague or mine with whom I've kept contact that is a serious high-grade guru with Solaris. Would you like me to put him in contact with you guys? Maybe he can give a hand or lend expertise? -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Migrating to labs from the toolserver
On 05/04/2013 08:18 AM, Magnus Manske wrote: > As I boldly ssh in where few toolserver users have ssh'ed in yet, I try > to keep a log of what I'm doing, what issues arise, and where I had to > stop and scratch my head. That kind of journal tends to prove invaluable to those who would follow in your footsteps, Magnus. Thanks for taking the time and effort to keep it. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver --> Labs
On 04/30/2013 11:10 PM, Jeremy Baron wrote: > I don't know what peter/asher had planned for the new Maria world. Yes, the fact that there are a couple of near-equivalent ways of doing it is part of the reason why we haven't been all that explicit yet -- I want to do some testing first to find the best solution before we encourage people to use it; there's little point in having everyone start doing it one way just to come back a few weeks later to announce "well, we're doing this another way after all". :-) -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver --> Labs
On 04/30/2013 11:02 PM, MZMcBride wrote: > It would help if the Labs folks could better explain > _why_ cross-database joins won't be supported (I think most developers > would agree with the reasoning) and offer better guidance and > documentation for how to work around this hurdle. (For example, what is a > federated table?) That's a very good point, MZM. That said, no small part of the reason we haven't given a great deal of explanation on how the alternatives will work is that we're not entirely certain what for those alternatives will take yet - at least in the detail. I think you're right that the developper community needs at least /some/ explanation sooner rather than later; even if it's a big picture that's lacking in detail. I'll try to write up something this week on that topic. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations
On 04/29/2013 03:12 AM, Maarten Dammers wrote: > Moving to early gives bad user experience and gives disappointed users > who turn their back on it (like me). That may well be the case; I have been careful to invite early adopters who had tools that had full support for now (i.e. do not require DB replication). Current feedback is mostly "It's a bit more complicated to set up, but rock solid." That said, I still have no idea what bad user experience you may have had, or what functionality you needed that was not available to you. Unless you are a bit more explicit about what the issues /were/, it's very difficult to address them. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations
On 04/28/2013 07:15 PM, DaB. wrote: > Even more useful would have been a "We improved since last time, please come > and check again". Well, I /do/ take some pains to post regular updates, on labs-l and here as well. :-) Watch for emails from me (generally on Mondays) on those lists. Also, [1] and [2] are good sources of information and generally kept pretty well up-to-date. I remember Silke and Sumanah having pointed at both at regular interval. > Until now I never found a tool-labs-URL in the wild. Is there a list of what > you are hosting? http://tools.wmflabs.org/ displays a list of publicly visible tools, I know there are a couple more being set up that aren't yet ready for prime time. -- Marc [1] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Needed_Toolserver_features [2] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Thoughts and questions to WMDE members about TS and expectations
On 04/28/2013 04:42 PM, Maarten Dammers wrote: > I tried labs and I hate it, if someone tries to force it on my now I'll > just shut down my tools and leave. Even more useful would be a list of actual issues you've encountered, so that we may address them in the future. That said, "far from ready" is neither fair nor very accurate; unless your tool requires access to the database dumps - which are coming - the tools project /is/ ready and quite functional; and there are already several tools running there. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Tool Labs maintenance (rescheduled)
Hello again, The maintenance has concluded successfully within the designated, and the Tool Labs instance now use the new NFS server for shared filesystems. This doubled as a hard test of the continuous bot start/restart system, since the entire cluster was disabled for rolling periods during the maintenance, and the filesystem on which the actual tools were running has been switch underneath them -- pretty much a worst case scenario to recover from. The result is that all but one tool that had been started as a continuous process restarted cleanly and automatically as the cluster returned to function with the new filesystem (the tool that did not failed to return for an unrelated reason). Thank you all for your patience! -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Tool Labs maintenance (rescheduled)
- "Whaddya say we try that again, huh?" - "Yes, yes. Yes. Without the oops." So, now that I have power and internet again, a reschedule for tomorrow (Thursday) at the same time: === Planned outage === When: Thursday April 25 at 18:00 UTC Duration: 1 hour Impact: * Jobs running on the grid engine will be stopped, and execution nodes will be temporarily disabled; * The login server will be restarted during the window, ending active sessions; * The web service will be unavailable during the maintenance window; and * Running processes not scheduled through the grid engine will be killed. Recovery plan: In case of unplanned failure during the maintenance window, configuration will be rolled back to the current version (that is, the gluster-based project storage will remain in place) and a new window will be planned after postmortem. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Department of the reporting of statuses
What's the worst that could happen? The big news this week is the replacement of Gluster as the provider of shared storage by NFS for the tools project. In doing so, we're going to gain a great deal of extra stability, and quite a bit of performance. In addition, there is now an automatic timetravel snapshot feature allowing users to look at the filesystem as it was during snapshots spanning: once an hour for the past three hours, once a day for the past three days, and two weekly snapshots (on Sundays). That said, NFS is robust but does not scale as much as we would like in the longer term; we will keep investigating clustered storage solutions for the future, with an eye to returning to it once we find a solution that is (a) no less robust than NFS, (b) no less reliable than the current storage and (c) at least as good from a performance standpoint. Technically, this storage is already available to all projects but the configuration necessary to /replace/ gluster with it would generally require a per-project outage. (Involving copying the contents of the previous store to the new one, and substituting one mount point for another -- a process which running processes can generally not cope with). In practice, the tools project will be the first transition "victim", with an outage tomorrow to make the switchover. Sadly, the process is necessarily disruptive and currently running processes will be affected (see below for details). As secondary news, thanks to the unwavering efforts of Asher, the database replication is now at a point where we are actually selecting what, exactly, is going to be replicated. Things are progressing there at a fair clip and we're still sticking pretty close to schedule. More news on that topic next week. === Planned outage === When: Tuesday April 23 at 18:00 UTC Duration: 1 hour Impact: * Jobs running on the grid engine will be stopped, and execution nodes will be temporarily disabled; * The login server will be restarted during the window, ending active sessions; * The web service will be unavailable during the maintenance window; and * Running processes not scheduled through the grid engine will be killed. Recovery plan: In case of unplanned failure during the maintenance window, configuration will be rolled back to the current version (that is, the gluster-based project storage will remain in place) and a new window will be planned after postmortem. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] HTTP outage
On 04/15/2013 09:20 AM, John wrote: > This needs fixed ASAP, its been happening for several days and I have > seen a lot of concern raised about this. I tried asking DaB but he > doesnt know anything about it I've seen Nosy working at it this weekend, I expect she'll give a status update soon. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] When is the best time of day to run programs?
On 04/10/2013 07:51 PM, Tim Landscheidt wrote: > Does Icinga have graphs? I wasn't sure you were talking to me. :-) Icinga have graphs, but they're uptime-related. The place with the pretty graphics is ganglia: http://ganglia.wmflabs.org/latest/?c=tools But there is no queued/running job graph right now. It's a very good idea though, and I'll add the matching instrumentation. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] When is the best time of day to run programs?
On 04/10/2013 02:41 PM, Byrial Jensen wrote: > unless there is some option I can use to tell that. What Tim mean is that, by default, SGE will schedule your job when sufficient resources are effectively available, rather that trying to predict when that will happen. That said, you /can/ specify both a minimal starting time (with -a) and a deadline (with -dl) creating a "window" during which SGE will try to run your job but, in general, it's easier and more reliable to let the gridengine pick the time. If your objective is to have your job run only when few others are trying to use the resources, you can also lower its priority (with -p) so that it will only execute your job when there isn't anything "better" to run. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] A (21) day in the Labs
On 04/08/2013 12:13 PM, Sumana Harihareswara wrote: > Since those dates are now in the past :-) can you help provide new > estimated times of arrival? Thanks! I think those dates are in the past because those bits are complete. :-) As far as I know, we're still on track. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] A (21) day in the Labs
Greetings, Programs! So, not many updates for a while, as things have been progressing at a fair clip in the "oh, my god boring gruntwork" front. The biggest news is the addition of Petr Bena to the tools project sysadmin team as its first volunteer. Petr has been very involved in the setup and administration of the Tool Labs' predecessor projects, and will continue to steer the bots project where the rules are a little more relaxed to facilitate more experimental development. He's also joining me on the tools project proper, to help provide support to maintainers over a wider range of times, and to increase availability of sysadmins. You can find him hanging around #wikimedia-labs, often at times where I am not available. There is some documentation-in-progress that give a lot of information on how to set up your tools on the Labs architecture at: https://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Help Please don't hesitate to comment if you see missing information, or if parts of it are less clear than idea. On the other fronts, the wikitech management interface is now in place for self-serve of tool account creation by Labs users; this requires moving already-existing tool accounts to the new scheme, and a brief outage for that purpose later this week (see note below) Experiments with a bulletproof replacement for gluster are well on their way; with NFS from a highly redundant server as the currently favored option. With a bit of luck, I'll use the opportunity given by the outage for the tool account switchover to move the shared tools filesystem to NFS as a trial run. The database replication is also well on its way; you can find the current roadmap at: https://wikitech.wikimedia.org/wiki/Tool_Labs/Database_plan === Planned outage === In order to move the extant tool accounts to the new, final scheme, and (progress permitting) move the shared filesystems to a new storage server, there will be a brief outage of the Tool Labs infrastructure this Thursday April 11 starting at 16:00 UTC. The outage is expected to last 20 minutes during which service will be intermittently unavailable. Announcements will be sent by email, on IRC and on the servers 30 minutes before the start of maintenance, at its start, and upon completion. Impact: * Jobs running on the grid engine will be stopped then restarted automatically at the end of the maintenance window. If you are running a job that cannot or should not be restarted automatically without intervention from its maintainers, please make certain that it has been stopped before the start of the maintenance window; * The login server will be restarted during the window, ending active sessions; * The web service will be intermittently unavailable; and * Running processes not scheduled through the grid engine will be killed. Recovery plan: In case of unplanned failure during the maintenance window, configuration will be rolled back to the current version and a new window will be planned after postmortem. Disruption of services will take place as noted and an announcement will be sent. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [TS logo] Fwd: Free as in Wikimedia Foundation
On 03/19/2013 11:35 AM, Daniel Schwen wrote: > Sorry, but this is alarmist hippie crap and typical "netizen-outrage". I'm not sure I'd have put it in such strong words, but I agree that this is very much overblown and misguided. It's important that any marks not be misused for "evil" purposes, and Trademark is the method to prevent it. That the foundation is willing to step up and handle the legalities is a /good/ thing. Making sure nobody can use a mark for bad reasons *means* having to okay proposed uses. If you want something "everyone can use without asking for any reason", you'll get something everyone *will* use for reasons you wish they wouldn't -- and then have no way to stop them. -- Coren / Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] This week's update, now with 7.3% more levity!
"Toto, I've a feeling we're not on the toolserver any more." So, things are progressing at a fair clip; there is now a database available to the tools, which means that most tools that do not depend on the presence of the database replicas should be workable in the new environment. Maintainers are welcome to poke me (by email or on IRC) if they want to start playing in the new environment, and I'll be glad to show them around and set them up. My plate this week is going to be occupied mostly with helping people with the new environment, setting up puppet classes to subsume the tools- instances' configuration, and writing documentation aimed at tool maintainers looking to join the project. (The latter being driven a great deal by what the actual maintainers do in fact stumble on). It's a low update week, much of the work this past week and in the next few is going to be in the "boring gruntwork" category; but that's a good thing because it means we're reaching cruising altitude. You may now use your personal electronic devices and move around the cabin; but for safety reasons, the captain requests that you keep your seatbelt on whenever you are at your seat. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Tool Labs update and news, oh my!
On 03/04/2013 01:58 PM, Platonides wrote: A few machines where you install the different tools. So you wouldn't need to manage a complete VM, or install apache for your web app. You would place the tool in the appropiate dir and it_should_ just work. Quite similar to how toolserver worked, but more tool-centric (as if you created a MMP per tool, but easier). That's a very good nutshell. The basic outline is thus: there is a compute grid and a webserver cloud (and, in the future, databases and other common services) that are managed at the project levels by the roots. Those are made available to all the tools. Each "tool" is, essentially, storage, a webspace, a set of permissions, a user, and a group. Members of the group are the tool's maintainers and have complete control over it. They can queue jobs or continuous tasks, and put what they need in their webspace to function. They'll also have permission over future resources (like databases, etc). -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Tool Labs update and news, oh my!
Greetings, programs! This week's update will be brief, if optimistic: On the news front, there have been performance and reliability issues with Gluster than are being worked on by Ryan Lane. He is experimenting with an upcoming network driver and tweaking with automount timing to improve the situation. Along with the database replication, the shared storage is one of the key component of the infrastructure that are going to receive the most attention in the short term. There is now a new project (named, predictably enough, "tools") that is the intended destination of the new architecture for the Tool Labs. We have a functionnal webservice environment, as well as a workable compute cluster to support work and long running processes. Already, a few brave souls have stepped forward to test that new environment; others are welcome to peek in or join the project with the usual "beta" caveats. I'm not ready to open the gates entirely yet, since none of the management is currently automatized (that's a big part of my TODO for the week); but tools which have no dependencies on access to databases should already be able to be experimentally moved to the new project. This week, I plan to concentrate on the first draft of an interface through which tool maintainers can manage their assets on the project, and do a first documentation pass regarding how to write and/or move a tool on the new project. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Hello, and the short term plans
On 02/25/2013 05:30 PM, Maarten Dammers wrote: Why start from scratch again? Why not just migrate the current Toolserver setup to the WMF cloud infrastructure? There are a number of obstacles to this, not least of which is the usage of Solaris and other non-Free tools in the infrastructure. But, more to the point, we have the opportunity to start with a clean slate and plan ahead; something which River and Daniel never had the chance to do. This allows us to reexamine the context today and provide for a more robust environment from its inception -- which is not feasible on an environment which is in use and is depended upon by dozens of maintainers. All of that said, much of the dev-visible architecture is designed to be as close to the toolserver's as possible to ease transition. I don't believe there are any gratuitous changes; the few differences to be found are meant to bring immediate tangible benefits to the tool maintainers. -- Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Hello, and the short term plans
Good news, everyone! As many of you know, I officially started my duties today as the WMF Operations Engineer attached to the Tool Labs. I intend to make a point of informing all of you of recent news, what I'm working on, and where I'm headed at regular intervals (probably weekly). First, a bit of news: I have had confirmation this weekend that the DB replication made available to Tool Labs users will, in fact, allow the creation of databases alongside the project ones. This means that one of the use cases that seemed the most troublesome in the transition (joins between the WMF databases and tool-specific ones) will be fully supported. We are making good strides in documenting the *impressive* inventory of tools that run on toolserver and their requirements (thanks, Silke!). The list-in-progress can be found at [1]. If you see missing or incorrect information, please feel free to adjust it -- the more precisely we know the requirements, the faster we can see about meeting them. I've started documenting my preleminary design for the shiny new Tool Labs infrastructure at [2]. This is a living document, and will see a great deal of revision before it's over (and will serve as the seed for the documentation). I will shortly create a new Labs project where that architecture is deployed in preproduction so we can shake out the kinks. The existing projects, "bots" and "webtools" will be left active for the forseeable future until (a) the new architecture has proven itself and (b) every user has sucessfuly moved their tools to it. I'm planning on having the new project be fully operational for new tools by the time the Amsterdam Hackathon takes place at the end of May at the very least. For the next week, I'll be mostly in information-gathering mode, as well as refining the design and requirements of the Tool Labs. Feel free to poke me for information (or /with/ information) by email or on IRC (where I am user 'Coren' and idle on #wikimedia-labs and #wikimedia-toolserve at the very least) -- Marc A. Pelletier [1] http://www.mediawiki.org/wiki/Toolserver/List_of_Tools [2] http://www.mediawiki.org/wiki/Wikimedia_Labs/Tool_Labs/Design ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Wikidata-moving
On 02/20/2013 08:03 PM, DaB. wrote:
it looks like the WMF-techs are moving wikidata to cluster s5 at the moment
(without announcement of course).
I wish I had already been in the loop, DaB, because I would have made
sure you had been notified as soon as it was known.
I think that's a communication failure on our* part that, as an external
user of that resource, you weren't put in the change notification loop.
I'll try to make certain that the toolserver admins are put on the
checklist of people to notify when performing a replication-impacting
change so that this does not occur again for planned interventions.
Again, sorry for the trouble this will have caused you.
-- Marc
* ('our' being a little presumptive here for a couple of days, but I've
no doubt that the oversight was accidental)
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Merry Christmas
On 24/12/2012 6:09 PM, Marco Fleckinger wrote: On 2012-12-24 21:35, Platonides wrote: +20 Merry Christmas to 255.255.255.255 :) Please stop flooding everybody with broadcasts! :) Frankly, if your router lets through global broadcasts, you should have a chat with your network admin that involves repeated application of a clue-by-four. :-) Oh, and I wish and/or acknowledge culturally appropriate sentiments relating to events you may celebrate during this period, if any. :-) -- Coren / Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] SSH client for iPad
On 22/06/2012 9:13 PM, Ja Ga wrote: Hey does anyone know of an iPad app that successfully SSH's to Toolserver? The freebies aren't working for me and I'd like to know it will work before I plunk down cash for one. iSSH is full of win, works perfectly for me, and I've never regretted the ~10$ it cost. It even has support for X11 and RDP over SSH. http://itunes.apple.com/us/app/issh-ssh-vnc-console/id287765826?mt=8 -- Coren / Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] operating systems and web servers
On 07/06/2011 4:49 AM, River Tarnell wrote: > > Why Linux and not FreeBSD, or Solaris, or OpenServer, or MP-RAS, or ...? > I've asked this many times in the past (usually when someone says "I > want Linux") and never had a real answer that I can remember.[0] Availability of expertise. Just here at Ubisoft, for instance, there are about 25 Linux sysadmins (ranging from intermediate to guru-level). Of all of us, there are only five that have Solaris or *BSD experience, and only *two* who have both. Whatever caused this originally, Linux ended up with a much bigger mindshare than the BSD did. You'll find more users and sysadmins with CentOS or Ubuntu experience than all other unices combined once we old farts start retiring. -- Coren / Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] RAM disk?
On 14/11/2010 4:21 PM, Derrick Coetzee wrote: > > In my experience, the use of temporary files is not that bad (for > example, gcc uses multiple temporary files in the course of compiling > a .c file). I once did a test build of the Linux kernel on a RAM disk > instead of the real disk and the difference in build time was less > than 1%. That's not a very good test: the kernel build uses the -pipe option of GCC and will not use temporary files, ramdisk or no. :-) The only difference you might see was the actual output files. -- Coren / Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] RAM disk?
On 14/11/2010 1:36 PM, Matthew P. Del Buono wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > >> You can use /tmp, which is a memory filesystem for exactly this reason. (But >> only on Solaris systems, not Linux, so you probably want to use willow.) >> > Wait, why isn't your /tmp be mounted with tmpfs? (Hm. Debian/Ubuntu only mount /var/run on tmpfs by default; that might be it). No reason for it not to be, really. -- Coren / Marc ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver paid accounts
River Tarnell wrote: > * dedicated virtual machine (Solaris zone), including root access: > EUR10/month. > Will it be okay if my cheque is cut for April 1st 2010? :-) -- Coren / Marc ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] SQL queries
Ilmari Karonen wrote: > Mashiah Davidson wrote: > >> I've implemented a bot, which for Ruwiki solves the problem of >> lonelypages and even isolated articles. The query you've just cited (an >> analogue to be honest) is the key point there as well and anyway there >> is a possibility to make it faster than one you cited on the TS. The >> problem here is that we deal with views, not the db itself, so this >> blocks further optimization somehow. >> > > I was going to disagree with you, but it does seem there's something > weird going on. Compare these queries: > > mysql> SELECT page_namespace, page_title FROM page WHERE page_title LIKE > '%fnord%' AND page_namespace=0; > Empty set (4.63 sec) > > mysql> SELECT page_namespace, page_title, page_id FROM page WHERE > page_title LIKE '%fnord%' AND page_namespace=0; > Empty set (4.90 sec) > > mysql> SELECT page_namespace, page_title, page_is_redirect FROM page > WHERE page_title LIKE '%fnord%' AND page_namespace=0; > Empty set (30.56 sec) > > mysql> SELECT * FROM page WHERE page_title LIKE '%fnord%' AND > page_namespace=0; > Empty set (41.23 sec) > > This isn't just random variation either, but seems completely > repeatable: including the page_is_redirect field (or, apparently, any > field other than page_namespace, page_title or page_id) in the query, > whether in the field list or in the WHERE clause, makes it run much more > slowly. WTF? > > That behavior is consistent with lack of indices, or the wrong *type* of indices, on the culpable columns forcing a seq scan. I should point out that it's not always a good idea to index less frequently selected columns because it increases the cost of insertions and updates significantly, so the solution isn't necessarily to add some either. Sometimes, if you are going to be doing a lot of selects on those columns, the creation of a temporary table with the needed index is, in fact, the most efficient solution despite the setup cost. -- Marc ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] wanted: toolserver admins
Mashiah Davidson wrote: > Do you have an idea, Mark, on how to improve the replication process > in terms of hw or sw, especially for s3 replication? I'd need to look into how it's currently being done in the first place; this isn't an aspect of the ts that I've fiddled with to date. Gimme a few days to look around and I'll see if I can think of something. :-) -- Marc ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] wanted: toolserver admins
Heya, I could give a hand. River Tarnell wrote: > [...] ideally, candidates > will: > > + have been a member of a Wikimedia project for a long time, with >some level of additional access (e.g. checkuser, OTRS, ...) I'm an admin on en.wp, a clerk of the en.wp Arbitration Commitee, a member of the BAG, and I am an OTRS agent. I also obviously have way too much free time on my hands. :-) > + have an advanced knowledge of Unix (at least several years >experience) > + have experience with Linux or Solaris system administration in a >production environment (another Unix would be okay if you're >willing to learn something new) > + have experience with MySQL database administration I've been a sysadmin for a bit over 20 years, most of past 10 with Linux. > + be available on the toolserver IRC channel to help users (this >isn't a requirement, but it would be helpful) I spend way too much time on IRC already, might as well be productive. :-) > > + experience with these products/technologies would be an advantage: >Apache; JIRA; PostgreSQL; Sun Web Server; Sun Directory Server / LDAP; >GlassFish; Kerberos; StorageTek SAM-QFS; Veritas. I grok Apache, Jira, Pg, and Krb5. I'm familiar with Veritas. The rest I mostly know /of/. > as this is a volunteer position, the amount of time you put into it > is entirely up to you; but at a minimum, you should be willing to > spend at least a couple of hours each week on the toolserver. I've already got a number of other tasks I do around the wmf, but I can certainly do a couple of hours of direct support work as well as idle in the ts channel to give a hand whenever I'm on IRC. -- [[User:Coren]] Marc A. Pelletier ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l ___ Toolserver-l mailing list Toolserver-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Re: [Toolserver-l] protip: i/o and infinite loops in php
River Tarnell wrote:
> never do this:
>
> [...]
>
> while (!feof($f)) {
> $x .= fgets($f);
> }
>
That may indeed have been a problem with a script of mine; I presumed
libc semantics where if the stream had gotten invalid feof() would
return an error (and, thus, not 0). At any rate, the script was
experimental and has since been disabled, but it's an important gotcha
to know of.
-- Marc
___
Toolserver-l mailing list
Toolserver-l@lists.wikimedia.org
http://lists.wikimedia.org/mailman/listinfo/toolserver-l
