Re: [Toolserver-l] Interwiki handling on the toolserver. (Was: FYI: Changing of interwiki-bots-account-approval)
Purodha Blissenbach wrote: >> with the beginning of the next year (2011) I will not longer accept new >> interwiki-bot-accounts, if: >> [...] > > While we're at it - in the future, we shall have interwiki bots reading the > replicated data bases to a great extent while gathering informations about > existing and prseumably missing interwiki links. This will be sparing lots of > request to the wmf servers which will then be bothered only when wiki pages > are actually altered. > > Using the replicated data instead of making http (api) requests should speed > up the data collection phase of large inteerwiki groups from several minutes > to a seconds or so. > > Another approach of making interwiki bots use the replicated data would > be to pre-process their interwiki data into a list or table of versioned > change requests, being published on the toolserver. > Interwiki worker bots running elsewhere would pick requests from the list & > process them. Picked requests are postponed for a while until replicated > data renders them done, or until a timeout (>replication lag) is exhausted. My current understanding is that a variety of different bot operators run interwiki.py from different accounts (both Toolserver accounts and Wikimedia accounts), using different lists, using very inefficient code, and bot operators do not check the edits manually. Is that correct? If so, there is an underlying, fundamental problem to interwiki.py that using database connections rather than HTTP API requests cannot fix. Do you know the status of getting a solution built in to MediaWiki (either in core or in an extension) that could make interwiki.py completely obsolete? It's my _strong_ recommendation that development effort be put into a real solution rather than focusing on ways to make interwiki.py suck less. MZMcBride ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] nightshade will be converted to Solaris on January 3, 2011
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jacobi Carter: > Will my statically compiled Go code run on Solaris like it did on > Linux? No, compiled software needs to be recompiled for Solaris. There is currently no Go compiler for Solaris, but if you need one, we can look at porting either of the existing compilers - river. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAkz4besACgkQIXd7fCuc5vIxMwCeKZjp3z5uSfAUNBk1IyJs+MSA 7G8AoKUQ5ZNRxBQYgGAOyMZw0iPzJg94 =FTKC -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] nightshade will be converted to Solaris on January 3, 2011
On Thu, Dec 2, 2010 at 6:53 PM, River Tarnell wrote: > No doubt there *are* some differences that will require changes to tools, and > OS differences that I didn't list above. This is why we gave users two months > to test their tools before announcing a date for the switch, and a further two > months after the announcement before the actual conversion. > > - river. Will my statically compiled Go code run on Solaris like it did on Linux? Not that I need to run it again, but before I write any more Go destined for the toolserver, it'd be useful to know if it will be supported. Thanks, Cobi. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] nightshade will be converted to Solaris on January 3, 2011
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Purodha Blissenbach: > I have reported *) a few incompatibilities in very basic tools and bash > that I have not the faintest idea of how to get around them under Solaris. Please files issues in JIRA and include "[solaris]" in the subject line. I don't remember the mail you quoted below, and if it's not in the archives, perhaps it never made it to the list at all. > Yes, these may be bugs in the Solaris tools or O/S, and some of my scripts > will not run without them fixed. My only option atm is moving them elsewhere > outside the toolserver cluster, when there will be no Linux host left. Your other option is to ask us for help with migrating your scripts. > > Why would we give up the flexibility of having a choice of operating > > systems to log in to? Well, what is "Linux"? I doubt there are many tools that require a particular OS kernel (such as Linux) in order to run. So, when people say they prefer Linux, what are they really looking for? I can think of several things: * Shell -- but this is the same between Linux and Solaris (bash) * Shell utilities (ls, find, awk, sed, grep, cut, ...) -- but these are the same between Linux and Solaris (GNU utilities) * "ps" -- but we provide our own version of "ps" that supports the BSD syntax that many Linux users are used to. * "top" -- top on Solaris is slightly different because it's "Unix top" instead of "procps top", but this isn't something that tools typically use, and it's not that hard for users to learn the difference. (prstat is better anyway ;-) * Default configuration for utilities. For example, some users complained about the "vim" on the Solaris systems because they expect it to be configured the same way that Debian Linux configures it. But "vim" is vim on every OS, and the default configuration can be changed. If uses report these issues to us, I have no problem with changing the Solaris configuration to match the Linux configuration where it makes sense. (For example, I think Debian's default vim configuration is quite sensible.) * GNU compiler -- in the past, users switching C or C++ tools from Linux to Solaris often had to change from GNU GCC to Sun Studio. But from Monday, GCC will be the standard compiler on both systems. * cron -- okay, Solaris doesn't support "/" syntax in cron. I personally think this is only a minor annoyance and hardly justifies providing a completely separate OS for people who just cannot work without "/". To address your specific concerns: Shell scripts mostly depend on shell utilities; all the ones you mentioned (Bash, awk, sed, grep, and cut) are GNU tools, and are identical between the Linux and Solaris login servers, because we provide the GNU userland by default. (The exception is that some tools on the Solaris systems may be newer versions than on Linux.) As for pywikipedia, I don't think it's correct to claim that this "runs with Linux only" -- in fact, there are several pywikipedia instances currently running on the Solaris login server. PHP is a third-party application and is completely unrelated to OS, except that the version on the Solaris servers is newer (5.3 vs 5.2). No doubt there *are* some differences that will require changes to tools, and OS differences that I didn't list above. This is why we gave users two months to test their tools before announcing a date for the switch, and a further two months after the announcement before the actual conversion. - river. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAkz4P5wACgkQIXd7fCuc5vKYoQCfWZasaeAg7Dd6ljV8MHjIjcUZ u6AAoMCaRE8/CwR7nyD2yraq7YkqZo/g =PZ2N -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] nightshade will be converted to Solaris on January 3, 2011
River Tarnell wrote: > Since no major issues have been raised regarding the conversion of > > nightshade to Solaris, this has been schedule for the maintenance > window on January 3rd, 2011. For more information, see I have reported *) a few incompatibilities in very basic tools and bash that I have not the faintest idea of how to get around them under Solaris. Yes, these may be bugs in the Solaris tools or O/S, and some of my scripts will not run without them fixed. My only option atm is moving them elsewhere outside the toolserver cluster, when there will be no Linux host left. I shall re-evaluate the scripts under Solaris during the next days, just in case, something has been altered meanwhile, and report remaining issues more detailed. *) Wanting to link to my earlier e-mail in the list archive, I could not find it :-( Here is a copy: On Thursday, 16. September 2010 04:02:35 Purodha wrote: > > ... we have > > decided to standardise on Solaris for the login and web servers. We will > > therefore be converting nightshade from Linux to Solaris at some point in > > the future. > > > > There is no fixed time frame for this at moment, but it won't happen > > sooner than a month from now, and we won't do anything until we are > > satisfied that all tools (and users) are ready for the migration. > > > >To start with, we want to identify and fix any issues which prevent users > > from moving their tools to Solaris. This will mainly include: > > > > * Software which needs to be installed or updated > > * Behaviour differences between Linux and Solaris where the Linux > > behaviour is more correct or preferable. > > I have several shell scripts (using bash, awk, sed, grep, cut, php, > pywikipedia) that run with Linux only. > > I have tried to make them work on Solaris, too, while we had an idle > machine, but it was impossible. I cannot quite remember what the > precise causes were, but there were several incompatibiliities > prohibiting one staight script for both systems without branching inside > the script as per system, and at least one issue that prohibited a solaris > version entirely - something in the realm of awk features, regexp's or > shell escapes. > > Unfortunately, it will take time before I shall have the problems spotted > again. I might need help, then. > Since I had been unable to fix them in the first ~135 tries I expect to > fail on the ~136-th one as well. > > > Unrelated,more general question: > > Why would we give up the flexibility of having a choice of operating > systems to log in to? > > Geetings - Purodha Greetings - Purodha___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Interwiki handling on the toolserver. (Was: FYI: Changing of interwiki-bots-account-approval)
Hello all, > with the beginning of the next year (2011) I will not longer accept new > interwiki-bot-accounts, if: > [...] While we're at it - in the future, we shall have interwiki bots reading the replicated data bases to a great extent while gathering informations about existing and prseumably missing interwiki links. This will be sparing lots of request to the wmf servers which will then be bothered only when wiki pages are actually altered. Using the replicated data instead of making http (api) requests should speed up the data collection phase of large inteerwiki groups from several minutes to a seconds or so. Another approach of making interwiki bots use the replicated data would be to pre-process their interwiki data into a list or table of versioned change requests, being published on the toolserver. Interwiki worker bots running elsewhere would pick requests from the list & process them. Picked requests are postponed for a while until replicated data renders them done, or until a timeout (>replication lag) is exhausted. Greetings - Purodha ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [Toolserver-announce] Hardware failure affecting s2/s5
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 River Tarnell: > However, we will need to shut down the system either today or later this week > to replace the faulty part, which will cause under 30 minutes outage for > s2/s5. This maintenance has been scheduled for 1030h UTC[0] tomorrow (3rd Dec). - river. [0] http://www.timeanddate.com/worldclock/fixedtime.html?day=3&month=12&year=2010&hour=10&min=30&sec=0&p1=0 -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAkz3qosACgkQIXd7fCuc5vJG0gCggJFnPowBnyujHBnzdrGtnnLD mbEAn032FO/GcMVq6A+FSM/KKojs0d1b =KQin -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] [Toolserver-announce] Hardware failure affecting s2/s5
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 River Tarnell: > The s2/s5 server is currently experiencing some hardware issues. As we > currently only have a single server for s2/s5, this may result in outages for > these clusters. Hi, We have identified the cause of the fault and do not expect any further failures. However, we will need to shut down the system either today or later this week to replace the faulty part, which will cause under 30 minutes outage for s2/s5. - river. -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAkz3l+QACgkQIXd7fCuc5vIwZwCfR2MLssyLBpG95IFQjjwavWB0 tOMAmwakPUb2idBQfrJpUaf9JGRJkIfh =jgeP -END PGP SIGNATURE- ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
