Re: [Toolserver-l] Data wipe tomorrow

[Platonides]

On 11/12/14 10:59, Marlen Caemmerer wrote:

Hello,

tomorrow is the day I will start to completely throw away all data on
the Toolserver.
After this nothing can be copied anymore.
If you feel you still need anything please let me know asap.

Cheers
nosy


Are http://svn.toolserver.org/ repositories kept somewhere?

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] [Labs-l] DEPRECATED: tools.wikimedia.de -> Get rid of it!

[Platonides]

Merlijn van Deen wrote:

On 11 May 2014 13:55, Silke Meyer mailto:silke.me...@wikimedia.de>> wrote:

It is not a trivial redirect: Wikimedia Deutschland will obviously not
give the wildcard SSL certificate for *.wikimedia.de
 to WMF (and WMF
would not want to have it). This would mean we would have to
completely delegate that subdomain to WMF and guarantee that it stays
like that forever. This is hard to guarantee and it is also misleading
to delegate a .de subdomain to the Foundation.


First of all: Why would the (sub)domain need to be delegated to the WMF?
The redirect could just be on WMDE servers.

If the redirect *has* to be on Foundation servers for some reason, it
could just use a specific tools.wikimedia.de 
certificat -- or we could just kill SSL altogether -- the
tools.wikimedia.de  domain is from before the
toolserver even had SSL support.


+1

I think you are viewing things more complex than they really are, Silke.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] hawthorn problems: broken sge mount and crond

[Platonides]

hawthorn is acting up a bit.
The /sge mount in hawthorn:
 /sge on ha-sge.esi:/global/misc/sge 
remote/read/write/setuid/devices/rstchown/vers=3/proto=udp/xattr/dev=4bc0006 
on Mon Mar 17 20:39:45 2014

seems broken:

??   ? ???? sge



There were also errors of:

-bash: fork: Not enough space

but that may be related to the 247 instances of UntagF5s.py by tim1357
(script confused due to the broken cronsub?) The 124 instances of
/opt/ts/sbin/crond look fishy, too.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] sql command

[Platonides]

On 18/12/13 18:17, Merlijn van Deen wrote:

On 18 December 2013 17:52, John  wrote:

I am in the process of migrating to labs and the command sql
performs differently between the two platforms, Can I get the source
for the toolserver version of it so that we can make the labs
version work the same.


Check /opt/local/bin/sql (on solaris) or /usr/local/bin/sql (on linux).


The file is a publicly readable shell script, but for the benefit of the 
archives, I'm attaching it.
#! /bin/ksh

PATH=$(getconf PATH):$PATH

usage() {
echo >&2 "usage: $0 [-r | -u]  [command]"
echo >&2 "-rConnect to round-robin database"
echo >&2 "-uConnect to user database"
exit 1
}

uflag=0
rflag=0

while getopts ru opt; do
case $opt in
u)  uflag=1;;
r)  rflag=1;;
*)  usage;;
esac
done

shift $(( OPTIND - 1 ))

if [[ $rflag = 1 && $uflag = 1 ]]; then
echo >&2 "-u and -r cannot both be specified"
usage
fi

if [[ $# = 0 ]]; then
usage
fi

if [[ $# > 1 ]]; then
query=$2
fi

if [[ $1 == "toolserver" ]]; then
server=sql-toolserver
elif echo "$1" | grep -q '^[up]_'; then
server=sql
elif [[ $1 == "centralauth_p" ]]; then
server=z-dat-s7-a
else
db=$(mysql -BN -hsql-toolserver -e "select server from wiki where 
dbname='$1';" toolserver)
if [[ -z "$db" ]]; then
echo >&2 "$0: no such database \"$1\""
exit 1
fi

if (( $uflag )); then
server=sql-s${db}-user
elif (( $rflag )); then
server=sql-s${db}-rr
else
server=sql-s${db}
fi

fi

if [[ -z "$query" ]]; then
mysql -h $server --comments $1
else
mysql -h $server --comments $1 -e "$query"
fi
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Redacted database content

[Platonides]

In which way were they non-conformant? Toolserver data was hiding (most)
non-public fields, being very similar to what labs hid (actually, I
remember fields hidden in toolserver available in labs, not the other way
around). What did you remove?
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] FYI: Toolserver account expiry in early January

[Platonides]

"the second tool deletes all this data as well as expires your account."

The tools migrated to labs should be 301-redirecting to the tools new
home, not showing an "Account expired" message.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cron on nightshade not running since Nov 2nd?

[Platonides]

On 07/11/13 01:30, Christian Thiele wrote:

Hello,

Am 06.11.2013, 22:49 Uhr, schrieb Platonides :


On nightshade, crontab *is* cronie.

The cron server is running and -after a quick test- working. The
problem seems to lie with emails, which are not sent.


I spoke about willow, and there crontab didn't work (I don't receive any
mails from cron (everything goes to /dev/null), but I write log files).
I don't know what's the case with nightshade. I didn't thought about it
because I changed my cron and on another MMP project crontab never
worked on willow (only cronie), so I thought it's some problem with
crontab. So I thought it was only my problem and switched to cronie for
my user account, too (and it worked again). But then a mail was written
by Danny. So it seems to be a general problem with crontab on willow.

Chris / APPER


Well, Danny was talking about nightshade. It is indeed the case that in 
willow crontab and cronie are different processes, and sometimes crontab 
dies (seems to be running, though).



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cron on nightshade not running since Nov 2nd?

[Platonides]

On 06/11/13 09:59, Federico Leva (Nemo) wrote:

I don't know if cronie is recommended, but
https://wiki.toolserver.org/view/Cron#Conversion

Nemo


On nightshade, crontab *is* cronie.

The cron server is running and -after a quick test- working. The problem 
seems to lie with emails, which are not sent.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] List of the default license of each Toolserver user

[Platonides]

On 15/10/13 01:18, Marc A. Pelletier wrote:

A question for legal remains; is the statement to which you've just
enumerated the answers to sufficient to apply licensing to things that
aren't otherwise clearly indicated?  Does anyone have the text of the
question handy?

-- Marc


$ setlicense

This program allows you to change your default Toolserver license.  Your
default license is the license your Toolserver-hosted tools are assumed to
be under if no other license is specified.

For more information about default licenses, please read
.


https://wiki.toolserver.org/view/Default_license:


Every account has a default license. This is the license that your tools are 
licensed under when there is no other license present in the source. The main 
purpose of the default license is to ensure tools are not abandoned when their 
owner leaves.

When an account is created, no default license is set. You can view and change 
the default license for your account by running setlicense on a Solaris host 
(like willow).

The default license only applies when there is no other license notice in the source 
code, so you may license some tools under a different license by adding a license header. 
Alternatively, you may add an "All rights reserved" header, to indicate there 
is no license for that particular tool.

There is no requirement that users select a default license, or to choose an open source 
license. However, the Toolserver administrators would urge users to do both where 
possible. If you do not wish to select a default license, please run setlicense and enter 
"None" as your license choice.





___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Need Tool Labs migration support?

[Platonides]

On 08/10/13 00:34, Marc A. Pelletier wrote:

On 10/07/2013 06:27 PM, Magnus Manske wrote:

* Forcibly port the most important, unmaintained tools to Labs


Sadly, that's only possible when the code is unambiguously licensed as
Open Source, which is often not the case (much of the code running on
the Toolserver has no licensing information at all).

-- Marc


Then the default license applies.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] TS and/vs. Labs

[Platonides]

On 17/09/13 01:22, Marc A. Pelletier wrote:

On 09/16/2013 04:13 PM, Platonides wrote:

labs db server do contain the non-public data.


Just to clarify things, here, that's a necessary artifact of the way
mediawiki works: many of the bits of data that should be made
unavailable are done so conditionally on /live data/. It is not possible
to replicate the visible contents of the views and have it not break as
column values may be nulled or *come back* depending on actions on the
projects (supression, deletion, etc).

While some of the more sensitive data never hits the replica (like IP
addresses), It is not /possible/ to create a replica without
transferring data that can be unsupressed or undeleted -- but might
never be.

-- Marc



I guess there could be a labs replica with triggers that deleted data at 
the point it became private, which could then serve as replication 
master providing public binlogs. The problem which breaks the idea 
completely are the restores, ie. non-public data coming back (the server 
would receive ‘show this again’, but it would need a full insert...).


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] TS and/vs. Labs

[Platonides]

On 17/09/13 00:11, Tim Landscheidt wrote:

Ryan Lane  wrote:


[...]



I'm more than happy to recommend a number of cloud services and am
more than willing to give advice on how to configure and run tools
and bots from those services. It's even possible to reuse the work
we're doing in the tools project, or in the Wikimedia infrastructure
via our puppet repository since our infrastructure is Open Source.



Very nice idea – how I get the mysql-replication-stream? I got several
offers of donation if the Toolserver would continue; the only problem is
the replication-data. But because the data is open-source, it shouldn’t
be a problem than, should it?



Assuming you found a non-profit, host your infrastructure somewhere that
doesn't cause legal issues and every person that has access to the data
stream signs an NDA it's likely doable. [...]


The NDA isn't necessary.  According to
https://wikitech.wikimedia.org/wiki/Tool_Labs/Database_plan,
the data set at the LabsDB stage is free of non-public data
(modulo MariaDB accounts information which should probably
not go off-site even with an NDA :-)).

So we could (and IMHO should) provide DB dumps/bin logs at
dumps.wikimedia.org or somewhere similar to anyone who can
download them.

Tim


labs db server do contain the non-public data. It's just not viewable. 
So there aren't bin logs for just the non-public data.
You *could* make a dump of the database (probably creating a new tool, 
as mysqldump would simply dump the view definition)... assuming you that 
in doing that you don't kill labs filesystem. ;)


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] [Maps-l] Maintenance: PHP update tomorrow

[Platonides]

On 05/09/13 22:56, Marlen Caemmerer wrote:

Hey,

I updated PHP and tested some tools. It looks good so far but I changed
the version from 5.3 to 5.4 so if anything does not work please consider
this too but still write an email or file a ticket.

Cheers
nosy


I suspect a number of the module errors in /var/log/userlog are related 
to this (mostly modules needing a recompile).


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] TS and/vs. Labs

[Platonides]

On 11/08/13 08:36, Silke Meyer wrote:

* Most tools will just run when you copy them to Tool Labs. Just in
case: Need help adapting a tool? Ask me and I'll do what I can to get
you help, e.g. from my colleague Johannes Kroll.


Provide a toolserver.wiki table please.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Pywikibot, GIT and TS

[Platonides]

On 10/08/13 13:05, Dr. Trigon wrote:

If you read the instructions
>>  (https://www.mediawiki.org/wiki/Manual:Pywikipediabot/Gerrit),
>>  it recommends you only clone with a --depth of 3 (or anything
>>  smaller), and it will take up very little space.

Yes of course there is a solution - but it again one thing that can go
wrong and another paramter you have to remeber and use. By the way it
would be a good thing if it would be possible to develop from TS also.
And then all the 200MB are needed, aren't they?


No, they aren't. You can happily develop from a shadow clone. You would 
only have problems if someone sent you a patchset based on an old 
version you didn't clone (but when working from gerrit, it won't happen).


You would of course not have the commits previous to the point you set, 
so you won't be able to properly view the commit list or blame. But 
that's not a big difference with svn, where those requiring downloading 
the full file history from the net (I went to ViewVC for those actions). 
In git they are nice with a full clone, but they won't work properly 
with a shallow clone (eg. they will attribute everything “old” to the 
earliest commit you cloned).


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Large core dumps found in home directory

[Platonides]

On 19/06/13 00:12, Derrick Coetzee wrote:

Hi all, I noticed today I had several large core dumps in my home
directory, and two buried under my public_html directory. The
directories I found them in contained no executables and as far as I
know were not working directories for executables (just PHP scripts used
by the web server), so I can't imagine how they got there.

-rw--- 1 dcoetzee users 191463424 May  5 06:56 core

-rw--- 1 dcoetzee users 38797312 Sep 29  2012 core

This exceeded my hard quota without my knowledge. Any idea what's up
with this? Can I prevent it from reoccurring? Thanks.

--
Derrick Coetzee
User:Dcoetzee


The php fastcgi script crashed and left a useless core.

This has happened for a long time, see 
https://jira.toolserver.org/browse/TS-1505 but it had (apparently) been 
fixed, so perhaps it got reintroduced when the web servers were 
reconfigured?


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Login issues

[Platonides]

On 23/07/13 22:18, Emilio J. Rodríguez-Posada wrote:

Hi;

I'm having login issues since some days ago. I'm trying ssh
emi...@login.toolserver.org  and get
password failures. I'm almost sure I have not changed my password...

Can you verify that my account is OK?

Thanks,
emijrp


Users can't do password authentcation. Do you have your ssh key loaded?
Your account is not expired.
I see a strange statement when fingering you:

Last login Tue Jul 23 21:55 on null


So perhaps it wasn't able to allocate a pty?

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] commonswiki_p corrupted

[Platonides]

Commons copy on s3, s4 and s6 are corrupted. The versions at s1 and s5 
are not affected.


Test case:
mysql -h $server commonswiki_p -e "SELECT page_title FROM page where 
page_id=21683917"


Good:
+--+
| page_title |
+--+
| Dworzec_główny_w_Gdyni.jpg |
+--+

Bad:
+--+
| page_title |
+--+
| Dworzec_główny.jpg |
+--+

It looks like a range of statements from November 2012 weren't replicated.

Other tests are:

SELECT log_id, log_title, log_type, log_action FROM 
logging_ts_alternative where log_namespace=6 AND 
log_title='Dworzec_główny.jpg'"
The wrong copies don't have «| 48917341 | Dworzec_główny.jpg | move | 
move |» (the other 7 entries appear in both)


SELECT rev_id, rev_timestamp, rev_comment FROM revision where 
rev_page=21683917
The wrong copies don't have the last 4 revisions (all of them from 
2012-11-03)


'Dworzec_główny_w_Gdyni_4.jpg', 'Dworzec_główny_w_Gdyni_3.jpg', 
'Dworzec_główny_w_Gdyni_2.jpg' are similarly affected, shown as just 
Dworzec_główny_0X.jpg in the wrong dbs.


Comparing the number of revisions per day, it seems to have fixed on 
2013-11-04, with the 3 being the most noticeable (27746 revisions where 
they should have been 53974), a difference of 80 revisions the previous 
day, 200 on 2012-10-27...


It should be possible to just copy the db from s1.

s1 and s5 are probably sane due to the reimports on January and April.

As an unexpected thing, sql-s2 and sql-s7 don't have a commonswiki_p 
copy. sql-s3 and sql-s6 don't have a commonswiki_p.revision table/view.


I created https://jira.toolserver.org/browse/TS-1667


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] toolserver web down

[Platonides]

toolserver.org isn't listening on port 80:

willow ~ $ telnet toolserver.org 80
Trying 2a02:ec80:101::1:4...
telnet: connect to address 2a02:ec80:101::1:4: Connection refused
Trying 185.15.59.214...
telnet: Unable to connect to remote host: Connection refused


Connecting directly to the web servers works.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Toolserver db outperformed by labs

[Platonides]

Today^W Yesterday, I was asked about some file numbers, which involved
subcategory traversing, which is an "inefficient" problem. It seemed a
good problem for comparing toolserver and labs. And toolserver db sucks:

willow: 31m5.157s (user 0m4.038s)
labs: 0m4.271s (user 2.488)

Toolserver was *436 times slower*.

Surely, the labs server is better (in hardware) than the one in TS. I
don't know how many scripts were hitting the TS db, while the labs one
would be almost-idle. Still, it seems a really big gap. Do we have
something wrongly configured? Did mariadb somehow massively improve vs
mysql? Are some parameters too small? Is it just a problem that the
mysql servers are underprovisioned of ram?



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Encoding issue using SGE

[Platonides]

On 23/05/13 00:38, Hercule Hercule wrote:
> Hello,
> 
> I deleted one of my script, and try to recreate it. But I now have an
> encoding issue when I run it with SGE. I searched a solution during all
> this evening and can't find any idea to solve it. I hope one of you will
> be able to help me.
> 
> I created the file /home/hercule/batch/LienFr.sh with this content :
> 
> #-*- coding: utf-8 -*-
> #!bin/bash
> #! /usr/bin/python

These shebangs are crazy. You would want just "#!/bin/bash" as the first
line.


> #$ -l h_rt=1:30:00
> #$ -l virtual_free=500M
> #$ -l arch='*'
> #$ -m as
> #$ -o /home/hercule/logs/LienFr.log
> #$ -j y
> #$ -wd /home/hercule/
> python /home/hercule/template.py "Lien" "Lien/Conversion automatique"
> -cat:"Page utilisant Lien pour un article existant" -summary:"[Bot] :
> transformation de liens avec le modèle {{Lien}} en lien interne, suite à
> la création de l'article correspondant" -assubst -pt:0 -always
> mutt -s "Fin de LienFr" hercule < /home/hercule/logs/LienFr.log
> 
> 
> When I simply run /home/hercule/batch/LienFr.sh in the terminal the
> result is correct :
> http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370887&oldid=93370870
> 
> But when I use the command line 
> qcronsub -N LienFr /home/hercule/batch/LienFr.sh
> 
> I have a problem with the summary :
> 
> http://fr.wikipedia.org/w/index.php?title=Uttar_Pradesh&diff=93370948&oldid=93370932
> 
> Do you know how to solve this ?
> 
> Regards
> 
> Hercule

Seems the string was converted to utf-8 twice. What did you use to edit
the file?

Try converting it with iconv:
iconv -f utf-8 -t iso-8859-1 < LienFr.sh > LienFr2.sh



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Maintenance: Rebooting ortelius web server

[Platonides]

On 09/05/13 18:57, Tim Landscheidt wrote:
> Marlen Caemmerer  wrote:
> 
>> I would like to reboot ortelius, one of the web servers at
> 
>> tomorrow, Tuesday 1830 UTC
> 
> Apparently, wolfsbane rebooted today as well:
> 
> | timl@wolfsbane:~$ uptime
> |  16:49pm  up   5:00,  2 users,  load average: 1.16, 1.24, 1.47
> | timl@wolfsbane:~$
> 
> Perhaps related to that, SGE queues on ortelius and wolfs-
> bane are in state "au" (alarm, unknown):

Yes, sge_execd seems not to be running on them.

Plus medium and longrun queues in yarrow are in error state. I tried
cleaning them, but they failed again.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Toolserver limitation to come?

[Platonides]

On 02/05/13 10:03, Marlen Caemmerer wrote:
> Hey,
> 
> On Wed, 1 May 2013, Ryan Lane wrote:
>> On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt
>> wrote:
>>>
>>> There were never answers to this, so I bring it up here
>>> again:
>>>
>>> 1. How were "almost /all/ of the problems the TS have had
>>>with replication" "caused by that redundancy and trying
>>>to keep it synced"?
> 
> Thanks for this question :) - I also want to know.
> From my perspective it does not look like this and even the data
> inconsistencies appear when we have no commons copy on a mysql instance.
> And: DaB experimented with federated tables for commons too and we
> decided to not do this since it does not perform from the start.
> Probably nowadays when I planned something new in this area (which does
> not seem to make sense for TS) I'd really give Galera a try -
> http://codership.com/content/using-galera-cluster

FWIW, my {{ref needed}} phrase in the log was also intended to that
statement by Coren, not to multichill reply.

Additionally, I rembember you data inconsistencies happen even with
native mysql replication (as informed by Nosy earlier).

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Survey: Moving to Labs

[Platonides]

On 01/05/13 21:53, DaB. wrote:
> Hello all,
> 
> until now I had the impression that we (you, the authors, and me) fight 
> together against WMF and WMDE for keeping the Toolserver and against Labs. 
> Some mails and discussion in the last days gives me now the impression that 
> this was wrong and (at least some of) you are eager to leave the toolserver 
> as 
> soon as possible.

It does look at times as if they wanted to remove the toolserver from
behind us, but it shouldn't be considered a fight.
In this situation, the interest for that new-über-replacement it's
completely normal. It doesn't mean that they love one more than the other.


> There is no point to beg the WMDE for new hardware and to invest much more 
> time if 2 weeks after Labs is "ready" the toolserver will be empty. For this 
> reason I created a survey at [1] that starts at midnight. Please take a 
> moment 
> of your time and place your nick in the section that suits you.
> 
> Sincerely,
> DaB.

Even if "labs being ready" happens in 2018?
"ready" will vary for each tool, but I foresee a process like this:

1. labs provides all the resources needed for $TOOL
2. $AUTHOR signs up in labs, gets added to the projects, etc.
3. $AUTHOR tests (benchmarks) labs and finds it acceptable for $TOOL
4. $AUTHOR learns all the labs-specific details.
5. $AUTHOR allocates some time for installing $TOOL in labs
6. Fix bugs in $TOOL when run in labs (aka. adapt $TOOL to labs)
7. (Optional) Redirect toolserver/$TOOL to labs/$TOOL

For the majority of tools, we haven't reached #1 yet.

Once labs provides (almost) everything available at toolserver, you can
start talking about when to stop supporting TS. But doing otherwise is
premature.
#2 is the only step that could take place before #1.

Then there is the 'lazy factor' for #2-7, although it is also known as
"too busy to fix this program which works ok in TS"
Some people may skip #3, while others will want to be damn sure that it
will work correctly.
The time required by #4 can be reduced making labs very similar to
toolserver.
If labs environment for the programs is very different, such as needing
to do the joins manually inside the program, that will increase #6 a lot.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Doxygen and coverage

[Platonides]

On 28/04/13 13:44, Dr. Trigon wrote:
> Hello Admins!
> 
> Could please someone take care of TS-1301 [1]? This ticket is now
> open for quite a while and having the possibility to generate docs
> and coverage reports on the TS would be very favourable.
> 
> [1] https://jira.toolserver.org/browse/TS-1301
> 
> Thanks a lot and Greetings! DrTrigon

I tried to add a puppet entry for that in the devel manifest, but my
commit was rejected (access to the file forbidden).

I left a question there regarding the coverage pkg.


PS: It seems that there was a temporary LDAP failure yesterday night,
I had a number of jobs in error status with reason «can't get password
entry for user "platonides". Either the user does not exist or NIS
error!»  and there were ‘temporarily not available’ queues.
I have reenabled the affected queues
(short-...@wolfsbane.toolserver.org, longrun...@yarrow.toolserver.org,
medium-...@wolfsbane.toolserver.org, medium...@mayapple.toolserver.org)

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cronie jobs with specified ENV are not executed

[Platonides]

On 24/04/13 20:16, Adam Klimont wrote:
> My cronie task list looks like this:
> 
> PATH=/opt/ts/gnu/bin:/sge/GE/bin/sol-amd64:/usr/bin:/bin
> 5 1 * * * qcronsub -N 'wymowa.py' -b y -l h_rt=4:00:00 -l
> virtual_free=50M /usr/bin/env PYTHONPATH=$HOME/wikt/pywikipedia
> /usr/bin/python2 $HOME/wikt/moje/wymowa.py >/dev/null
> 6 2 * * * qcronsub -N 'frequency.py' -b y -l h_rt=0:30:00 -l
> virtual_free=20M /usr/bin/env PYTHONPATH=$HOME/wikt/pywikipedia
> /usr/bin/python2 $HOME/wikt/moje/frequency.py >/dev/null
> ...and so on
> 
> It stopped working around 2-3 weeks ago. Now I get an error message:
> /usr/bin/env: No such file or directory
> 
> What should I change in these lines in order for the scripts to work again?
> 
> alkamid

There's no python2 binary on debian nor solaris. Just replace 'python2'
with 'python'. In both OS you will get a python 2 release (2.6.6 and
2.7.1 respectively).

Although I agree it would be nice to have python2 as a symlink.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] I can't login any more

[Platonides]

On 24/04/13 16:25, Alex Brollo wrote:
> Thanks - really I felt that I should get the renew notice but I
> obviously missed/deleted the mail; I've been confused from some
> toolserver stop-and-go notices, and from some messages by other user,
> with login troubles. 
> 
> Yes, you are true: I found the notice "Your account will expire
> tomorrow" into the group of deleted mails I apologyze for  your
> wasted time. 
> 
> Me stupid. 
> 
> Alex

Being busy is different than being stupid. Plus this was easy to
determine, don't worry.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] I can't login any more

[Platonides]

On 24/04/13 10:54, K. Peachey wrote:
> IIRC that should still log you in, but you get a "You account has been
> disabled message" and your web folder should show a "this account has
> expired" message.

I'm sorry, but it does seem disabled: http://toolserver.org/~alebot/

Your account was set to expire on on 2013-04-05 (15800). Open a jira bug
(or otherwise bug a toolserver admin) to request it is enabled again.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] I can't login any more

[Platonides]

On 24/04/13 00:15, Alex Brollo wrote:
> Thanks, but server closes connection as soon as I try to login - both
> with PuTTY and by WinSCP.
> 
> No matter; I don't want to waster your time any more; I'll search for
> help y some friends into it.wikisource.
> 
> Alex

What's your username? Did your perhaps allow your account to expire?

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Qcronsub and qstat trickery

[Platonides]

I have been trying several variations of:
 qcronsub -l h_rt=00:00:20 -l virtual_free=10M -l arch=lx ./helloworld.py

and it is always working.
Is the original qcronsub method still failing for you?


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Qcronsub and qstat trickery

[Platonides]

On 21/04/13 19:22, Theopolisme wrote:
> Okay, I found this useful old message in the toolserver-l archives:
> [http://osdir.com/ml/toolserver-l/2011-01/msg00015.html]. I configured
> everything like the message suggested
> [https://github.com/theopolisme/theobot/commit/302e7383be5aac9de9afc5fe17160232bff5e132],
> then ran and got this error:
> 
> /home
> /theo/pysub[3]: /sge62/default/common/settings.sh:  not found/
>
> Looks like an issue with the bash code itself. Thoughts?

/sge62 is an old folder. It is empty now (why wasn't it removed completely?)

Replace
. /sge62/default/common/settings.sh
with
. /sge/settings.sh


Your previous way should have worked, too.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Qcronsub and qstat trickery

[Platonides]

On 21/04/13 17:21, Theopolisme wrote:
> The full content of the script is
> [https://github.com/theopolisme/theobot/blob/master/bsr_notifier.py].
> Nowhere in the script is any sort of image processing. I'm suitably
> mystified.

The shebang got ignored, somehow, (did bsr_notifier.py have execute
permissions?), and sge tried to be execute it as a shell script.

Thus the import lines called import(1) from imagemagick (this program
tries to open the X server to take a screenshot)

On linux there's also a from command which prints out the mail header
lines from the invoker's mailbox, which produced the /var/mail errors.

See the beginning of the file annotated with the errors given:
> #! /usr/bin/env python <- Comment
> import mwclient <- import: unable to open X server `' @
error/import.c/ImportImageCommand/362.
> import re <- import: unable to open X server `' @ 
> error/import.c/ImportImageCommand/362.
> import sys <- import: unable to open X server `' @ 
> error/import.c/ImportImageCommand/362.
> import datetime <- import: unable to open X server `' @ 
> error/import.c/ImportImageCommand/362.
> import pickle <- import: unable to open X server `' @ 
> error/import.c/ImportImageCommand/362.
> from theobot import bot <- from: can't read /var/mail/theobot
> from theobot import password <- from: can't read /var/mail/theobot
> 
> # CC-BY-SA Theopolisme <- Comment
> # Task 9 on [[User:Theo's Little Bot]] <- Comment
> 
> def sokay(donenow): <- Syntax error: "(" unexpected

I see you recently changed the shebang in order to add a space
before the path:
https://github.com/theopolisme/theobot/commit/fe58c45b6e8997177d27f49ef9f25dba42171ee5
Both shebangs are ok.

I will send you privately an email regarding a problem I noticed in your
tool.

Regards

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Wikidata tables

[Platonides]

Sorry, but your mail doesn't make much sense.

El 19/04/13 11:14, Patricia Pintilie escribió:
> Ok so how about we recocnize what the overal goal is first . Then
> establish the point that its trying to convay. Only then can we meet in
> the middle and set a plan in motion. I can only assist when a plan of
> action is clear with a definite plan without it im lost on where to
> begin. It seems as if im doing my natural instinct research then I get
> mail from the people im reading about... Very interesting this is
> because im left to think my mind is linked to the problems at hand. TS
> is my old signature, my server os will pull up my IP searches. Which
> leads me to believe this is why I am always being brought up in the
> middle of these outstanding conversations you guys are having lol.
> Please send detailed instructions as to how I can help,there should be a
> file known as Mila.eu also known as ro.eula. Find it and run whatever it
> has, thanks.
> -patiently waiting your responce.
> -MilaStarX-TS

TS was being used in this thread as an abbreviature of ToolServer. This
mailing list is about the Wikimedia Toolserver, so no wonder that it's
mentioned a lot here. :)
I don't know if you have a toolserver account, or even if you're a
wikimedian. I don't know what you refer to with “my server os will pull
up my IP searches” nor where is that “file known as Mila.eu also known
as ro.eula” supposed to be.
If you were asking for someone to make a query for you in the
toolserver, try including the request in the email, or at least a link
to what you want.

Best regards

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Wikidata tables

[Platonides]

On 19/04/13 01:19, DaB. wrote:
> as you may know there is a rev_text_id-field in the revision-table. This 
> field 
> points to the text-table where the actual text is – or should be. Because the 
> WMF doesn’t store the text here, but only a pointer 
> ("DB://cluster25/11458305" 
> for example). If you query different wikis you will see that most of them 
> point 
> to the same cluster or one with a number short by. That says me (and I was 
> also told so before) that all text of all wmf-projects are stored together.
> The task would now to separate wikidata from the rest – but the storage-area 
> has no clue from where a text is which makes the separating very hard. And 
> there is another problem: Deleted texts are also in this area, so even more 
> filtering would be needed.
> I very doubt that this situation will change at the TS and I also doubt that 
> it will be different for WikiLabs. So I guess your best bet is the API here.
> 
> Sincerely,
> DaB.

I think the only hope would be if wikidata was stored under its own
cluster (for easier differenciation) and at least one server of that
group (the master?) only had that (so toolserver could get its binlogs).

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Output files

[Platonides]

On 18/04/13 02:09, Avocato wrote:
> Hello all. I use this way
> 
> for running crons. I make a file named /something.sh /for example,
> encluding the following:
> 
> /#!/bin/sh
> #$ -j y
> #$ -o /dev/null
> cd pywikipedia
> python anyscript.py/
> 
> Then, I put a cron like:
> /00 21 * * * cronsub -s something % sh $HOME/something.sh/
> 
> The problem is that I want my crons to stop producing output files at
> "home" folder inside my account, I don't want it to produce outputs at
> all. How can I do that?

Remove the -s flag of cronsub. It is giving qsub the parameters -j y -o
$HOME/${JOBNAME}.out and it seems to be overriding the script ones.

So you would put just
 /00 21 * * * cronsub something % sh $HOME/something.sh/

By the way, cronsub is deprecated in favor of qcronsub

Cheers


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] srwiki_p database incosistency

[Platonides]

On 12/04/13 10:12, Marlen Caemmerer wrote:
> Hello,
> 
> On Tue, 9 Apr 2013, Горан Обрадовић wrote:
> 
>> Date: Tue, 9 Apr 2013 23:55:28
>>
>> Around 2000 (~0.01%) revisions from replicated srwiki_p database seem
>> to be missing on Toolserver.
>>
>> For example, revision 5242120 exists in database[1], but doesn't exist
>> on Toolserver.
>>
>> Does anybody know the reason for this anomaly?
> 
> This reminds me of a big bug I was not able to track down.
> It seems we have database inconsistencies quite soon after we setup
> fresh data from WMF db servers.

What kind of inconsistencies? Is it always on the same fields?


> This problem occurs even if we only use the replication integrated into
> MySQL.
> 
> I wrote to the database gurus at WMF about that some weeks ago but I got
> no mail back and I dont know how to resolve this issue.

Barring a mysql bug (hopefully unlikely), and given that WMF slaves are
-supposedly- unaffected I can only think in a configuration difference
which is affecting in the interpretation of the replicated statements.
Is toolserver ignoring some tables? I remember a mysql bug where actions
on ignored tables corrupted replication (but I thought it was fixed!).


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Ipv6 issues (fwd)

[Platonides]

On 07/04/13 09:28, Marlen Caemmerer wrote:
> Sorry, seems I used the wrong sender address...
> 
> -- Forwarded message --
> Date: Sat, 6 Apr 2013 23:59:12
> From: Marlen Caemmerer 
> To: Wikimedia Toolserver 
> Subject: Re: [Toolserver-l] Ipv6 issues
> 
> Hey,
> 
> On Sat, 6 Apr 2013, Merlissimo wrote:
> 
>>
>> I think nosy needs some sleep because she hasn't slept last night.
>>
>> "getent ipnodes willow" does not return the ipv6 address configured at
>> /etc/hostname6.bnx0
> 
> Well...
> rnosy@hemlock:~$ host willow
> willow.toolserver.org has address 185.15.59.202
> willow.toolserver.org has IPv6 address 2a02:ec80:101::2:4
> rnosy@hemlock:~$ host 2a02:ec80:101::2:4
> 4.0.0.0.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.1.0.1.0.0.8.c.e.2.0.a.2.ip6.arpa
> domain name pointer willow.toolserver.org.
> 
> 
> root@willow:~# ifconfig -a
> lo0: flags=2001000849 mtu
> 8232 index 1
> inet 127.0.0.1 netmask ff00
> bnx0: flags=1000843 mtu 1500 index 2
> inet 185.15.59.202 netmask ffc0 broadcast 185.15.59.255
> ether 0:1d:9:71:74:cb
> lo0: flags=2002000849 mtu
> 8252 index 1
> inet6 ::1/128
> bnx0: flags=2000841 mtu 1500 index 2
> inet6 fe80::21d:9ff:fe71:74cb/10
>     ether 0:1d:9:71:74:cb
> bnx0:1: flags=2000841 mtu 1500 index 2
> inet6 2a02:ec80:101::2:4/64

It gives a different output for root? :S

platonides@willow ~ $ ifconfig -a
lo0: flags=2001000849 mtu
8232 index 1
inet 127.0.0.1 netmask ff00
bnx0: flags=1000843 mtu 1500 index 2
inet 185.15.59.202 netmask ffc0 broadcast 185.15.59.255
lo0: flags=2002000849 mtu
8252 index 1
inet6 ::1/128
bnx0: flags=2000841 mtu 1500 index 2
inet6 fe80::21d:9ff:fe71:74cb/10
bnx0:1: flags=2000841 mtu 1500 index 2
inet6 2a02:ec80:101::2:4/64
bnx0:2: flags=20a0841 mtu
1500 index 2
inet6 subnet 2620:0:862:101::/64
bnx0:3: flags=20a0841 mtu
1500 index 2
inet6 subnet 2a02:ec80:101::/64
bnx0:4: flags=2000840 mtu 1500 index 2
inet6 2a02:ec80:101::2:4/64
bnx0:5: flags=2000840 mtu 1500 index 2
inet6 2a02:ec80:101::2:4/64
bnx0:6: flags=2000840 mtu 1500 index 2
inet6 2a02:ec80:101::2:4/64
bnx0:7: flags=2000840 mtu 1500 index 2
inet6 2a02:ec80:101::2:4/64
bnx0:8: flags=2000840 mtu 1500 index 2
inet6 2a02:ec80:101::2:4/64


(bnx0:1,3,4,5,6,7,8 have 2a02:ec80:101::2:4, bnx0:2 2620:0:862:101::)

It looks a problem with the routes:
platonides@willow ~ $ route  get -inet wikimedia.org
   route to: wikimedia-lb.pmtpa.wikimedia.org
destination: default
   mask: default
gateway: vl102-ve6.csw1-esams.wikimedia.org
  interface: bnx0
  flags: 
  recvpipe  sendpipe  ssthreshrtt,ms rttvar,ms  hopcount  mtu
  expire
0 0 0 0 0 0  1500
  0


(correct, it's the same gateway as the linux servers)

platonides@willow ~ $ route  get -inet6 wikimedia.org
   route to: wikimedia-lb.pmtpa.wikimedia.org
destination: default
   mask: default
gateway: fe80::20c:dbff:fefc:b00
  interface: bnx0
  flags: 
 recvpipe  sendpipe  ssthreshrtt,ms rttvar,ms  hopcount  mtu
 expire
0 0 0 0 0 0  1500
  0

fe80::/10 are link local addresses. I expect that the geteway would be
2a02:ec80:101::1 per teh output of the route to be applied on reboot:
platonides@willow ~ $ route  -p show
persistent: route add -inet6 default 2a02:ec80:101::1

2a02:ec80:101::1 and fe80::20c:dbff:fefc:b00 appear to be different
hosts. Although the later does seem to belong to a router, as it runs
RomSShell_4.61 [1]


As a workaround, a bit of googling suggests that writing
DEFAULT_IP=IP_VERSION4 into /etc/default/inet_type should disable ipv6.
Unplumbing the inet6 interfaces should also leave with a working ipv4
system until it is fixed.

Cheers

1- http://www.allegrosoft.com/embedded-ssh-client-server

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] IP Renumbering of the complete Toolserver cluster

[Platonides]

On 04/04/13 11:39, Artem Korzhimanov wrote:
> Hello,
> 
> I'm not sure that this is a problem of a toolserver-side, but I am
> unable to login to (and even ping) submit.toolserver.org
> 
> 
> Probably, it is the problem of DNS because it returns an
> IP-address 81.15.59.215 which is not in a range given by DaB earlier.
> 
> There are no problems with any other hosts maintained yesterday.
> 
> Best regards,
> Artem.

Indeed. 81.15.59.215 is an ip of an Islandic ISP (Stafraen Dreifing -
BTnet)- I suspect it should have been 185.15.59.215. Connecting to that
ip leads you to clematis as expected.



PS: Connecting to other servers from mayapple fails with warnings:
get_socket_address: getnameinfo 8 failed: Name or service not known

**


215.59.15.81.in-addr.arpa domain name pointer adsl-h-59-215.heimsnet.is.

dig +trace submit.toolserver.org

; <<>> DiG 9.9.2-P2 <<>> +trace submit.toolserver.org
;; global options: +cmd
.   42785   IN  NS  l.root-servers.net.
.   42785   IN  NS  b.root-servers.net.
.   42785   IN  NS  i.root-servers.net.
.   42785   IN  NS  e.root-servers.net.
.   42785   IN  NS  m.root-servers.net.
.   42785   IN  NS  k.root-servers.net.
.   42785   IN  NS  h.root-servers.net.
.   42785   IN  NS  c.root-servers.net.
.   42785   IN  NS  d.root-servers.net.
.   42785   IN  NS  f.root-servers.net.
.   42785   IN  NS  j.root-servers.net.
.   42785   IN  NS  a.root-servers.net.
.   42785   IN  NS  g.root-servers.net.
;; Received 699 bytes in 1194 ms

org.172800  IN  NS  c0.org.afilias-nst.info.
org.172800  IN  NS  b0.org.afilias-nst.org.
org.172800  IN  NS  b2.org.afilias-nst.org.
org.172800  IN  NS  a0.org.afilias-nst.info.
org.172800  IN  NS  d0.org.afilias-nst.org.
org.172800  IN  NS  a2.org.afilias-nst.info.
org.86400   IN  DS  21366 7 1 
E6C1716CFB6BDC84E84CE1AB5510DAC69173B5B2
org.86400   IN  DS  21366 7 2
96EEB2FFD9B00CD4694E78278B5EFDAB0A80446567B69F634DA078F0 D90F01BA
org.86400   IN  RRSIG   DS 8 1 86400 2013041100 
2013040323 20580 .
x4Zr+8+dgqzWSs8I29Hqp8lNZdqbZx9Ghj4CBqtWhAK2LFoNwDgQaEpm
EVhCmeZJQhD+3/RPBw7sC51xD3wFAZRvf63JhvOQ4zIGWh7ZjxrJqoDm
drh5c6kSimM2/oF6s/H5pn8UTHWfVLHKl3d7kDeejs4eOQSkNgisPR0n y28=
;; Received 695 bytes from 192.228.79.201#53(192.228.79.201) in 1120 ms

toolserver.org. 86400   IN  NS  a.ns.toolserver.org.
toolserver.org. 86400   IN  NS  b.ns.toolserver.org.
toolserver.org. 86400   IN  NS  c.ns.toolserver.org.
h9p7u7tr2u91d0v0ljs9l1gidnp90u3h.org. 86400 IN NSEC3 1 1 1 D399EAAB
H9Q3IMI6H6CIJ4708DK5A3HMJLEIQ0PF NS SOA RRSIG DNSKEY NSEC3PARAM
h9p7u7tr2u91d0v0ljs9l1gidnp90u3h.org. 86400 IN RRSIG NSEC3 7 2 86400
20130425102001 20130404092001 31380 org.
IMU6xAGbmrHglK04PC4JeByhizESnMfQSY7eU7uYrh1VxzCLWAdCYjvH
+CSzIgixGqBAFXe3vOc1+UA7uQRhT6mrATJZEnocz4A/0HeTtr0Ou76m
cRAcK1NKNOnkm81pZ81Ue+BkXzBOHsroUDIuMNXt/wuKqoNYGjJB6QQX Ufg=
nsponcan0vjpuni6koga4ohtiamtifjf.org. 86400 IN NSEC3 1 1 1 D399EAAB
NSRRP6DVK007BOUB3LSL25BNCSFOO3OH NS DS RRSIG
nsponcan0vjpuni6koga4ohtiamtifjf.org. 86400 IN RRSIG NSEC3 7 2 86400
20130422155807 20130401145807 31380 org.
K18lkybBZGvXT2Hw+u+wuZqvk50u5RwfI6j3gUpQmt/4jRwkIGwpWF7T
ZDyoHy0rdM6ShI4XHz+O7mxK0A/5EOM1Cddbjn5ysfRN6Bt/UskZORtl
g2RxNHmosLgea7gZFcK5WTJE0ns3dbgBKcgp5sPYpnvBirDRwmraGe9t Gfo=
;; Received 698 bytes from 199.249.112.1#53(199.249.112.1) in 551 ms

submit.toolserver.org.  3600IN  A   81.15.59.215
submit.toolserver.org.  3600IN  RRSIG   A 10 3 3600 20130504063421
20130404063421 29051 toolserver.org.
UUEgPU9jEdVYadUwV54hljgpsIh2eRtcmEGmDv2OuA2WsOD/X8orhUeC
tQi8gqgZsiVZBBmgtXJd27vGQfEg2ODmFWIVKH8m/s+gr9lKHAsQymt0
htCH+41tigtXd3JSkneQ0+XghbPJCNA4GpCTNuYaB7BZeltJ3qn2sXq9 znY=
toolserver.org. 86400   IN  NS  a.ns.toolserver.org.
toolserver.org. 86400   IN  NS  c.ns.toolserver.org.
toolserver.org. 86400   IN  NS  b.ns.toolserver.org.
toolserver.org. 86400   IN  RRSIG   NS 10 2 86400 20130504063421
20130404063421 29051 toolserver.org.
ISbKD75gqp0G+z6s0X7vflPG8Qa8oLrIEtHSPTThbNLprQJfMntvRkvX
wWBqkbKJJ5D2o3yMVRZ6HsUx6gbdAo0MCO323du53KxhKq0hJE4x2aSG
1A6X1r3t9micANWYbuUYjyr+rDFGDB80OHcMZ7k9p29r+kUsfw8wK1Z7 bMs=
;; Received 1439 bytes from 208.80.152.226#53(208.80.152.226) in 201 ms

___
Toolserver-l mailing list (Toolserve

toolserver-l@lists.wikimedia.org

[Platonides]

I suspect the above means "I can't login into the toolserver". I have
problems, too (it is asking for a password). Some scripts are also
randomly failing.
Perhaps LDAP is down?

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] /tmp is not a waste dump

[Platonides]

On 17/03/13 19:15, Tim Landscheidt wrote:
>> Why not just install tmpreaper?
> 
> Because it doesn't work on Solaris.  D'oh!
> 
> Tim

{{Reference needed}}


I don't see why it wouldn't work in Solaris. It took me a bit to find
out its source package [1], but it compiles (and seems to run)
flawlessly [2].
I only regret that it doesn't have a blaming option to notify users on
how dirty the left the filesystem. :)


[1]
http://ftp.debian.org/debian/pool/main/t/tmpreaper/tmpreaper_1.6.13+nmu1.tar.gz
[2] willow:/tmp/tmpreaper/tmpreaper-1.6.13+nmu1

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Tool Labs update and news, oh my!

[Platonides]

On 04/03/13 19:30, MZMcBride wrote:
> The Toolserver is a pool of shared resources, with a few roots, and a
> structure that's largely single user. That is, each Toolserver user gets
> his or her own directory inside /home and then works in that area.
> 
> Wikimedia Labs, as I understand it, is generally a pool of shared
> resources, but the idea is to create virtual machines that each
> individually have roots and only a few users per virtual machine (i.e.,
> per project).
> 
> My confusion is how this "tools" project within Wikimedia Labs will be
> structured.
> 
> MZMcBride

A few machines where you install the different tools. So you wouldn't
need to manage a complete VM, or install apache for your web app. You
would place the tool in the appropiate dir and it _should_ just work.
Quite similar to how toolserver worked, but more tool-centric (as if you
created a MMP per tool, but easier).

Disclaimer: This my own conception, I may have understood everything
wrong, etc.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Short downtime for s2 today

[Platonides]

On 15/02/13 21:38, DaB. wrote:
> I admit that there is still room for optimization (z-dat-s2-b is our first 
> Debian-db-server and I started with a clean config), but if you look at the 
> monthly or yearly graph you can see that there were also these peaks before.
> I will look what I can do, but there are several things I have to care for in 
> parallel at the moment.

An option would be to clone the configuration used for wmf servers. It
must be puppetized, and it should apply quite well into debian.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] New Rule: SGE-constraint for bots

[Platonides]

On 11/01/13 23:01, DaB. wrote:
> c.) if you start a bot by hand for testing 
> (no screen, no cron, no while).
Does this forrbid to run a bot by hand with an attached screen?
Ie. you have run screen because you are afraid your local
computer/connection could reset, not to ‘run & forget’. You would be
attached eg. 90% of time, and pay (some) attention at the screen output...

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] s2/s5 will be read-only at Saturday

[Platonides]

On 09/02/13 18:46, DaB. wrote:
> A more unpleasant news: I received a SMS by Nosy telling me that she is in 
> the 
> hospital for at least 1 week. Let us all hope that she and her child will be 
> better soon!
:(

Do you know who was who needed hospitalization? I hope it was not her
baby. I wish all your family gets healthy soon, Nosy.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] New Rule: SGE-constraint for bots

[Platonides]

On 11/01/13 23:43, DaB. wrote:
> Hello,
> At Friday 11 January 2013 23:42:03 DaB. wrote:
>> Is TS-1479[1] a valid exception B for not using SGE for some specific
>> scripts?
> 
> yes. But I will speak with Merlissimo about a fix for this (AFAIR he had a 
> problem with the given patch).

It looks good to me (note it is *not* using embedded quotes as suggested
in comment #1).

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Crons overflow/server overload on Willow (other)

[Platonides]

For the record, sun crontab seems to be working again since 15:50 UTC.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cron on willow

[Platonides]

On 11/01/13 15:01, Carl (CBM) wrote:
> On Fri, Jan 11, 2013 at 8:33 AM, Platonides  wrote:
>> /usr/sbin/cron does seem to be running :S
> 
> It is, but something is not right. The line
> 
> * * * * * /home/cbm/touch.sh
> 
> at the beginning of my crontab is not being executed.  The script runs
> fine from the command line on the same host (and a script error should
> give an email from cron rather than silence, anyway).

I know, I know. Wolfgang ten Weges may be right in that it's overloaded.


>> fyi, cronie is working fine.
> 
> That's not particularly relevant for people who have set crontabs with
> /usr/bin/crontab
> 
> - Carl

It was given as an aside, not as a solution although I guess you could
migrate:
 crontab -l | cronie


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cron on willow

[Platonides]

On 11/01/13 14:23, Carl (CBM) wrote:
> On Fri, Jan 11, 2013 at 8:11 AM, Platonides  wrote:
>> Are you using Solaris crontab or cronie?
> 
> /usr/bin/crontab, like always. I have checked that the same scripts
> work with crontab on nightshade, so it is something specific to
> willow.  But it was working correctly on willow up until a few days
> ago, when it seems to have broken somehow.
> 
> - Carl

/usr/sbin/cron does seem to be running :S

fyi, cronie is working fine.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Cron on willow

[Platonides]

On 11/01/13 13:45, Carl (CBM) wrote:
> Has anyone else noticed this? I am having problems with cron on
> willow, which does not seem to be executing my crontab. I tried to
> test it by adding a line to my crontab that should execute every
> minute:
> 
> * * * * * /home/cbm/touch.sh
> 
> That script runs correctly from the command line, but cron does not
> seem to execute it.  Some of my other other cronjobs appear not to
> have run in a couple days, even though they were working fine for
> months before.
> 
> - Carl

Are you using Solaris crontab or cronie?


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Can't explain queries?

[Platonides]

On 04/01/13 21:05, OQ wrote:
> On Fri, Jan 4, 2013 at 1:55 PM, Aaron Halfaker  
> wrote:
>> Hello,
>>
>> I've been working on a new script that will access the MySQL database.  In
>> testing this script, I was worried about the complexity of some of my
>> queries, so I naturally went to the MySQL client and ran them with
>> "explain".  It appears that I do not have permission to do so.  Is this
>> intended?  Is something wrong?
> 
> EXPLAIN can leak information from underlying tables, so if you dont
> have permissions for those tables the explain will fail with the error
> message provided.

It was possible until the last mysql update. And yes, being able to
EXPLAIN queries is *very* useful. Otherwise the already-a-guessing-game
of making efficient queries is even harder.

I don't think an EXPLAIN can leak information not already available to
us by querying the view (possibly multiple times).

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Merry Christmas

[Platonides]

+20

Merry Christmas to 255.255.255.255
:)



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] My meeting with Denis Barthel today

[Platonides]

On 20/12/12 16:58, DaB. wrote:
> Hello all,
> 
> I just came back from a meeting with Denis Barthel. Denis is a long-time 
> employee of Wikimedia Deutschland (WMDE) and works in the resort "Community".
> Denis was assigned to help to normalize the relationship between the 
> Toolserver, WMDE and the Wikimedia Foundation (WMF); especially in the field 
> of 
> WikiLabs/Toollabs. His opinion is that if we all work together the move from 
> the toolserver to Wikilabs can work. I still doubt that, but I agreed to give 
> WMF a little more AGF (assume good faith).
> As you know the general meeting of WMDE decided that the WMF has to guarantee 
> that the features of the toolserver exists in WikiLabs within 6 months 
> (otherwise WMDE has to look for a way to continue the toolserver). Denis 
> asked 
> me to provide such a list of short-time-base. 

The toolserver allows propietary tools. This is required by some users
and should thus be considered a toolserver feature (“i wrote code at
work for my company and reuse parts for my bot framework. I have not the
right to declare this code as open source which is needed by labs
policy.” [1]).

Given that it has been stated from WMF side that closed-apps won't be
supported, it seems impossible that “the features of the toolserver
exists in WikiLabs”.


[1] Mail on Sep 26th by Merlissimo.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] JIRA session loss

[Platonides]

On 14/12/12 16:42, DaB. wrote:
> sorry, no news yet. If the situation doesn't change until the new year we 
> will 
> switch to another software like bugzilla (can anybody recommend one?)
> 
> Sincerely,
> DaB.

What features should it have?

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Result of the general member meeting of WMDE

[Platonides]

On 04/12/12 09:07, Chris Grant wrote:
> On Tue, Dec 4, 2012 at 5:05 AM, Magnus Manske wrote:
>> Doubleplusgood for DaB staying in the fight for another round!!!
> 
> Seconded! Following that analogy, I hope he knocks a few teeth out!
> 
> (I'm joking, I'm joking - the last thing we want is an aggressive
> relationship between the toolserver and everyone else)
> 
> -- Chris

So, you are proposing a relationship in the ministry of Love? ;)



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Result of the general member meeting of WMDE

[Platonides]

On 26/11/12 19:33, Dr. Trigon wrote:
> Another question; I just learned about "Wikimedia Polska
> toolserver" [1], how are we related to them? Are we at all? Is it
> the same as the german one but paid from poland wikimedia? (May be
> you can get admin there, if you like?)

AFAIK it's a more simple server. Like the German Toolserver at the
beginning. Noticeably, with no db access, which is what makes TS special.
(Why would he want being an admin there? Not enough problems here? :D
Do they actually *need* an admin?)

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Anyone else having TS trouble?

[Platonides]

On 25/11/12 16:42, Magnus Manske wrote:
> Started a few minutes ago. getting alternating errors on reloading a
> page; 500 (on a static HTML page, not CGI, mind you!), "This webpage is
> not available: The connection to toolserver.org 
> was interrupted.", 404.

The same thing happpened here.
The servers didn't allow logins, and accessing the web pages return
404s. Apparently ldap was down.
It recovered now. I don't know if it there was a human involved, if it
solved itself or if it was turnera taking over damiana automatically.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] SGE queue waiting forever?

[Platonides]

On 24/11/12 21:38, Dr. Trigon wrote:
>> @All: If you are working on big files please copy them to local 
>> temp first (on sge $TMP contains an individual temp dir for the 
>> job). E.g. piping big files to other slow programs causes much
>> nfs load because data must be read in small packages which cause
>> high load on servers. That's why sge cannot schedule new jobs on 
>> nightshade since days.
> 
> What is a big file? Is it ok if the file is in user-home?
> 
> Thanks and greetings DrTrigon

/home is also mounted with nfs

Although it's strange that reading from big files overloads the
servers. stdio or the equivalent functionality in the language they
are made should be making it work in blocks.

Looking at willow mounts, /shared and /home are mounted with nfsv3
over udp. But /mnt/user-store and /install don't show it, so they are
probably using nfsv4 over tcp. Is that intended?



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] SGE queue waiting forever?

[Platonides]

On 24/11/12 17:37, Wolfgang Faust wrote:
> Logging in to submit.toolserver.org  takes
> a really long time recently (starting a few days ago). Clematis doesn't
> seem to have any load though, so I don't know what's going on.

Thousands of processes running df -k /mnt/user-store? :)

$ ssh submit "ps -ef | grep -c 'df -k /mnt/user-store'"
15408

Seems we have /mnt/user-store problems again.
clematis ~ $ time ls /mnt/user-store
NFS server thyme not responding still trying
NFS getattr failed for server thyme: error 16 (RPC: Failed (unspecified
error))
ls: cannot access /mnt/user-store: Connection timed out

But those instances are stuck. There are processes since Nov 12.
Seems that nosy just killed them.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Defective database s7

[Platonides]

On 16/11/12 21:44, Marlen Caemmerer wrote:
> Now I tried several thing to check which table might be corrupted.
> Innodbchecksum reported everything fine.
> Mysqlcheck crashed  the mysql daemon when accessing centralauth.localnames.
> Oh? Why? Checking the table again crashes mysql. Hm.
> Tried a repair table - "storage engine does not support this"...hm.
> 
> The log says
> InnoDB: Page lsn 268 3672100478, low 4 bytes of lsn at page end 3672100478
> InnoDB: Page number (if stored to page already) 192520,
> InnoDB: space id (if created with >= MySQL-4.1.1 and stored already) 428
> InnoDB: Page may be an index page where index id is 0 1174
> InnoDB: (index "PRIMARY" of table "centralauth"."localnames")
> InnoDB: Error in page 192520 of index "PRIMARY" of table
> "centralauth"."localnames"
> 121116 13:02:46 - mysqld got signal 11 ;
> 
> I tried to remove the index  to rebuild it but this does not work due
> to  innodb_force_recovery = 3.

You may need to set  innodb_force_recovery back to 0 before being able
to drop the index (we got lucky there if it's just the index).

http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html:
> The database must not otherwise be used with any nonzero value of 
> innodb_force_recovery. As a safety measure, InnoDB prevents users from
> performing INSERT, UPDATE, or DELETE operations when
innodb_force_recovery
> is greater than 0.

(I would have expected DROP INDEX to work, as DROP or CREATE tables are
allowed in innodb_force_recovery, but seems it's not)



> Mysqldump fails - crashes the mysql daemon too.
I expect it only crashes when dumping that table :)

You may be able to get the values out of the table by forcing usage of
the other index (ln_name, ln_wiki). Although it could be safer to get a
new dump for that able from wmf, or a mixture of them (such as getting a
dump at both sides, then rsyncing).


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] thyme – postmortem

[Platonides]

On 13/11/12 01:46, DaB. wrote:
> Thyme also carried my wikidata-replication-program which failed too (so the 
> replag of wikidata everywhere increased). I moved it to another server now.
> A strange thing is that the mysql-process on thyme is still running; even 
> replication is working so the replag will not increase there.

What about puppetd ?
If it's still running, it could provide a way to restart the server.

It looks as if sshd had simply died. I would had blamed the OOM killer
but thyme is Solaris, and it affected both sshd and the nfsd at the same
time... (but not mysqld). Maybe an error with the filesystem umounting
itself, but in that case sshd should still work.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] mdbtools

[Platonides]

On 13/11/12 09:12, Marlen Caemmerer wrote:
> Hello,
> 
> I am a bit confused.
> I found libtool installed on yarrow and nightshade which are the linux
> userland servers.
> I tried to autogen.sh the version of mdbtools from github and it worked
> until it tried to generate the man pages but this problem could be
> solved quite quick too.
> So I have a working copy of mdbtools 0.7 in my home done by autogen.sh.


It's strange:
> platonides@nightshade ~$ cd /mnt/user-store/mdbtools/brianb-mdbtools-004cc9f
> platonides@nightshade brianb-mdbtools-004cc9f $ ./autogen.sh 
> /usr/bin/libtoolize
> 
> **Error**: You must have `libtool' installed.
> Get ftp://ftp.gnu.org/pub/gnu/libtool-1.2d.tar.gz
> (or a newer version if it is available)

Maybe it's not really libtool but some other dependency and the error
message is misleading, what it's needing, but it's odd that it worked
for you...

Ok, there is a bug in autogen.sh, it tries to run "libtooloze", not
"libtoolize":
> which libtoolize && (libtooloze --version) 

Fixing that, it does compile (not even failing at the man pages).
The result is at /mnt/user-store/mdbtools/mdbtools-0.7 and it seems to
work with Extract_ODB_V10.2.10.mdb.

Marlen, you probably ran autoconf directly so you didn't hit the
libtoolize call.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] SQL-S1 maintenance

[Platonides]

On 12/11/12 00:04, DaB. wrote:
> Hello,
> At Sunday 11 November 2012 23:54:39 DaB. wrote:
>> Remember that there is a trick where you dump the user-databases with
>> the binlog enabled (and noting its position), leaving the database
>> writable. You then transfer and import the dump. Finally you make it
>> readonly, copy the changes from the binlog (which will be much smaller
>> and thus results in a shorter read-only interval) and point the dns to
>> the other server.
> 
> AFAIK this would need a complete LOCK of all tables during the dumping (to 
> get 
> a consistent state) which would be the same as switching the server 
> read-only. 
> Also somebody would have to write a program that read the bin-log and filter 
> out all non-user-database-entries (enwp, commons and heartbeat in this case). 
> The 3rd disadvantage would be that you would have to wait for the dump to 
> complete before you can start the import,
> The only advantage I see it that sql-s1-user would be writable between the 
> end 
> of the dump and the time I awake in the morning.
> 
> Sincerely,
> DaB.

You are completely right. Still, I suspect it *is* possible to do so
efficiently. Starting a global read transaction and then resuming writes
(I am assuming that all will be InnoDB, which is very likely).
Maybe it needs a special tool.

I am copying Asher in case he can enlighten us.

There are tools for the binlog filtering, but it is not a problem in
this case, since we could just stop replication before the export and
let the lag rise during the backup. In the general case we would
reenable them and let the server pick up after it finishes, but in this
case we are just going to overwrite it :)


How much data is in user tables and how long does it take to import the
(uncompressed, ready) wiki tablespaces?
It might turn out to be faster to switch the tablespaces than moving the
dbs around.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] SQL-S1 maintenance

[Platonides]

On 11/11/12 17:37, DaB. wrote:
> I will switch sql-s1-user to read-only at 
> 
> tomorrow, Monday after 20:05 UTC
> 
> and than dump the user-databases (and import them on thyme). I do not know 
> how 
> long this will take, but I guess the hole European night. When the import is 
> done somewhen on Tuesday I will switch sql-s1-user to thyme and switch it to 
> read/write. Later (or in parallel to the user-database-dumping, not sure yet) 
> Nosy or I will dump commons from rosemary too for a later re-import.
> When everything is dumped and imported I will re-setup rosemary with the 
> fresh 
> dump too.
> 
> Sincerely,
> DaB.

Remember that there is a trick where you dump the user-databases with
the binlog enabled (and noting its position), leaving the database
writable. You then transfer and import the dump. Finally you make it
readonly, copy the changes from the binlog (which will be much smaller
and thus results in a shorter read-only interval) and point the dns to
the other server.

Regards

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] mdbtools

[Platonides]

On 10/11/12 16:58, Andre Koopal wrote:
> Hi,
> 
> Today I am working on a project where I need to convert an access database
> dump to something else (mysql likely). Most things I found on the web
> are not really suitable to script this process (it will need to run every
> month), but mdbtools looks to be promising. However I tried to compile
> that on toolserver all I got out of it are coredumps :-(. (Tried both on
> willow and nightshade). The ubuntu pacakge does work on my home server.
> 
> Did anybody ever look into mdbtools and have it working? Is it perhaps
> possible to have it globally installed on toolserver, as it probably is
> usefull to some of the glam-projects as well?

I was able to compile mdbtools-0.6pre1 for linux, see the result at
/mnt/user-store/mdbtools

I admit that compilation was a bit tricky.  When it complains that
backend.c:31: error: static declaration of 'mdb_backends' follows
non-static declaration, remove the static keyword from that line.
yacc and flex are not installed in nightshade, I worked around that by
running those of willow.
The flex rule expected an output file of .c, so I ran ln -s lex.yy.c .c
(otherwise you get an empty lexer.c)

However, the generated binaries didn't output any table for your file.

I then realised that my local mdbtools does work with yout file. Seems
that mdbtools-0.7 is in GitHub, but not in SourceForge:
https://github.com/brianb/mdbtools

However, in order for just running autoconf.sh, it needs libtool, which
is not installed in Linux or Solaris.





> 
> If people want an mdb file to test with, the file I need to convert is
> at /mnt/user-store/rce-nl-data.
> 
> Regards,
> 
> Andre
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette
> 


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt?

[Platonides]

On 04/11/12 02:29, Erik Moeller wrote:
> My most recent inquiry to you and Nosy was unrelated (I asked for the configs 
> so
> we can compare them as we begin setting up replication in Labs.)
> 
> Erik

It's the first notice I get that labs replication is being set up. Who
is handling this? Asher? I would expect Ryan to be involved, but he is
currently on vacation...

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] rosemary's copy of enwiki_p also corrupt?

[Platonides]

On 04/11/12 00:56, DaB. wrote:
> Shortly after Asher Feldman provided a dump in the new xtrabackup-binary-
> format the wmf uses for dumps. It took ~2 days to transmit this dump to the 
> toolserver. 

Was the rate-limiting issue fixed?


> It is the first time the toolserver has to handle this new format and it took 
> some days to find the needed tools (the wmf uses Ubuntu for their database-
> servers while we uses Solaris). 

This may be a good time to migrate one of rosemary/thyme to Debian.


> Asher told me that the dump is compressed in 
> some way, but I could not decompress it. While I tried to figure out how to 
> decompress the file, I accidentally overwritten the file at Thursday.

Failures happen.
Looking aroound, it isn't obvious the way they get compressed. Given you
could not decompress it initially, it probably used xbstream (the other
common compression format with xtrabackup seems to be using a gzipped tar).
I would have expected Asher to provide you a full page documenting how
to restore the backups, though.
"How to restore a backup" is a critical piece which can't be left
undocumented (or untested).


Thanks for the update, DaB.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Maintenance of a Switch tomorrow

[Platonides]

On 25/10/12 14:59, Marlen Caemmerer wrote:
> Hello,
> 
> tomorrow a switch in one rack will be replaced. This means some downtime
> of about 30 min probably tat will occur between 9 am and 13 am utc.
> Sorry for not having a better time frame but the exact arrival of the
> onsite technician is not know to me.
> 
> Cheers
> nosy

Please send out a notice when it's done. It is planned to open the WLM
survey (and thus start advertising it) tomorrow at 15:00 UTC.
It would be sad that the technician came very late and the maintenance
took place just when lots of users are getting notified to go to the TS
to fill out the survey. :)

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Database connection problem

[Platonides]

On Mon, Oct 22, 2012 at 1:40 PM, ant  wrote:
> Hi,
>
> I'm new to the toolserver and I'm trying to install a web application[1].
>
> I created a database on sql-s1-user to which I can connect via my PHP
> code. Also I have a C program that needs to connect as well. If I call
> the C program from the command line, it works perfectly. However, if it
> is called from the PHP script using shell_exec() by the web server user,
> I get the following error message:

The program is compiled for solaris?


> Access denied for user 'ant'@'damiana-bge0.esi.toolserver.org' (using
> password: NO)

This is clearly wrong, as you have to provide a password in order to connect.

Maybe your program is relying in $HOME being set? I think you need to
get it from
getpwuid()

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Mail forwarding not working

[Platonides]

On 17/10/12 23:23, Tim Landscheidt wrote:
>> Actually, no need to replace it. $USER is set automatically.
>> You can run:
>> $ for SERVER in clematis hawthorn nightshade ortelius willow wolfsbane
>> yarrow; do ssh $USER@$SERVER.toolserver.org test -s /var/mail/$USER &&
>> echo $SERVER; done
> 
>> and it will list for you the servers where you have mail.
> 
> Actually, that's only true for those whose username on the
> toolserver is the same as on their private box :-).  I'd
> have to name my username on the toolserver either explicitly
> or rely on my ~/.ssh/config:
> 
> | Host *.toolserver.org
> |   User timl
> 
> and always escape the second $USER so that it is expanded on
> the toolserver.
> 
> Tim

I had run the above in the login server. Hadn't noticed you were doing
so from local.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Mail forwarding not working

[Platonides]

On 17/10/12 22:53, Tim Landscheidt wrote:
> Hi,
> 
> for those of you not having seen TS-1553, mail forwarding
> seems to have stopped working.  So if you haven't received
> the usual job reports that you were expecting, you might
> want to login to all servers and check if there is mail for
> you.  You can query all servers by:
> 
> | for SERVER in clematis hawthorn nightshade ortelius willow wolfsbane 
> yarrow; do
> |   ssh $USER@$SERVER.toolserver.org ls -l /var/mail/$USER
> | done
> 
> replacing $USER with your username.
> 
> Tim

Actually, no need to replace it. $USER is set automatically.
You can run:
$ for SERVER in clematis hawthorn nightshade ortelius willow wolfsbane
yarrow; do ssh $USER@$SERVER.toolserver.org test -s /var/mail/$USER &&
echo $SERVER; done

and it will list for you the servers where you have mail.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Query killer and replag

[Platonides]

On 14/10/12 13:53, John wrote:
> Either tsbot or the Query killer are broken I just got an email about
> a killed query (Not the issue) and query killer said the lag was
> around 14000 seconds while tsbot and the relag graph are showing
> almost no lag.

The enwiki db has pretty much no lag in the user server, but a lag of 5h
in the round-robin one (yes, that's quite strange).


BTW, may I ask why are you using 17520 as limit? Seems an odd choice.




___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] How to have qsub mail output?

[Platonides]

On 01/10/12 14:19, Tim Landscheidt wrote:
> Thanks.  I don't want to fiddle to much with SGE's in-
> testines, so I will probably either use "| mail timl" in my
> script or have my MUA insert the log in the status mail.
> 
>   I looked if I could submit this totally fascinating and
> innovative idea of mailing the output as a RFE upstream, but
> amazingly I didn't see a bugtracker at Oracle :-).  I would
> even have had another idea: Impromptu jobs à la "echo true |
> at now" :-).
> 
> Tim

Oracle closed-sourced it. There are a number of forks.
Quick link: http://gridengine.org/blog/2011/11/23/what-now/

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 01/10/12 01:28, Carl (CBM) wrote:
> * "User databases will not be able to be joined against replicated
> databases." The reasoning behind this seems to be a misunderstanding
> of the role of "application logic".  For Mediwiki itself, which works
> with only a few articles at a time, joins can be efficiently simulated
> within Mediawiki itself, or by making changes to the database schema
> on the live servers. For a toolserver application that works with
> millions of articles in a single query, it is silly to essentially
> re-implement a SQL engine in the application logic - joins of this
> size, which may require filesorts, should be done at the database
> server level, not at the application level.  That's why we use a
> database server in the first place.

Note that it's false that MediaWiki doesn't perform joins.
There are many of them. The statement about that was (probably) talking
about cross-db joins.

But the reason for using different dbs on toolserver is based on user
permissions, not in that those tables conceptually should go with the
data in the wiki db.


> Looking at the discussions about labs in the past few days, it seems
> clear to me that labs will be useful for some projects, particularly
> for developing Mediawiki extensions.

Even for MediaWiki extensions, not having user tables can be hard. If
your extension needs a new table, with you can create it at another db
and link them with $wgSharedTables (a configuration change, without
touching the extension code).
Without that, testing one extension with an additional table would need
it to be created in the master wiki db! (plus have a user with
permissions to write it, and all related maintenance)

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Platonides]

On 01/10/12 13:03, Tim Landscheidt wrote:
> Even more: If Labs replication isn't bound by Toolserver
> tradition, it would be *very* nice not to fragment the data
> according to the different WMF clusters, plus Commons or
> not, plus (separate) user databases or not, but have one
> cluster where users can join as logic suggests.  As Toolser-
> ver merges Commons onto other clusters already, this seems
> to be possible with MySQL.
> 
> Tim

It's possible, you just need bigger servers which can hold all dbs.
(plus some master/slave replication for user tables)

I suppose it will use the same clusters as WMF. After all, there's a
reason the WMF clusters needed to be splitted.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] How to have qsub mail output?

[Platonides]

On 01/10/12 13:21, Tim Landscheidt wrote:
> (anonymous) wrote:
> 
>> Please see the documentation on Toolserver wiki [1] to set this up. It uses 
>> a command of -m with another variable depending on when you want mail sent.
> 
>> [1] 
>> https://wiki.toolserver.org/view/Batch_job_scheduling#arguments_to_qsub/qcronsub
>> [...]
> 
> That just sends notifications about the job's status, not
> the job's output itself.
> 
> Tim

No, there's no such option.
I looked into it recently. We could get that by adding a final script
which mails the user the output file if qstat -j $JOB_ID | grep -q
mail_options:.*e.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 30/09/12 03:31, Krinkle wrote:
> On Sep 29, 2012, at 9:41 PM, Samuel Klein  wrote:
>> And why is the WMF considering not providing db replication for it?
> 
> [citation needed]
> 
> I think you misunderstood.
> 
> -- Krinkle

Written by Erik Moeller the 25th Sep:
> Chapters are autonomous organizations, and it's WM-DE's call how much
> / whether it wants to continue to invest in infrastructure of any kind
> (and the decision of funding bodies like the FDC to accept or reject
> that proposition). However, for our part, we will not continue to
> support the current arrangement (DB replication, hosting in our
> data-center, etc.) indefinitely.

He is writing with his "WMF hat", the above 'we' refers to the WMF. So no, 
there's no misunderstanding: The WMF has threaten to stop providing the db 
replication.
That would be a really bad move to do, of course (and a gratuitous one, 
since there aren't technical or financial reasons for doing that). And I 
hope they won't do that.

- http://lists.wikimedia.org/pipermail/toolserver-l/2012-September/005294.html

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Platonides]

On 27/09/12 01:07, Ryan Lane wrote:
>>> We currently have no plans for having the user databases on the same
>>> servers as the replicated databases. Direct joins will not be
>>> possible, so tools will need to be modified.
>>
>> -50
>>
>> It's such a useful feature, that it would be worth making a local mysql
>> slaves for having them.
>> I know, the all-powerful labs environment is unable to run a mysql
>> instance, but we could use MySQL cluster, trading memory (available) to
>> get joins (denied).
>>
> 
> I'm not the one setting up the databases. If you want information
> about why this won't be available, talk to Asher (binasher in
> #wikimedia-operations on Freenode). Maybe he can be convinced
> otherwise.
> 
> Of course, in the production cluster we don't do joins this way. We
> handle the joins in the app logic, which is a more appropriate way of
> doing this.

I disagree. In production you can just create a new table in the wiki
db. We can't create new tables there in the toolserver (the dbs are a
mirror or what there is in production). Thus, we create a new db in the
same server and use a cross-db join instead of joining a new table.

Joining several wiki tables is probably more strange, with the exception
of commons, which is more often joined to others, as the commons images
"are also at the local wikis".

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Platonides]

On 27/09/12 17:21, Andrei Cipu wrote:
>> Additionally, we have a number of volunteer driven projects. Here's a
> 
>> few choice ones:
>>
>> * Bots
>> * Deployment-prep
>> * Maps (for OpenStreetMaps)
>> * Wikistats
>> * Wikitrust
>> * Signwriting
>> * Phabricator
>> * Metavidwiki
>> * Huggle
>> * Glam
>> * Wiki loves monuments
>> * Blamemaps
>> * Counter vandalism network
>>
> 
> Where can we find more information about these projects, especially OSM and 
> WLM?

You can go to https://labsconsole.wikimedia.org/ List projects, take a
look at project members and bug them to tell you their evil plans :)

The project for WLM is actually one for making a tool for judging the
images (Wlmjudging). There are a couple of VMs, I don't know if it
produced something. I'd ask Ynhockey about it.

All "real" work for Wiki Loves Monuments is done in the toolserver (plus
http://wlm.wikimedia.org/ which has a copy of the data produced at TS).


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Platonides]

On 26/09/12 20:25, Ryan Lane wrote:
>> temporary blockers
>> * no replication of wikimedia wiki databases
>> ** joining of user databases with wiki databases
> 
> We currently have no plans for having the user databases on the same
> servers as the replicated databases. Direct joins will not be
> possible, so tools will need to be modified.

-50

It's such a useful feature, that it would be worth making a local mysql
slaves for having them.
I know, the all-powerful labs environment is unable to run a mysql
instance, but we could use MySQL cluster, trading memory (available) to
get joins (denied).




>> * no support for script execution dependency (on ts: currently done by sge)
> 
> There's less of a need for this in Labs. If whatever you are running
> is really expensive, you can have your own instance. That said, I was
> looking at integrating a global queuing system. It won't be SGE,
> though.
> 
> If someone is really keen on SGE, then I recommend they work with us
> to puppetize it. Thankfully, open grid engine is already packaged in
> ubuntu, which should make that much easier.

SGE is a strong queue system. We have people and tools already trained
to use it. It would be my first option.
That said, if the presented alternative has the same user interface, it
shouldn't be a problem. For instance, I don't have an opinion about
which of the SGE forks would be preferable.


>> * no support for servlets
> 
> I'm not sure what you mean by servlet?

J2EE, I guess.




>> * no DaB.
>>
> 
> I'd love DaB to help us improve Labs.
> 
> Everything about Labs is fully open. Anyone can help build it, even
> the production portions.
> 
> - Ryan

Would it be worth our efforts? I sometimes wonder why we should work on
that (yes, I'm pessimistic right now).
For instance the squid in front of *.beta.wmflabs.org. It was configured
by Petan and me. We had absolutely no support from the WMF. The squid
wasn't purging correctly. It worked on production, so there was a config
error somewhere.
We begged to see the squid config for months. But as it was in the
private repository, no, it can't be shown, just in case it has something
secret (very unlikely for squid config). Yes, we will clean them up and
publish, eventually. Months passed (not to mention how publishing the
config had been requested years ago). It could have been quickly
reviewed before handing out, and we weren't going to abuse it if there
really something weird was there. Replicating the WMF setup was done
without viewing that same setup. I finally fixed it. I was quite proud
of having solved it.
Where is that file right now? It vanished. The file was lost in one of
the multiple corruptions of labs instances. It was replaced with a copy
of the cluster config (which was finally published in the meantime).
So it feels like wasted effort now. I'd have liked to save a local copy
at least.

It's not enough to leave tools there and say "It is fully open. Anyone
can help build it"


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 26/09/12 21:20, Ryan Lane wrote:
> Labs is an infrastructure for building infrastructure. It isn't a
> Toolserver replacement.
> 
> - Ryan

Yet at high level it has been treated as such replacement.
We could also replace the Wikimedia servers with turing machines.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 26/09/12 20:55, Ryan Lane wrote:
> They can be created in LDAP by making a labsconsole account.
> Additionally, unless the account needs to directly log in via ssh,
> there's no request process needed for the user to be used.
> 
> Of course right now there isn't open registration in labsconsole.
> We're waiting on some Jenkins changes for that to happen. Should be
> any day now.
> 
> Alternatively, the user could be created as a system account via puppet.
> 
> - Ryan

I thought the jenkins changes were ready, after Antoine wrote:
> I have finished the preparation work in Jenkins. It will be deployed as
> soon as we agree on the new workflow, which I still have to write down.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 26/09/12 12:40, Sumurai8 (DD) wrote:
> 2012/9/26 Platonides:
>> On 26/09/12 08:27, Federico Leva (Nemo) wrote:
>>>> Yes, I know about MMP. It was introduced much later, after people were
>>>> already very used to doing things in an individualized manner. From
>>>> what I'm told MMPs aren't used much.
>>>
>>> So you think that even new users (I believe many users arrived after MMP
>>> were introduced) are not using MMP due to bad old habits of the other
>>> users?
>>> I've had a hard time trying to convince some users to use a MMP; I think
>>> it's not trivial to understand what models actually work and are liked.
>>
>> The reason is as simple as being much more easier, run mkdir and start
>> coding vs open a jira request to get a new MMP for a project which may
>> or may not be interesting (heh, most tools were probably born after
>> coding for a couple of hours after having a good idea) and which nobody
>> is wanting to maintain with you, probably.
>>
> 
> It is not hard to start coding in your own workspace and move code
> from your newly created directory to a MMP. A MMP doesn't necessarely
> have to be maintained by multiple users. While it's convenient having
> someone in the project that can jump in in case your tool breaks badly
> and you are not around to fix it, it is not by any means necessary.

Right. I was explaining why there aren't so many MMPs out there.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 26/09/12 08:27, Federico Leva (Nemo) wrote:
>> Yes, I know about MMP. It was introduced much later, after people were
>> already very used to doing things in an individualized manner. From
>> what I'm told MMPs aren't used much.
> 
> So you think that even new users (I believe many users arrived after MMP
> were introduced) are not using MMP due to bad old habits of the other
> users?
> I've had a hard time trying to convince some users to use a MMP; I think
> it's not trivial to understand what models actually work and are liked.

The reason is as simple as being much more easier, run mkdir and start
coding vs open a jira request to get a new MMP for a project which may
or may not be interesting (heh, most tools were probably born after
coding for a couple of hours after having a good idea) and which nobody
is wanting to maintain with you, probably.



Ryan wrote:
> Toolserver's model is fundamentally different. It's based on an old
> concept of shared hosting. Labs is built on a model more like a VPS
> (really more like EC2). Due to that, it's possible to give users far
> more rights. 

labs model didn't exist when the toolserver started. This route was the
only 'normal' one to follow, specially with a single server.
It even looks too new for labs right now.



> If you guys want to build the exact same Toolserver
> environment as a Labs project, go for it. I have a good feeling you'll
> start doing things differently when you see the affordances given by
> having more rights, though. 

I have a few ideas on how to improve it on webtools, based on toolserver
experience, but without big changes.

> MMPs aren't a very elegant solution. I'd prefer not to
> force an inelegant solution into a system that allows much more
> elegant approaches.

Actually, I'm unsure on how to replicate MMP accounts there. It
shouldn't need to manually create them everywhere. The elegant solution
is probably to have those accounts in LDAP... Any ideas?


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 25/09/12 23:38, Federico Leva (Nemo) wrote:
> Indeed the first answers on
> 
> seem to indicate that all tools and services currently on Toolserver
> will just be trashed and people forced to reinvent them from scratch
> because a completely different approach is thought to be better.
> 
> Nemo

Both approaches have its merits. The WMF could have copied the
toolserver recipe much more easily than gooing the route of labs. They
were brave pursuing the VM based system, and it allows solving different
problems.
But both of them have their use. It's not a case where one should "win"
the other, or the toolserver is "bad" because it is currently hosted by
WMDE instead of WMF.

Organizations trying to defeat one another will end up with one big
loser: the community.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 25/09/12 20:48, Erik Moeller wrote:
> 1) WMF is a technology organization. Hosting the core infrastructure
> for Wikimedia projects is very much what we do. This includes data
> center operation, monitoring and backups, software deployments,
> software/service upgrades, code versioning infrastructure, bug
> tracking infrastructure, additional support systems and services (like
> this mailing list), etc.
> 
> Toolserver is in fact hosted by the Wikimedia Foundation today, in our
> Amsterdam data-center. We provide space, power and racks for the
> toolserver cluster, at a cost of about $65,000/year to WMF according
> to our Director of TechOps.

Something we should all be grateful for.


> We also maintain the database replication
> on our end which enables tools to function.

A replication which WMF is already doing for itself, adding an extra
slave hosted by a trusted party doesn't make a techical difference.
While WMF gains a backup server (at some point the toolserver was the
only externally hosted backup). It has also inadvertedly dropped the
binlogs needed by TS when doing server cleanup.

So while very welcome, the WMF side of the db replication is not the
most important piece of the toolserver (having a db copy still makes the
TS better than any other alternative available at the time, though).
The value is probably in how WMDE was able to open that database and
many fruitful tools emerged.



> We can't provide the same level of service for the toolserver
> infrastructure as we do for core operations, and it makes no sense for
> a chapter to build out the required staffing and expertise to do so
> (set up/maintain all or some of the aforementioned functions). Even
> with slightly increased investment, toolserver would always suffer
> from being second or third tier infrastructure.

labs is also a second class citizen.


Moreover, it is explicitely stated not to be for production-like level.

What will happen if a really successful tool reaches to a point where it
de-facto needs it ? (eg. a WLM tool, tools to Request to get an Account
Created, or Request an Unblock appeal...)



> 2) We're not comfortable hosting the toolserver infrastructure as-is.
> There are too many idiosyncratic aspects of its configuration; 

> it has its own wiki,

Since when is that a problem? To take an obvious example, labs also
began by making its own wiki, instead of incorporating itself into an
existing one.


> its own (closed source) version control system, 
The vcs isn't closed source, it's just plain subversion.

The *repository viewer* is closed source, although with a an Open Source
license. I don't know if people really use it too much.


> its own (closed source) issue tracker. 
I'm not a fan of jira, but this is the silliest reason to drop the
toolserver :)


> There are hacks like TUSC that we want
> to replace with better systems/services (e.g. OpenID/OAuth).

OpenID nor OAuth are currently available. If they were, TUSC wouldn't
have been developed in the first time. When they appear, they will
-slowly at first- take over TUSC, as it should.
This ball is at WMF side.

Not to mention, if these «idiosyncratic aspects» were really a problem,
they could easily be changed.


> Chapters are autonomous organizations, and it's WM-DE's call how much
> / whether it wants to continue to invest in infrastructure of any kind
> (and the decision of funding bodies like the FDC to accept or reject
> that proposition). However, for our part, we will not continue to
> support the current arrangement (DB replication, hosting in our
> data-center, etc.) indefinitely.

This sounds as forcing them to go this route.


> The timeline we've discussed with Wikimedia Germany is roughly as follows:
> 
> - Wind down new account creation on toolserver by Q2 of 2013 calendar year
> - Decommission toolserver by December 2013
> 
> WMF can't commit to providing technical support for tool transition
> (there are too many tools), so if there's any area where I think it
> would make sense to ramp up investments on WM-DE's part, it's in
> engineering capacity to support tool developers in porting tools to
> Labs.

So the tool authors are "on their own", while forced to move out.
What is the WMF *offering*?


> That said, there may be a need for emergency purchases/investments to
> keep TS in a usable state until December 2013 (and perhaps allow for
> some buffer room beyond that). That's not our call to make.

But you are refraining WMDE from investing to it now. Would for instance
WMF be willing to lend a few servers to the toolserver until December 2013?


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 25/09/12 00:51, DaB. wrote:
> Hello all,
> 
> in these days WMDE (the chapter that finance the toolserver) is discussing 
> the 
> budget for the next year (2013); you can find it at [1]. At the moment there 
> is 
> no money for new toolserver-hardware in this budget and the CEO Pavel Richter 
> is unwilling to change this ([2] in german) – because he fears that there 
> will 
> be a Wikilabs in 2014. It is not possible for me to run the toolserver for 
> another year with the current hardware – you all know why. For this reason I 
> will request a change of the budget at the general meeting at November, so 
> there will be a vote about. If this vote should fail (and we get no money for 
> new hardware), I am going to retire from my job as root at 30. December 2012.
> I'm not longer able to tolerant the behavior of the german chapter and the 
> WMF 
> in matter of the toolserver; I do this for free and for fun, and it is not 
> longer fun.
> 
> Sincerely,
> DaB.


So, out of fear that there may be a better alternative in two years
time, he decides to stop supporting it now?

I know labs, and I'm not sure it will really be a better alternative.
Currently, it isn't. The toolserver is much more reliable and flexible.
I don't think labs will "win" in the near future, either. Although it
might change in two years. Specially if no attention is payed to the
toolserver for that time.

I see two big problems:
1) If labs really becomes the "perfect tool hosting" in 2014, What
happens before we reach that? "Yes, your tool doesn't work, migrate to
labs next year"

2) We risk ending up with no good alternative at that time. labs is
still not good enough, toolserver has degraded so much it's unusable.



Not to mention the migration cost that would need to be payed by tool
authors if forced to move. Although perhaps he doesn't care.






> P.S: If you are in a board of a chapter that gives money to WMDE for the 
> toolserver: Make sure that it will be spend for hardware. 
> 
> [1] 
> http://meta.wikimedia.org/wiki/Wikimedia_Deutschland/2013_annual_plan_draft/en
> [2] 
> meta.wikimedia.org/wiki/Talk:Wikimedia_Deutschland/2013_annual_plan_draft/de#Toolserver
> 
> 
> 
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Very slow connections to the sql servers

[Platonides]

I am seeing my tools very slow connecting to the sql servers, such as
taking 33 or 79 seconds
just for figuring out the right server and connecting to it, and even
failing with «Can't connect to
MySQL server on 'sql-s4-rr.toolserver.org'» (not only on this server)

Are sql servers overloaded? Is it a problem with willow? Should we give
a kick to the tcp stack?

This anomaly is also visible from the command line, varying from immediate
connection to slow one:

$ time mysql -h sql-s4-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m0.098s
user0m0.003s
sys 0m0.009s

# Shouldn't it always so quick?

$ time mysql -h sql-s4-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m4.803s
user0m0.003s
sys 0m0.009s

$ time mysql -h sql-s4-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m1.619s
user0m0.003s
sys 0m0.009s

$ time mysql -h sql-s4-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m4.222s
user0m0.003s
sys 0m0.009s

$ time mysql -h sql-s4-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m15.434s
user0m0.003s
sys 0m0.008s

 $ time mysql -h sql-s3-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m13.463s
user0m0.003s
sys 0m0.009s

$ time mysql -h sql-s3-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m2.356s
user0m0.003s
sys 0m0.009s

$ time mysql -h sql-s3-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m1.850s
user0m0.003s
sys 0m0.008s

$ time mysql -h sql-s3-rr.toolserver.org  -e 'SELECT VERSION()'
+---+
| VERSION() |
+---+
| 5.1.59|
+---+

real0m24.724s
user0m0.003s
sys 0m0.008s


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] qcronsub warning: "Please add maximum runtime"

[Platonides]

On 24/09/12 18:07, Krinkle wrote:
> Can someone decode this? What is this?
> 
> -- Krinkle
> 
> 
> Begin forwarded message:
> 
>> *From: *r...@toolserver.org  (Cron Daemon)
>> *Subject: **Cron  qcronsub -N dbbot_wm -m n -j y -b
>> y -l h_rt=INFINITY -l virtual_free=90M "$HOME/bots/dbbot-wm-start.sh"*
>> *Date: *September 24, 2012 6:05:07 PM GMT+02:00
>> *To: *krin...@toolserver.org 
>>
>> warning: Please add maximum runtime by adding parameter -l
>> arch=sol|lx

The text asks you to place a time limit. The parameter (embedded in
posix colors despite not being output to a terminal) to specify if it
needs a linux or solaris server.

However, if I try to execute it, I get a much saner message:
$ qcronsub -N dbbot_wm -m n -j y -b y -l h_rt=INFINITY -l
virtual_free=90M "/home/krinkle/bots/dbbot-wm-start.sh"
> Unable to run job: Script not executable: 
> /home/krinkle/bots/dbbot-wm-start.sh.
> Exiting.
> warning: Please add the os this job can run on by adding parameter -l 
> arch='*'|sol|lx
> For more information read documentation at 
> https://wiki.toolserver.org/view/Job_scheduling

As this is a php script, your parameter would be «-l arch='*'»

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Web servers unresponsive

[Platonides]

On 24/09/12 15:03, emijrp wrote:
> Can you kill them?
> 
> I always connect to login.toolserver.org <http://login.toolserver.org>.
> How do I change to rosemary?

rosemary is a sql server.

You'd do something like this:

> $ mysql -h rosemary
>
> mysql> show full processlist;
> +---++-+--+-+--+---+---+
> | Id| User   | Host| db   | Command | 
> Time | State | Info  |
> +---++-+--+-+--+---+---+
> | 142886093 | platonides | willow.toolserver.org:62886 | NULL | Query   |
> 0 | NULL  | show full processlist |
> +---++-+--+-+--+---+---+
> 1 row in set (0.00 sec)
>
> mysql> kill 142886093;

Of course, you would have some problems doing it when you can't even
connect to the server :)

I had the same problem on Saturday with s2-rr, even after killing all
php.fcgi instances on the webservers (the ones from which those 15
connections could have originated), the server wasn't allowing me to
connect.


While we're blaming the sql servers, I had a script failing this night
with «Lost connection to MySQL server at 'reading initial communication
packet', system error: 0 (sql-s3-rr.toolserver.org)» but it is working
to now.


Regards

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Web servers unresponsive

[Platonides]

On 23/09/12 21:07, DaB. wrote:
> Hello,

>>> All error out on with no response and a time out.
>>>
>>> I can still SSH into wolfsbane and ortelius from willow, though.
>>
>> I will now investigate this. Until now the only problem I found is that
>> hemlock is down.
> 
> I restored the web-access now. As far as I see hemlock lost its external 
> array 
> and became out of memory around 2:30 UTC. I have no idea why this influence 
> our 
> webserver. I rebooted hemlock to free the memory and restarted the webserver 
> on ortelius and wolfsbane; the webpages are back AFAIS.

Maybe they got blocked trying to access something at /mnt/user-store ?
Following a symlink, perhaps.
The last successful answer by both webservers was at 23/Sep/2012:02:35
before the restart.

This NFS failure was quite bad since there was no timeout when trying to
access /mnt/user-store, just being blocked forever.
I guess TS-1519 and TS-1520 may be closed now.

Thanks


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] z-dat-s4-a (s4-user) is down (was: Re: Reboot

[Platonides]

On 20/09/12 13:12, DaB. wrote:
> Hello all,
> At Thursday 20 September 2012 13:10:18 DaB. wrote:
>> I'm noticing that some data has gone missing from s4 database(s). It
>> appears that this goes back at least a few days; do we know how far,
>> and if any of this data is recoverable? (Of less immediate concern is
>> why this happened...)
> 
> I told Nosy to use our backup from the 18 (that's the day the mistake 
> happened 
> and so only few data should be lost). But only she knows that she has done.
> 
> @Nosy: Can you please respond to Hersfold?
> 
> Sincerely,
> DaB.

According to my logs, the backup was done shortly after 2012-09-17T03:36

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] When to execute cron-tasks

[Platonides]

On 14/09/12 01:42, Dr. Trigon wrote:
> What about a tool that gather some statistics from all users
> cron(ie)tab files and show them on a public (or for logged-in users
> only) page/place.
> 
> The idea is that user can get an clue on what times have high or
> low job loads.
> 
> Something like:
> 
> 00 - used xx times - xx% of all jobs 01 - used xx times - xx% of
> all jobs 02 - ... 03
> 
> Might also include hours and more info, BUT NO per user data just 
> overall averages and counts.
> 
> Greetings DrTrigon

Ideally, you could mark a task as being daily-I-don't-care-when, or
perhaps "run each 20-28h", and cron would choose the time that best
suited itself, taking all registered jobs into acocunt.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] sql.toolserver.org will become deprecated

[Platonides]

What about adding user entries to the "switchserver table"?
Currently, if you want to access an arbitrary wiki table, you need to
look it up first in toolserver.wiki. Seems appropiate to extend that to
user databases. Frameworks setting db connections would need little
changes to also handle user dbs like they do with wmf views. (I'd
actually put it into a new table eg. toolserver.databases, but changing
the db name used in the framework at the same time -keeping wiki for
other uses, of course- should be trivial)

Regards



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Why does Perl use so much more vmem on Linux than on Solaris?

[Platonides]

To begin with, the perl used in Solaris is 32 bit and the Linux one 64
bit. They are also different versions (v5.12.3 vs v5.10.1).

But the main difference is probably in the linux version mapping
/usr/lib/locale/locale-archive, a 86M file, while in Solaris it seems to
be using /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3,
/usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3 and
/usr/lib/locale/en_US.UTF-8/libc.so.1 instead.

>From locale-gen(8):
> Locale data files can be  stored  either  in  a  single  archive  file,
> /usr/lib/locale/locale-archive,  or  in  a  deep  tree where individual
> files are stored under /usr/lib/locale//LC_*.

As all processes will be sharing the same locale-archive file, the two
methods shouldn't matter too much for memory usage, although
locale-archive will probably bite us when dumping core.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Web services are down

[Platonides]

On 02/09/12 17:37, Magnus Manske wrote:
> Me too. Occasionally, it's "502 bad gateway" though. Just in the
> middle of testing stuff and thought it was my broken code.
> Frustration.

wolfsbane has a near-full /tmp
Use http://ortelius.toolserver.org/ until Dschwen or a root gets to it.

Jira bug https://jira.toolserver.org/browse/TS-1504

Maybe we should use a LRU filesystem on the webservers /tmp ?

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Compiling an apache module on solaris, with or without stack-protection

[Platonides]

On 02/09/12 09:04, Kai Krueger wrote:
> Hello everyone,
> 
> I would like to try and update an apache module (mod_tile for serving
> map tiles [1]) on Ptolemy, which still operates on Solaris. After fixing
> a bunch of standard portability issues with the source code, I have run
> into a problem that I am not sure about.
> 
> With the standard compile options, the build fails with:
> Undefined   first referenced
>  symbol in file
> __stack_chk_fail_local  ./.libs/dir_utils.o  (symbol scope
> specifies local binding)
> ld: fatal: Symbol referencing errors. No output written to
> ./.libs/mod_tile.so
> 
> The suggested fix for this issue seems to be to add -fno-stack-protector
> to the CFLAGS. After figuring out how to pass that to the apache build
> system it does now seem to compile fine. (if one tries to do that during
> the ./configure stage, ./configure fails by complaining that the
> C-compiler can't produce an executable. Possibly because ./configure
> uses the sun compiler rather than gcc which doesn't support
> -fno-stack-protector?)
> 
> However, the apache build system (apxs) claims that apache on ptolemy
> was built with stack protection. Does it matter if a module is built
> without stack protection while apache is? Is there a way to test out if
> the built module works correctly with the installed apache? Is there a
> better way to fix this compile issue?
> 
> Currently I just compiled it from source in the project home directory.
> Would it have to be "ported" to ts-spec to install it on ptolemy?
> 
> Thanks,
> 
> Kai
> 
> [1] http://svn.openstreetmap.org/applications/utils/mod_tile/

I'd clean all the .o files and rebuild. It's possible that you have some
files built with gcc and others with cc, giving weird errors.

You should use the same compiler as the one used for apache (it may work
with another, but it's better to play safe).
If you run CC= ./configure, it should pick the one you wanted.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Web services are down

[Platonides]

On 29/08/12 09:22, Marlen Caemmerer wrote:
> Hello,
> 
> wolfsbane ran out of space which means one of the web servers was not
> able to deliver any more web pages which has the effect of "sometimes
> the page works - sometimes not".
> Should be fixed now - if you still see errors please let us know.
> 
> Cheers
> nosy

Thanks nosy.

Shouldn't the balancer have automatically considered wolfsbane down and
direct all requests to ortelius?

Cheers

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] sql-s1 will be read-only on Wednesday

[Platonides]

On Thu, Aug 16, 2012 at 1:33 PM, DaB.  wrote:
> Hello,
> At Thursday 16 August 2012 13:31:27 DaB. wrote:
>> Is it possible that such db belongs to anyone else and got missasigned in
>> some db move?
>>
>> You don't need to import that, DaB.
>
> I can not guarantee that there wasn't a mistake somewhen in the past, but I
> doubt it. But I would say we solve it in the practical way: Just drop what you
> don't need :-) (if somebody would miss it, he/she would have already tell us,
> wouldn't he/she?)

Maybe I created them for some temporary thing, then forget it. Part of
the reason
of this mail was to see if someone came up saying "those are my tables!"

Tables dropped from rosemary.

Regards

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] sql-s1 will be read-only on Wednesday

[Platonides]

On Wed, Aug 15, 2012 at 5:44 PM, DaB.  wrote:
> 68616734 u_platonides.dump.gz

I seem to have a db in thyme listing the hashes of enwiki images
and their size, but I don't remember setting up that thing.

Is it possible that such db belongs to anyone else and got missasigned in
some db move?

You don't need to import that, DaB.

Regards

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] s1 replag update, suggestion, and question

[Platonides]

On Wed, Aug 8, 2012 at 3:46 PM, Russell Blau  wrote:
> There is a possible workaround.  The TS could treat this like a server
> outage; copy user databases from thyme to rosemary and then point
> sql-s1-user to rosemary, which currently has no replag. Rosemary would
> then have to handle twice the load, but thyme should start to recover
> very quickly with no user-generated queries hitting it. Once thyme has
> recovered, point sql-s1-rr to it.
>
> Downsides: (1) this would require several hours of downtime for
> sql-s1-user while the user databases are copied; all tools that require
> access to user databases would be offline entirely for this period. (2)
> it would have to wait until our volunteer TS admins have time to do it.
Actually, it could probably be reduced from "downtime" to "readonly
user databases". If thyme were writing at the binlog, it could
probably stay accepting writes for the most part of it. This comes at
the expense of TS admin time, of course.

> (3) the added load on rosemary could cause replag to grow there,
> although I doubt it would come anywhere near the 14+ days replag we are
> dealing with now on thyme.

Depending on the insert speed without queries, another option would be
the time needed for copying the db from rosemary to thyme.
(I'm assuming it would be much slower than the downtime moving user
dbs but it's just a guess, if it weren't this could replace that
move).

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette