Re: [Toolserver-l] Future of the toolserver

[Federico Leva (Nemo)]

> Yes, I know about MMP. It was introduced much later, after people were
> already very used to doing things in an individualized manner. From
> what I'm told MMPs aren't used much.

So you think that even new users (I believe many users arrived after MMP 
were introduced) are not using MMP due to bad old habits of the other users?
I've had a hard time trying to convince some users to use a MMP; I think 
it's not trivial to understand what models actually work and are liked.


> Toolserver's model is fundamentally different. It's based on an old
> concept of shared hosting. Labs is built on a model more like a VPS
> (really more like EC2). Due to that, it's possible to give users far
> more rights. MMPs aren't a very elegant solution. I'd prefer not to
> force an inelegant solution into a system that allows much more
> elegant approaches.

I'm not a technical person but yes, I can probably understand that MMP 
is not very elegant. But, do you think users are not using MMP because 
they're not elegant?


Nemo

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ryan Lane]

On Wed, Sep 26, 2012 at 1:37 AM, Federico Leva (Nemo)
 wrote:
>> Toolserver's way of doing things is very individualized. It doesn't
>> promote collaboration, and it leads to tools disappearing when a
>> user's account is disabled.
>
> Wrong. Do you know MMP https://wiki.toolserver.org/view/MMP ?
> Toolserver already allows collaboration as you describe it. What you seem to
> be saying is that you want to *force* people to collaborate: are you sure
> this works?
> I'd rather suggest to investigate what works or not in the MMP (very simple)
> model and why people don't use it more in Toolserver, so that you can build
> on that experience and understand better the users' needs.
>

Yes, I know about MMP. It was introduced much later, after people were
already very used to doing things in an individualized manner. From
what I'm told MMPs aren't used much.

Toolserver's model is fundamentally different. It's based on an old
concept of shared hosting. Labs is built on a model more like a VPS
(really more like EC2). Due to that, it's possible to give users far
more rights. MMPs aren't a very elegant solution. I'd prefer not to
force an inelegant solution into a system that allows much more
elegant approaches.

Of course, as I mentioned before, there's nothing forcing this new
model at all. If you guys want to build the exact same Toolserver
environment as a Labs project, go for it. I have a good feeling you'll
start doing things differently when you see the affordances given by
having more rights, though. Either path chosen, we'll help you with
infrastructure issues along the way.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Federico Leva (Nemo)]

> Toolserver's way of doing things is very individualized. It doesn't
> promote collaboration, and it leads to tools disappearing when a
> user's account is disabled.

Wrong. Do you know MMP https://wiki.toolserver.org/view/MMP ?
Toolserver already allows collaboration as you describe it. What you 
seem to be saying is that you want to *force* people to collaborate: are 
you sure this works?
I'd rather suggest to investigate what works or not in the MMP (very 
simple) model and why people don't use it more in Toolserver, so that 
you can build on that experience and understand better the users' needs.


Nemo

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Krinkle]

On Sep 25, 2012, at 11:12 PM, "DaB."  wrote:
> Hello Erik,
> At Tuesday 25 September 2012 22:24:33 DaB. wrote:
>> 
>> The initial focus for Labs has been to provide functionality that
>> toolserver doesn't - get root on a VM or set of VMs to install/test
>> arbitrary software/services, and get it ready for production
>> deployment.
>> 
> 
> It is nice to have root on a (virtual) machine, but I doubt most tools need it
> [..]
> 


Indeed, however the reason this is crucial for labs is because its scope is much
wider than Toolserver.

For example, in the "deployment" project we simulate nearly the entire WMF
production cluster (including db hosts, apaches, squids, varnish, scalers,
etc.).

This makes one of the very different goals of Labs possible, namely to allow
volunteers to contribute to operations (as opposed to the software we run).

Once everything is puppetized one can basically create a new labs project,
use "wmf-production" as template and instantiate a complete wmf cluster (not
with all the database contents, just the server setup, though it'd contain
sufficient sample data, the purpose is to simulate the servers to develop new
configurations, not use as web site). Give it a subdomain and you'd immediately
have stuff like commons.wikimedia.myproject.wmflabs.org.

Back to the subject, does that mean users will have to learn to manage a VM and
require a public IP and subdomain? No, not at all. We're confusing Dev Labs with
Tool Labs (perhaps we shouldn't name them like that as isolated projects).

Implementation of Tool Labs isn't decided on afaik, but I believe it will 
naturally
solve itself by being distributed among various projects. Behind the scenes they
will likely be a regular labs project, but abstracted for users (e.g. not an
instance-group or even an instance per tool, but all in one instance-group, with
a group of servers for different purposes, like Toolserver has web servers, sql
servers, login/application servers).

E.g. the tools project in wmflabs would have various web servers and application
servers[1]. Users wanting to run queries, bots and long-running/periodic 
processes
would use the application servers. Ideally we'd encourage use of SGE (or
something alike) from the beginning so that the application servers are
optimally used, and it would make it easy to start a process in the background
of an application server from a process on the web server

Access to the wmf wiki replicated dbs is public across the entire wmflabs
network so that's a given within the toollabs project as well.

-- Krinkle

[1] The "bots" project exists already. It doesn't have SGE yet but it's a first
step. There is also a generic "webtools" project being set up as we speak.
Perhaps these two could be merged so that users have shared project storage for
bots generating data to be used by bots and vice-versa.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ryan Lane]

On Tue, Sep 25, 2012 at 9:41 PM, Hersfold  wrote:
> I think the vast majority of the people on the Toolserver don't need a lot
> of that extra "beefy" stuff. To run our tools, all we need is:
>
> * A database providing replication from the Foundation's databases.
> * A web server capable of running PHP, CGI, and other web scripts.
> * A linux or solaris machine that we can run bots on.
> * A platform on which all of this runs reliably.
>

Excluding the database replication, all of these things are possible now.

> If any of us needed a Mediawiki installation, we'd have set up our own site
> for it years ago. Sure, I can see you wanting to make options like that
> available to developers, particularly those building extensions to MW, but
> what the Toolserver community - and really, all of the Wikimedia projects -
> need right now is something that meets those four bullet points. With the
> Toolserver down as often as it's up lately, and Labs not ready to provide
> any of that support until December 2013 (not to mention no assistance moving
> there), either a) the Toolserver needs funding so it can get back to working
> order, even if it means no further accounts or projects can be created on
> it, or b) Labs needs to get that stuff running much sooner.
>

Database replication should be done in the next 2-3 months. It may be
done sooner, but that's our current goal date. Brave community members
can start building any other missing infrastructure.

> And frankly, I don't see why WMDE should be footing the bill for all the
> transitioning work when the Foundation isn't providing them any support to
> do so while setting up a competing system at the same time.
>

I've never built Labs to be a competing system. Labs should have
similar functionality, but I find the toolserver portions of Labs to
be a small part of its function. My biggest goal with integrating
Toolserver features is to bring our Toolserver community more closely
together with our developer community. They are fairly disjoint right
now.

WMDE offered to help with the transition work. The decision to move
the community from Toolserver to Labs was made jointly between WMDE
and WMF. It's not possible for WMF to handle building the Labs
infrastructure and to transition the tools. We don't have enough
Toolserver experience to do so properly anyway.

Toolserver was mostly built by the community from the start. Even WMDE
working the transition won't be enough. We need people like you to
help transition. My team will help you as much as needed, and will
gladly answer any questions you have.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Chris Grant]

We're forgetting something very important here.

Firstly, I would say that we need the toolserver running, for at least 2
more years. My guess is, it will take most of next year, to get labs to a
state where it is a viable replacement that people *want* to switch to;
then the next year as people slowly move over.

However, what we should really be concerned about, is not the hardware, or
how labs is different to the toolserver. It is DaB. If we screw up with
hardware, labs, etc, we can start over, buy new stuff, whatever. DaB we
cannot replace. And if the toolserver needs to keep running for the next
year or two (which it does), then we certainly need DaB. And quite frankly,
that is the real issue at heart here.


-- Chris
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Hersfold]

I think the vast majority of the people on the Toolserver don't need a 
lot of that extra "beefy" stuff. To run our tools, all we need is:


* A database providing replication from the Foundation's databases.
* A web server capable of running PHP, CGI, and other web scripts.
* A linux or solaris machine that we can run bots on.
* A platform on which all of this runs reliably.

If any of us needed a Mediawiki installation, we'd have set up our own 
site for it years ago. Sure, I can see you wanting to make options like 
that available to developers, particularly those building extensions to 
MW, but what the Toolserver community - and really, all of the Wikimedia 
projects - need right now is something that meets those four bullet 
points. With the Toolserver down as often as it's up lately, and Labs 
not ready to provide any of that support until December 2013 (not to 
mention no assistance moving there), either a) the Toolserver needs 
funding so it can get back to working order, even if it means no further 
accounts or projects can be created on it, or b) Labs needs to get that 
stuff running much sooner.


And frankly, I don't see why WMDE should be footing the bill for all the 
transitioning work when the Foundation isn't providing them any support 
to do so while setting up a competing system at the same time.



User:Hersfold
hersfoldw...@gmail.com

On 9/25/2012 9:30 PM, Erik Moeller wrote:

On Tue, Sep 25, 2012 at 2:27 PM, Platonides  wrote:

Hi Platonides.


labs is also a second class citizen.

Moreover, it is explicitely stated not to be for production-like level.

What will happen if a really successful tool reaches to a point where it
de-facto needs it ? (eg. a WLM tool, tools to Request to get an Account
Created, or Request an Unblock appeal...)

Labs doesn't take options away in those cases -- it's just designed to
make it a lot more straightforward to take a tool and integrate it
well, if that's the best path to take for a particular tool.

It opens up additional possibilities:

1) It makes it easy to set up a MediaWiki instance (we're working on a
standard VM config for this to make it really straightforward) so you
can prototype an extension, if that's a direction you want to take for
a particular tool. E.g. some of the above sound like they should be
ideally done as special pages.

I buy into some of the premise stated here (that a lot of tool
developers really don't _want_ to do MediaWiki development), but I
also think it's our job to make it as easy as possible to take
software down the path that makes the most sense.  In contrast, large
web apps like MediaWiki are explicitly prohibited from being made
publicly available per https://wiki.toolserver.org/view/Rules

2) It makes it possible to prototype a standalone service, puppetize
it, and prepare it for production deployment on allocated hardware.

For tools that aren't mission-critical and that are only lightly
integrated, I don't view it as an issue for a tool to run at
tools.wmflabs.org/some/cool/tool (or wherever it's going to live)
indefinitely. (Ryan may want to chip in on that.)


So the tool authors are "on their own", while forced to move out.
What is the WMF *offering*?

We're offering free access to very beefy infrastructure for
Wikimedia-relevant purposes. We're in this for the long haul and
intend to develop Labs into the best environment for testing/staging,
bottom-up experimentation and tool development that we can possibly
build.

We're not able to provide hands-on support for porting stuff over, but
as Ryan suggested, it may be feasible for interested folks to build an
environment in Labs that makes transition easier for toolserver users.
The Labs team (which includes volunteers) is generally very responsive
and helpful, within reason.


Would for instance WMF be willing to lend a few servers to the toolserver until
December 2013?

WMDE has to decide what level of transitional support is appropriate
and budget for it. As for support, we've kicked around the possibility
of them purchasing some additional hardware and us buying it from them
later and using it for other purposes, but we'd have to carefully work
out whether that's feasible in practice.

Erik



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Erik Moeller]

On Tue, Sep 25, 2012 at 2:27 PM, Platonides  wrote:

Hi Platonides.

> labs is also a second class citizen.
>
> Moreover, it is explicitely stated not to be for production-like level.
>
> What will happen if a really successful tool reaches to a point where it
> de-facto needs it ? (eg. a WLM tool, tools to Request to get an Account
> Created, or Request an Unblock appeal...)

Labs doesn't take options away in those cases -- it's just designed to
make it a lot more straightforward to take a tool and integrate it
well, if that's the best path to take for a particular tool.

It opens up additional possibilities:

1) It makes it easy to set up a MediaWiki instance (we're working on a
standard VM config for this to make it really straightforward) so you
can prototype an extension, if that's a direction you want to take for
a particular tool. E.g. some of the above sound like they should be
ideally done as special pages.

I buy into some of the premise stated here (that a lot of tool
developers really don't _want_ to do MediaWiki development), but I
also think it's our job to make it as easy as possible to take
software down the path that makes the most sense.  In contrast, large
web apps like MediaWiki are explicitly prohibited from being made
publicly available per https://wiki.toolserver.org/view/Rules

2) It makes it possible to prototype a standalone service, puppetize
it, and prepare it for production deployment on allocated hardware.

For tools that aren't mission-critical and that are only lightly
integrated, I don't view it as an issue for a tool to run at
tools.wmflabs.org/some/cool/tool (or wherever it's going to live)
indefinitely. (Ryan may want to chip in on that.)

> So the tool authors are "on their own", while forced to move out.
> What is the WMF *offering*?

We're offering free access to very beefy infrastructure for
Wikimedia-relevant purposes. We're in this for the long haul and
intend to develop Labs into the best environment for testing/staging,
bottom-up experimentation and tool development that we can possibly
build.

We're not able to provide hands-on support for porting stuff over, but
as Ryan suggested, it may be feasible for interested folks to build an
environment in Labs that makes transition easier for toolserver users.
The Labs team (which includes volunteers) is generally very responsive
and helpful, within reason.

> Would for instance WMF be willing to lend a few servers to the toolserver 
> until
> December 2013?

WMDE has to decide what level of transitional support is appropriate
and budget for it. As for support, we've kicked around the possibility
of them purchasing some additional hardware and us buying it from them
later and using it for other purposes, but we'd have to carefully work
out whether that's feasible in practice.

Erik
-- 
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ryan Lane]

> I doubt that you understand the toolserver. The toolserver-CLUSTER is make of
> 18 different servers. Some run Solaris, some run Debian, some are (big)
> database-servers, some are userland-servers, some are web-servers, some are
> HA-nodes and some are aux-servers (forgetting such details like virtual
> machines, network-devices and external storage). It's a grown infrastructure
> and you can not just move it to labs or rebuild it there as a simple virtual
> machine or cloud-instance. Even if I would begin this very instance I doubt I
> would be finish in December 2013; and it would need the same 18 servers (or
> better: more).
>

The database servers (replicated and user-databases) are going to be
provided by the Labs infrastructure. That's the majority of
Toolserver's servers. We have shared storage provided already.

The capacity of Labs is far greater than Toolserver's. We have a
cluster in the pmtpa and eqiad datacenters. Each cluster has 7 compute
nodes for virtual machines, 2 database servers for replicas, 1
database server for user databases and 4 storage nodes for shared
storage access.

In total that's:

* 14 compute nodes with a total of 2.5TB of RAM, 224 CPU cores, and
14TB of storage for virtual machine images
* 4 database nodes for replicas with a total of 740GB of RAM and 64 CPU cores
* 2 database nodes for user-databases, with a total of 370GB of RAM
and 32 CPU cores
* 8 storage nodes with a total of 128TB of storage

In a project you can create as many instances as you'd like. The
default quota limit per-project is 10 instances, but we can increase
that number as much as needed.

It's definitely possible to re-create Toolserver inside of Labs. I'm
not suggesting that's what should be done for a migration, but it's an
option.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[DaB.]

Hello,
At Wednesday 26 September 2012 00:40:24 DaB. wrote:
> +1. If people really love the Toolserver way of doing things, they
> are more than welcome to re-create the environment as a Labs project.

I doubt that you understand the toolserver. The toolserver-CLUSTER is make of 
18 different servers. Some run Solaris, some run Debian, some are (big) 
database-servers, some are userland-servers, some are web-servers, some are 
HA-nodes and some are aux-servers (forgetting such details like virtual 
machines, network-devices and external storage). It's a grown infrastructure 
and you can not just move it to labs or rebuild it there as a simple virtual 
machine or cloud-instance. Even if I would begin this very instance I doubt I 
would be finish in December 2013; and it would need the same 18 servers (or 
better: more).

Sincerely,
DaB.

P.S: There is no OSM in the 18 servers. OSM is again completely difference 
(Postgres vs. mysql for example).

-- 
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885


signature.asc
Description: This is a digitally signed message part.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ryan Lane]

> Both approaches have its merits. The WMF could have copied the
> toolserver recipe much more easily than gooing the route of labs. They
> were brave pursuing the VM based system, and it allows solving different
> problems.
> But both of them have their use. It's not a case where one should "win"
> the other, or the toolserver is "bad" because it is currently hosted by
> WMDE instead of WMF.
>
> Organizations trying to defeat one another will end up with one big
> loser: the community.
>

+1. If people *really* love the Toolserver way of doing things, they
are more than welcome to re-create the environment as a Labs project.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ryan Lane]

On Tue, Sep 25, 2012 at 5:38 PM, Federico Leva (Nemo)
 wrote:
> Indeed the first answers on
> 
> seem to indicate that all tools and services currently on Toolserver will
> just be trashed and people forced to reinvent them from scratch because a
> completely different approach is thought to be better.
>

Toolserver's way of doing things is very individualized. It doesn't
promote collaboration, and it leads to tools disappearing when a
user's account is disabled.

The model in Labs is fundamentally different. Tools, bots, etc. are
meant to be managed by a community, rather than an individual. That
isn't to say that tools and bots can't be created and managed in
exactly the same way as it is in Toolserver, just that it's highly
discouraged.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 25/09/12 23:38, Federico Leva (Nemo) wrote:
> Indeed the first answers on
> 
> seem to indicate that all tools and services currently on Toolserver
> will just be trashed and people forced to reinvent them from scratch
> because a completely different approach is thought to be better.
> 
> Nemo

Both approaches have its merits. The WMF could have copied the
toolserver recipe much more easily than gooing the route of labs. They
were brave pursuing the VM based system, and it allows solving different
problems.
But both of them have their use. It's not a case where one should "win"
the other, or the toolserver is "bad" because it is currently hosted by
WMDE instead of WMF.

Organizations trying to defeat one another will end up with one big
loser: the community.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Federico Leva (Nemo)]

Indeed the first answers on 
 
seem to indicate that all tools and services currently on Toolserver 
will just be trashed and people forced to reinvent them from scratch 
because a completely different approach is thought to be better.


Nemo

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 25/09/12 20:48, Erik Moeller wrote:
> 1) WMF is a technology organization. Hosting the core infrastructure
> for Wikimedia projects is very much what we do. This includes data
> center operation, monitoring and backups, software deployments,
> software/service upgrades, code versioning infrastructure, bug
> tracking infrastructure, additional support systems and services (like
> this mailing list), etc.
> 
> Toolserver is in fact hosted by the Wikimedia Foundation today, in our
> Amsterdam data-center. We provide space, power and racks for the
> toolserver cluster, at a cost of about $65,000/year to WMF according
> to our Director of TechOps.

Something we should all be grateful for.


> We also maintain the database replication
> on our end which enables tools to function.

A replication which WMF is already doing for itself, adding an extra
slave hosted by a trusted party doesn't make a techical difference.
While WMF gains a backup server (at some point the toolserver was the
only externally hosted backup). It has also inadvertedly dropped the
binlogs needed by TS when doing server cleanup.

So while very welcome, the WMF side of the db replication is not the
most important piece of the toolserver (having a db copy still makes the
TS better than any other alternative available at the time, though).
The value is probably in how WMDE was able to open that database and
many fruitful tools emerged.



> We can't provide the same level of service for the toolserver
> infrastructure as we do for core operations, and it makes no sense for
> a chapter to build out the required staffing and expertise to do so
> (set up/maintain all or some of the aforementioned functions). Even
> with slightly increased investment, toolserver would always suffer
> from being second or third tier infrastructure.

labs is also a second class citizen.


Moreover, it is explicitely stated not to be for production-like level.

What will happen if a really successful tool reaches to a point where it
de-facto needs it ? (eg. a WLM tool, tools to Request to get an Account
Created, or Request an Unblock appeal...)



> 2) We're not comfortable hosting the toolserver infrastructure as-is.
> There are too many idiosyncratic aspects of its configuration; 

> it has its own wiki,

Since when is that a problem? To take an obvious example, labs also
began by making its own wiki, instead of incorporating itself into an
existing one.


> its own (closed source) version control system, 
The vcs isn't closed source, it's just plain subversion.

The *repository viewer* is closed source, although with a an Open Source
license. I don't know if people really use it too much.


> its own (closed source) issue tracker. 
I'm not a fan of jira, but this is the silliest reason to drop the
toolserver :)


> There are hacks like TUSC that we want
> to replace with better systems/services (e.g. OpenID/OAuth).

OpenID nor OAuth are currently available. If they were, TUSC wouldn't
have been developed in the first time. When they appear, they will
-slowly at first- take over TUSC, as it should.
This ball is at WMF side.

Not to mention, if these «idiosyncratic aspects» were really a problem,
they could easily be changed.


> Chapters are autonomous organizations, and it's WM-DE's call how much
> / whether it wants to continue to invest in infrastructure of any kind
> (and the decision of funding bodies like the FDC to accept or reject
> that proposition). However, for our part, we will not continue to
> support the current arrangement (DB replication, hosting in our
> data-center, etc.) indefinitely.

This sounds as forcing them to go this route.


> The timeline we've discussed with Wikimedia Germany is roughly as follows:
> 
> - Wind down new account creation on toolserver by Q2 of 2013 calendar year
> - Decommission toolserver by December 2013
> 
> WMF can't commit to providing technical support for tool transition
> (there are too many tools), so if there's any area where I think it
> would make sense to ramp up investments on WM-DE's part, it's in
> engineering capacity to support tool developers in porting tools to
> Labs.

So the tool authors are "on their own", while forced to move out.
What is the WMF *offering*?


> That said, there may be a need for emergency purchases/investments to
> keep TS in a usable state until December 2013 (and perhaps allow for
> some buffer room beyond that). That's not our call to make.

But you are refraining WMDE from investing to it now. Would for instance
WMF be willing to lend a few servers to the toolserver until December 2013?


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[DaB.]

Hello Erik,
At Tuesday 25 September 2012 22:24:33 DaB. wrote:
> Hi all,
> 
> let me weigh in with a few initial comments, and I'll ask the Labs
> folks to participate here and on Meta as well with regard to technical
> questions.
> 
> The initial focus for Labs has been to provide functionality that
> toolserver doesn't - get root on a VM or set of VMs to install/test
> arbitrary software/services, and get it ready for production
> deployment.

It is nice to have root on a (virtual) machine, but I doubt most tools need 
it. I would also bet that most tool-authors should not have root-access and 
should not be able to install software.
I guess what most people and the WMF and also WMDE does not understand is: 
Most tool-authors are no system-ops or have a Master of Informatic Science. 
Most are amateur programmers who have fun to code a tool and do not maintain 
it once it is done (they also not document it). The tools they are write a 
slow and need way too much resources. They have no backup of their ssh-keys 
and do not extend their accounts in time. And all this is ok. Because (Nosy 
and) I am there to give them a infrastructure they can use, kick them a little 
bit if their tools misbehavior or really need an update, extend their accounts 
and add new ssh-keys. Some times this job sucks, but most times it is a good 
feeling to know that other people use these tools and it helps the wikimedia-
projects.
I very doubt that ANY sysop at WMF would do that (can you imagine Domas 
helping an user to create its first ssh-key (and the second shortly after 
because the user commit the private key instead of the public)?).
I'm very sure that I have users on the TS that are able to use Wikilabs (like 
Merlissimo), but for most users it will be too complex.
> 
> Toolserver is in fact hosted by the Wikimedia Foundation today, in our
> Amsterdam data-center. We provide space, power and racks for the
> toolserver cluster, at a cost of about $65,000/year to WMF according
> to our Director of TechOps. We also maintain the database replication
> on our end which enables tools to function.

It is true that WMF hosts the TS and I am thankful for this. But these costs 
will not go away when the toolserver dies and everyone move to labs – because 
than you have to pay for Labs.
If "we created a mysql-user in the past which took 5 minute and never touched 
it again" is "maintain the database replication on our end " then it's also 
true. The ssh-tunnel to get the data is maintained by our side and most times 
the wmf-sysops are not able to even announce to us when the mysql-master 
changed (we will notice some hours later when our replag increase) or when 
they create a new wiki (we will learn weeks later when an user searches for an 
wiki in our database and can not find it).

> We can't provide the same level of service for the toolserver
> infrastructure as we do for core operations, and it makes no sense for
> a chapter to build out the required staffing and expertise to do so
> (set up/maintain all or some of the aforementioned functions). Even
> with slightly increased investment, toolserver would always suffer
> from being second or third tier infrastructure.

It made sense for years and worked.

> 
> 2) We're not comfortable hosting the toolserver infrastructure as-is.
> There are too many idiosyncratic aspects of its configuration; it has
> its own wiki, its own (closed source) version control system, its own
> (closed source) issue tracker. There are hacks like TUSC that we want
> to replace with better systems/services (e.g. OpenID/OAuth).

Because we use JIRA instead of Bugzilla and Fisheye (which is not a version 
control system BTW) instead of Gerrit the Toolserver has to die? Are you 
kidding? We get these system for free because River asked nicely, but if that 
is really such a problem we can move away from them.
And I was at the tech-meeting in Berlin 3 or 4 years ago where some wmf-techs 
told me, that OpenID would be "next year" – as we can all see the WMF has 
other things to do because there is no OpenID yet. To be clear: We WILL use 
OpenID when it is availablem but we can not use what is not there!

> 
> So, what's next?

What will happen next: I will request a change of the budget for WMDE at the 
general member-meeting of the WMDE.
If my change is accepted there will be some money to extend the toolserver in 
2013 so it can run again properly. Somewhen in 2013 Wikilabs will be moved to 
late 2014 and the TS has to run for another year.
If my change is not accepted, I will retire from the TS at 30. December 2012 
and it is not longer my beer what will happen. Somewhen in January the TS will 
collapse, because of non-maintenance, a wild-running tool or a security 
problem no-one are there to fix. Later this year Wikilabs will be moved to late 
2014. There will be no working plattform for at least two years.

There is also still the possibility that  WMDE comes to senses and changes the 
budget-plan of t

Re: [Toolserver-l] Future of the toolserver

[Hersfold Wikipedia]

+1. Data replication aside, a number of projects depend quite heavily
on the tools hosted on the toolserver. If we won't be able to complete
a transition to labs until 2014, *someone* needs to help keep the
Toolserver supported until then - if WMDE cannot, WMF should supply the
costs needed to maintain what is, at the end of the day, one of their
servers. DaB. also needs support with this, as he's been forced to be
the public face of the degrading server, and subjected to the ire of
all of us as a result (which I have participated in, and apologize for,
DaB. I think we all know you and Nosy are doing the best you can under
the circumstances).

Could the WMF and WMDE please collaborate to work out a plan to keep
the toolserver running as best as it can until labs is fully prepared
to take over?


User:Hersfold
hersfoldw...@gmail.com

Sent from my Windows Phone
From: Merlissimo
Sent: 9/25/2012 16:08
To: Wikimedia Toolserver
Subject: Re: [Toolserver-l] Future of the toolserver
Am 25.09.2012 20:48, schrieb Erik Moeller:
> Toolserver is in fact hosted by the Wikimedia Foundation today, in our
> Amsterdam data-center. [..] We also maintain the database replication
> on our end which enables tools to function.
>
I don't know all internal systems, but i think by maintaining you mean:
grant access to mysql binlogs, traffic costs and sometimes creating a dump.

Why can WMF not administrate the hole database replication servers for
toolserver users in short-term if WMDE should not spend money on this
anymore? Setting up new replication servers at production system is done
quite often. Adding views for hiding private data and adding access
control based on toolserver ldap should be possible. The rest of the
toolserver infrastructure won't be touched by this change.

Currently the replication of database cluster of s3/s6/s7 (all on server
hyacinth) is lagging for more than an hour, performance is very low and
complex queries are taking 10 times longer than normal, so that some of
my queries can not finish within maximum allowed runtime (which brakes
some of my tools since about five days). To solve this problem new
hardware is need. I as toolserver user don't care if support comes from
WMDE or WMF as long as this problem is fixed. I am the one with the
oversized user talk page because other authors asked me why my tools are
not working.

Merlissimo

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Tim Landscheidt]

Merlissimo  wrote:

>> Toolserver is in fact hosted by the Wikimedia Foundation today, in our
>> Amsterdam data-center. [..] We also maintain the database replication
>> on our end which enables tools to function.

> I don't know all internal systems, but i think by
> maintaining you mean: grant access to mysql binlogs, traffic
> costs and sometimes creating a dump.

> Why can WMF not administrate the hole database replication
> servers for toolserver users in short-term if WMDE should
> not spend money on this anymore? Setting up new replication
> servers at production system is done quite often. Adding
> views for hiding private data and adding access control
> based on toolserver ldap should be possible. The rest of the
> toolserver infrastructure won't be touched by this change.
> [...]

Apparently, it is not that smooth for WMF admins
(http://permalink.gmane.org/gmane.org.wikimedia.toolserver/3166):

| > one obvious solution would be to treat the Toolserver database servers
| > as part of WMF's domain so they can use their SOPs (*1) to maintain
| > them instead of manually coordinating with the Toolserver admins (or
| > not).

| I don't think that would be any easier.  Our database systems are
| completely different from theirs, so it would require all WMF admins to
| learn (and remember) a new procedure before they could change anything
| on their side.  Since they usually don't think to notify us of major
| changes in advance, I don't think they would be very happy with that.

Tim


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Merlissimo]

Am 25.09.2012 20:48, schrieb Erik Moeller:

Toolserver is in fact hosted by the Wikimedia Foundation today, in our
Amsterdam data-center. [..] We also maintain the database replication
on our end which enables tools to function.

I don't know all internal systems, but i think by maintaining you mean: 
grant access to mysql binlogs, traffic costs and sometimes creating a dump.


Why can WMF not administrate the hole database replication servers for 
toolserver users in short-term if WMDE should not spend money on this 
anymore? Setting up new replication servers at production system is done 
quite often. Adding views for hiding private data and adding access 
control based on toolserver ldap should be possible. The rest of the 
toolserver infrastructure won't be touched by this change.


Currently the replication of database cluster of s3/s6/s7 (all on server 
hyacinth) is lagging for more than an hour, performance is very low and 
complex queries are taking 10 times longer than normal, so that some of 
my queries can not finish within maximum allowed runtime (which brakes 
some of my tools since about five days). To solve this problem new 
hardware is need. I as toolserver user don't care if support comes from 
WMDE or WMF as long as this problem is fixed. I am the one with the 
oversized user talk page because other authors asked me why my tools are 
not working.


Merlissimo

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Erik Moeller]

Hi all,

let me weigh in with a few initial comments, and I'll ask the Labs
folks to participate here and on Meta as well with regard to technical
questions.

The initial focus for Labs has been to provide functionality that
toolserver doesn't - get root on a VM or set of VMs to install/test
arbitrary software/services, and get it ready for production
deployment. Labs doesn't have DB replication yet, and it doesn't yet
have an environment optimized for the development of small tools that
are not geared towards deployment in production. There are some
communities within Labs, most notably bots.wmflabs.org, that have
started to optimize their environment for certain categories of tools
(in this case bots).

The second phase of Wikimedia Labs is called "Tool Labs" and its
explicit goal is to be an alternative to the toolserver. This is
outlined here:

https://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Milestones_by_quarter_2

and here:

https://www.mediawiki.org/wiki/Wikimedia_Labs#Tool_Labs

In other words, it's not suprising that Labs cannot yet function as an
effective toolserver alternative for most purposes, because we've not
started work on the required functionality yet. Our timeline is to do
so beginning in Q1 of the next calendar year. With regard to DB
replication in particular, we're investigating whether we can
accelerate the schedule since it's so highly requested.

It is definitely the goal of ''Tool Labs'' to support the kind of
small-scale, non-deployed tools that toolserver authors love to
create.

Petr Bena, a Labs user, has created this page, which would definitely
benefit from more participation as well:
https://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted_in_Tool_Labs

It is true that we (WMF) have generally asked chapters to reduce
investment in core infrastructure/services, and specifically asked
WM-DE to work with us in transitioning from toolserver to Labs. There
are a number of reasons for this:

1) WMF is a technology organization. Hosting the core infrastructure
for Wikimedia projects is very much what we do. This includes data
center operation, monitoring and backups, software deployments,
software/service upgrades, code versioning infrastructure, bug
tracking infrastructure, additional support systems and services (like
this mailing list), etc.

Toolserver is in fact hosted by the Wikimedia Foundation today, in our
Amsterdam data-center. We provide space, power and racks for the
toolserver cluster, at a cost of about $65,000/year to WMF according
to our Director of TechOps. We also maintain the database replication
on our end which enables tools to function.

We can't provide the same level of service for the toolserver
infrastructure as we do for core operations, and it makes no sense for
a chapter to build out the required staffing and expertise to do so
(set up/maintain all or some of the aforementioned functions). Even
with slightly increased investment, toolserver would always suffer
from being second or third tier infrastructure.

2) We're not comfortable hosting the toolserver infrastructure as-is.
There are too many idiosyncratic aspects of its configuration; it has
its own wiki, its own (closed source) version control system, its own
(closed source) issue tracker. There are hacks like TUSC that we want
to replace with better systems/services (e.g. OpenID/OAuth).

So, what's next?

Chapters are autonomous organizations, and it's WM-DE's call how much
/ whether it wants to continue to invest in infrastructure of any kind
(and the decision of funding bodies like the FDC to accept or reject
that proposition). However, for our part, we will not continue to
support the current arrangement (DB replication, hosting in our
data-center, etc.) indefinitely.

The timeline we've discussed with Wikimedia Germany is roughly as follows:

- Wind down new account creation on toolserver by Q2 of 2013 calendar year
- Decommission toolserver by December 2013

WMF can't commit to providing technical support for tool transition
(there are too many tools), so if there's any area where I think it
would make sense to ramp up investments on WM-DE's part, it's in
engineering capacity to support tool developers in porting tools to
Labs.

That said, there may be a need for emergency purchases/investments to
keep TS in a usable state until December 2013 (and perhaps allow for
some buffer room beyond that). That's not our call to make.

Hope this clears up some questions around what's going on, and happy
to answer further questions.

All best,
Erik

-- 
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Federico Leva (Nemo)]

Delphine and Ariel have already showed the approach to follow.
 
is indeed very lacking; it should be expanded and each item moved to 
bugzilla. Tool Labs is currently not even on the radars, as far as I 
understood; some schedule is needed from the WMF.


Given the tone of this thread, I also want to point out that WMDE 
doesn't seem to have any fault to me here: it was the WMF's (sudden) 
decision, a year or two ago, to "replace" (aka kill) the Toolserver 
rather than helping it (after the one and only little grant in 2009) or 
letting other chapters help it. It obviously doesn't make sense to 
invest much on the Toolserver while the WMF is spending millions to 
duplicate it.
Again, a document like the one Delphine requested would also help WMF 
understand what's needed to make the transition less traumatic: they 
need a clear plan and it's their responsibility to coordinate better 
with the service the state to be replacing, including financial aid. The 
Funds Dissemination Committee doesn't have any role about the part of 
WMF's budget which includes Labs, but they'd supposedly do something 
about such a problem if they want their work to mean anything IMHO.


Nemo

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Matthew Bowker]

The Account Request process and the Unblock Request process on the English 
Wikipedia also run on Toolserver.  Account requests is interesting, because the 
en.wikipedia community recently decided not to move this process back on-wiki: 
https://en.wikipedia.org/wiki/Wikipedia_talk:ACC#Account_Request_extension.2C_revisited

Not sure if it matters, but my web-based tools are located at 
https://github.com/Matthewrbowker/toolserver and my bots will be located at 
https://github.com/Matthewrbowker/toolserverBots .  

Matthew Bowker

On Sep 25, 2012, at 4:45 AM, Andre Koopal  wrote:

> On Tue, Sep 25, 2012 at 11:10:13AM +0200, Platonides wrote:
>> On 25/09/12 00:51, DaB. wrote:
>>> Hello all,
>>> 
>>> in these days WMDE (the chapter that finance the toolserver) is discussing 
>>> the 
>>> budget for the next year (2013); you can find it at [1]. At the moment 
>>> there is 
>>> no money for new toolserver-hardware in this budget and the CEO Pavel 
>>> Richter 
>>> is unwilling to change this ([2] in german) – because he fears that there 
>>> will 
>>> be a Wikilabs in 2014. It is not possible for me to run the toolserver for 
>>> another year with the current hardware – you all know why. For this reason 
>>> I 
>>> will request a change of the budget at the general meeting at November, so 
>>> there will be a vote about. If this vote should fail (and we get no money 
>>> for 
>>> new hardware), I am going to retire from my job as root at 30. December 
>>> 2012.
>>> I'm not longer able to tolerant the behavior of the german chapter and the 
>>> WMF 
>>> in matter of the toolserver; I do this for free and for fun, and it is not 
>>> longer fun.
>>> 
>>> Sincerely,
>>> DaB.
>> 
>> 
>> So, out of fear that there may be a better alternative in two years
>> time, he decides to stop supporting it now?
>> 
>> I know labs, and I'm not sure it will really be a better alternative.
>> Currently, it isn't. The toolserver is much more reliable and flexible.
>> I don't think labs will "win" in the near future, either. Although it
>> might change in two years. Specially if no attention is payed to the
>> toolserver for that time.
>> 
>> I see two big problems:
>> 1) If labs really becomes the "perfect tool hosting" in 2014, What
>> happens before we reach that? "Yes, your tool doesn't work, migrate to
>> labs next year"
>> 
>> 2) We risk ending up with no good alternative at that time. labs is
>> still not good enough, toolserver has degraded so much it's unusable.
>> 
>> 
>> 
>> Not to mention the migration cost that would need to be payed by tool
>> authors if forced to move. Although perhaps he doesn't care.
> 
> I think we should help DaB to make a case why toolserver is important. We
> should list the projects depending on toolserver and how, and all the other
> things that are important. What I can quickly think of:
> 
> - Wiki Loves Monuments: All the tooling around this really need toolserver.
> - Commons: commons has a lot of toolserver tools more or less integrated
>  in the interface, and there are supporttools running.
> - GLAM: a lot of glam-related work is done on toolservers
> - small tools for wiki support: all the bots archiving talkpages, create
>  administrative pages automatically etc, etc ...
> - steward support: There are a lot of tools (also used generally) that really
>  make cross-wiki abuse detection and other stewardwork much easier.
> 
> With a document like that we can make a stronger case for WMDE to reserve
> budget, or have them apply for a separate grant to the foundation.
> 
> Regards,
> 
> Andre
> 
>>> P.S: If you are in a board of a chapter that gives money to WMDE for the 
>>> toolserver: Make sure that it will be spend for hardware. 
>>> 
>>> [1] 
>>> http://meta.wikimedia.org/wiki/Wikimedia_Deutschland/2013_annual_plan_draft/en
>>> [2] 
>>> meta.wikimedia.org/wiki/Talk:Wikimedia_Deutschland/2013_annual_plan_draft/de#Toolserver
>>> 
>>> 
>>> 
>>> 
>>> ___
>>> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
>>> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>>> Posting guidelines for this list: 
>>> https://wiki.toolserver.org/view/Mailing_list_etiquette
>> 
>> 
>> ___
>> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
>> Posting guidelines for this list: 
>> https://wiki.toolserver.org/view/Mailing_list_etiquette
>> 
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.

Re: [Toolserver-l] Future of the toolserver

[Merlissimo]

I think its the wrong way to how the migration is done. Currently the 
plan is to disabled toolserver at the same time as tool Labs is full 
available.


I am running very complex tools and queries which are highly optimized 
for the toolserver infrastructure so that results are returned in an 
acceptable time. Migrating these tools to a new environment would take 
very much time. So to run these tools without an outage there need to be 
along time both projects must be available.


Why is WMF not helping maintaining parts of the toolserver? My 
impression is that most of load problems caused on the toolserver are 
database server problems. Many queries are very complex for the mysql 
database to handle because they are not key based (and they cannot be 
rewritten to be key based). Why can WMF not maintain only these database 
replication servers in short-term and make them accessable for 
toolserver user? Even if these are only rr-server on the first step this 
would be a big benefit. Yesterday is learned that wmf exmploys 90+ 
people that should have much experience for administration servers. 
After sql servers are maintained by wmf admins and hardware the current 
toolserver database server could be reused for other parts (maybe as 
webserver).


Btw: On sunday i submitted a critical bug to bugzilla because since 
saturday my interwiki bot shows that there must be some misconfigured 
api squids (perhaps because they are out of sync). Nobody of these 90 
wmf admins has taken care of this bug until now. Maybe solving this is 
not explicitly contained in the job descrition of most of these admins 
and so they do not get a point for their year goals. Toolserver also had 
a problem on sunday and volunteer admin DaB. solved this problem within 
the red-letter day.


Merlissimo

Am 25.09.2012 15:20, schrieb Thehelpfulone:


On 25 September 2012 14:15, Ariel T. Glenn mailto:ar...@wikimedia.org>> wrote:

It might be helpful to put together a list of functions that the
toolserver supports but that labs currently does not; such a list could
serve as a basis for talks with the WMF.  Perhaps the labs folks could
makes some guesses at when those functions would be available and stable
there, which would give everyone a better idea about how long the
transition would realistically take.

If I am not mistaken, one of the big items is the ability to run
expensive db queries without impacting production.  I don't believe this
is possible from labs right now, and I'm not sure what their plans are
for that.

Ariel

p.s. this post is by me as a former toolserver user, having nothing to
do with my status as a wmf staff member etc.

There is a partial list at
http://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted.
According to the milestones at
http://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Milestones_by_quarter_2,
we should be expecting database replication from production and user
databases in January-March 2013.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Thehelpfulone]

On 25 September 2012 14:15, Ariel T. Glenn  wrote:

>  It might be helpful to put together a list of functions that the
> toolserver supports but that labs currently does not; such a list could
> serve as a basis for talks with the WMF.  Perhaps the labs folks could
> makes some guesses at when those functions would be available and stable
> there, which would give everyone a better idea about how long the
> transition would realistically take.
>
> If I am not mistaken, one of the big items is the ability to run
> expensive db queries without impacting production.  I don't believe this
> is possible from labs right now, and I'm not sure what their plans are
> for that.
>
> Ariel
>
> p.s. this post is by me as a former toolserver user, having nothing to
> do with my status as a wmf staff member etc.
>

There is a partial list at
http://www.mediawiki.org/wiki/Wikimedia_Labs/Toolserver_features_wanted.

According to the milestones at
http://www.mediawiki.org/wiki/Wikimedia_Engineering/2012-13_Goals#Milestones_by_quarter_2,
we should be expecting database replication from production and user
databases in January-March 2013.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Thehelpfulone]

On 25 September 2012 14:01, DaB.  wrote:

>  You are right in 1 thing: WMDE has no plan to abandon the toolserver, it
> has
> already abandon it. In 2012 there was no investment in new server and I
> very
> doubt that there will be any new servers this year. You just finance the
> "running", meaning that WMDE pay replacement parts if something breaks and
> you
> pay the loan of Nosy. But that is not enough: The load is growing. The
> toolserver has more users like in 2011, more tools like in 2011, and more
> people use this tools than in 2011; but there is no "more toolserver". That
> was bearable in 2012, but it will definitively not possible in 2013
> because we
> have exceeded our possibilities.
>

Am I correct in thinking that the WMF gives WMDE a grant for the running of
the toolserver, or was this just a one time thing back in 2009 (
http://meta.wikimedia.org/wiki/Grants:WM_DE/Improve_toolserver_reliability)?


If it's regular, does anyone know the details of it (is it annual, how much
is it, is there a breakdown of how it is expected to be spent etc)?

Actually, is there a page with a list of all the financial (and in-kind)
support that is provided to the Toolserver, by the WMF, chapters and
individuals?

THO
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ariel T. Glenn]

Στις 25-09-2012, ημέρα Τρι, και ώρα 14:15 +0200, ο/η Delphine Ménard
έγραψε:


> 
> I hope you can find support for writing up this document and we
> welcome your input of what is needed to ease the transition into the
> year 2013. You all might want to make sure that you reach out also to
> the Foundation to explain what is needed for Labs to take over
> smoothly.


It might be helpful to put together a list of functions that the
toolserver supports but that labs currently does not; such a list could
serve as a basis for talks with the WMF.  Perhaps the labs folks could
makes some guesses at when those functions would be available and stable
there, which would give everyone a better idea about how long the
transition would realistically take.

If I am not mistaken, one of the big items is the ability to run
expensive db queries without impacting production.  I don't believe this
is possible from labs right now, and I'm not sure what their plans are
for that.

Ariel

p.s. this post is by me as a former toolserver user, having nothing to
do with my status as a wmf staff member etc.



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[DaB.]

Hello,
At Tuesday 25 September 2012 14:39:57 DaB. wrote:
> As Pavel has stated before, there is no plan to abandon the
> toolserver, and all necessary investments will be made to make sure
> that it actually performs at least as well as it has until now until
> Labs becomes a suitable replacement. However, we can't at this stage
> enter in expensive improvements,

You are right in 1 thing: WMDE has no plan to abandon the toolserver, it has 
already abandon it. In 2012 there was no investment in new server and I very 
doubt that there will be any new servers this year. You just finance the 
"running", meaning that WMDE pay replacement parts if something breaks and you 
pay the loan of Nosy. But that is not enough: The load is growing. The 
toolserver has more users like in 2011, more tools like in 2011, and more 
people use this tools than in 2011; but there is no "more toolserver". That 
was bearable in 2012, but it will definitively not possible in 2013 because we 
have exceeded our possibilities.
Lets make an analogy: The Toolserver is like a firm that has cars for its 
employees (for customer-visits). There are 30 cars, which was great in 2011 
because there were 25 employees. In 2012 the company have grown and there are 
now 35 employees with 30 cars – it was problematic but it was somehow working 
most of the time. Now the guy who oversees the cars knows that in 2013 there 
will be 40 or maybe 50 employees and so the firm has to buy new cars. But the 
management just says: You will get no further cars, but if one broke we will 
repair it. Do you think that this will work?

And please notice that neither you or Pavel decides about the WMDE-bugdet for 
2013, but the members of WMDE.

Sincerely,
DaB.

-- 
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885


signature.asc
Description: This is a digitally signed message part.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Andre Koopal]

On Tue, Sep 25, 2012 at 02:15:35PM +0200, Delphine Ménard wrote:
> Hi Andre,
> 
> As Pavel states, we are also looking at the idea of maybe making
> improvements that can then be handed over to Labs. Note that this
> might be complicated, as this would mean buying hardware in Europe
> that would then somehow have to be administrated from the US, so we
> are not sure what such a scenario might look like and if it is
> feasible at all.
> 
Just as a very quick comment to this one, the foundation already has
hardware in europe, administrated from the US, even in the same datacentre
as toolserver is now, so those structures do exist.

Regards,

Andre

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Delphine Ménard]

Hi Andre,

On Tue, Sep 25, 2012 at 12:45 PM, Andre Koopal  wrote:

>
> I think we should help DaB to make a case why toolserver is important. We
> should list the projects depending on toolserver and how, and all the other
> things that are important. What I can quickly think of:
>
> - Wiki Loves Monuments: All the tooling around this really need toolserver.
> - Commons: commons has a lot of toolserver tools more or less integrated
>   in the interface, and there are supporttools running.
> - GLAM: a lot of glam-related work is done on toolservers
> - small tools for wiki support: all the bots archiving talkpages, create
>   administrative pages automatically etc, etc ...
> - steward support: There are a lot of tools (also used generally) that really
>   make cross-wiki abuse detection and other stewardwork much easier.
>
> With a document like that we can make a stronger case for WMDE to reserve
> budget, or have them apply for a separate grant to the foundation.

Thanks for this constructive approach. I believe this would be indeed
the best way to go forward on this issue and would help the General
Assembly of Wikimedia Deutschland make the right decision about this.
As Pavel has stated before, there is no plan to abandon the
toolserver, and all necessary investments will be made to make sure
that it actually performs at least as well as it has until now until
Labs becomes a suitable replacement. However, we can't at this stage
enter in expensive improvements, because Labs is, in the short to
mid-run, destined to replace the "toolserver as you know it"
completely. It falls under the attributions of the Foundation, as
hosting provider of the Wikimedia Projects, to provide authors,
developers and contributors with the technical tools they need to make
their work easier.
As Pavel states, we are also looking at the idea of maybe making
improvements that can then be handed over to Labs. Note that this
might be complicated, as this would mean buying hardware in Europe
that would then somehow have to be administrated from the US, so we
are not sure what such a scenario might look like and if it is
feasible at all.

I hope you can find support for writing up this document and we
welcome your input of what is needed to ease the transition into the
year 2013. You all might want to make sure that you reach out also to
the Foundation to explain what is needed for Labs to take over
smoothly.


Best regards,


Delphine

-- 
Delphine Ménard
Treasurer
Wikimedia Deutschland

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Fwd: Re: [Wikimediauk-l] Fwd: Future of the toolserver

[Katie Chan]

If we're going to forward the original email, then let's forward the 
reply to it from Pavel as well.


 Original Message 
Subject:Re: [Wikimediauk-l] Fwd: [Toolserver-l] Future of the toolserver
Date:   Tue, 25 Sep 2012 11:08:22 +0200
From:   Pavel Richter 
Reply-To:   UK Wikimedia mailing list 
To: UK Wikimedia mailing list ,
Toolserver_Management 



Just to clarify: As I have stated many time before, Wikimedia
Deutschland is committed to run and maintain the Toolserver as long as
Wikilabs are not in position to offer the same level of service as the
Toolserver. This committment includes the purchase of new hardware,
where necessary to keep the servers running. As in the past years, I
hope for support by other entities in this project, both financially and
with administration of the Toolserver.


Mit freundlichen Grüßen,

Pavel Richter
CEO

Wikimedia Deutschland e.V.
Tel.: +49 - 30 - 219 158 260
Twitter: @pavel



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Danny B .]

> no money for new toolserver-hardware in this budget and the CEO Pavel Richter 
> is unwilling to change this ([2] in german) – because he fears that there 
> will 
> be a Wikilabs in 2014. It is not possible for me to run the toolserver for 
> another year with the current hardware – you all know why. For this reason I 
> will request a change of the budget at the general meeting at November, so 
> there will be a vote about.

Is there any list of necessary items (minimal and optimal) together with 
estimated price so we can know what and how much we're talking about? That 
would be helpful for instance when looking for potential sponsors.

Also, are we rather interested in couple brand new boxes or bunch of used is 
acceptable as well?


Kind regards


Danny B.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Andre Koopal]

On Tue, Sep 25, 2012 at 11:10:13AM +0200, Platonides wrote:
> On 25/09/12 00:51, DaB. wrote:
> > Hello all,
> > 
> > in these days WMDE (the chapter that finance the toolserver) is discussing 
> > the 
> > budget for the next year (2013); you can find it at [1]. At the moment 
> > there is 
> > no money for new toolserver-hardware in this budget and the CEO Pavel 
> > Richter 
> > is unwilling to change this ([2] in german) – because he fears that there 
> > will 
> > be a Wikilabs in 2014. It is not possible for me to run the toolserver for 
> > another year with the current hardware – you all know why. For this reason 
> > I 
> > will request a change of the budget at the general meeting at November, so 
> > there will be a vote about. If this vote should fail (and we get no money 
> > for 
> > new hardware), I am going to retire from my job as root at 30. December 
> > 2012.
> > I'm not longer able to tolerant the behavior of the german chapter and the 
> > WMF 
> > in matter of the toolserver; I do this for free and for fun, and it is not 
> > longer fun.
> > 
> > Sincerely,
> > DaB.
> 
> 
> So, out of fear that there may be a better alternative in two years
> time, he decides to stop supporting it now?
> 
> I know labs, and I'm not sure it will really be a better alternative.
> Currently, it isn't. The toolserver is much more reliable and flexible.
> I don't think labs will "win" in the near future, either. Although it
> might change in two years. Specially if no attention is payed to the
> toolserver for that time.
> 
> I see two big problems:
> 1) If labs really becomes the "perfect tool hosting" in 2014, What
> happens before we reach that? "Yes, your tool doesn't work, migrate to
> labs next year"
> 
> 2) We risk ending up with no good alternative at that time. labs is
> still not good enough, toolserver has degraded so much it's unusable.
> 
> 
> 
> Not to mention the migration cost that would need to be payed by tool
> authors if forced to move. Although perhaps he doesn't care.

I think we should help DaB to make a case why toolserver is important. We
should list the projects depending on toolserver and how, and all the other
things that are important. What I can quickly think of:

- Wiki Loves Monuments: All the tooling around this really need toolserver.
- Commons: commons has a lot of toolserver tools more or less integrated
  in the interface, and there are supporttools running.
- GLAM: a lot of glam-related work is done on toolservers
- small tools for wiki support: all the bots archiving talkpages, create
  administrative pages automatically etc, etc ...
- steward support: There are a lot of tools (also used generally) that really
  make cross-wiki abuse detection and other stewardwork much easier.

With a document like that we can make a stronger case for WMDE to reserve
budget, or have them apply for a separate grant to the foundation.

Regards,

Andre

> > P.S: If you are in a board of a chapter that gives money to WMDE for the 
> > toolserver: Make sure that it will be spend for hardware. 
> > 
> > [1] 
> > http://meta.wikimedia.org/wiki/Wikimedia_Deutschland/2013_annual_plan_draft/en
> > [2] 
> > meta.wikimedia.org/wiki/Talk:Wikimedia_Deutschland/2013_annual_plan_draft/de#Toolserver
> > 
> > 
> > 
> > 
> > ___
> > Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> > https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> > Posting guidelines for this list: 
> > https://wiki.toolserver.org/view/Mailing_list_etiquette
> 
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette
> 

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[DeltaQuad Wikipedia]

Maybe someone can translate the German better than stupid google translate
for me, but it sounds like he's willing to support the existing
infrastructure but not go any further (as in upgrade or supply anything we
need) because Labs is going to be 'the equivalent' of toolserver or better
by 2014, supposedly. Honestly, right now I doubt many people on TS 1) want
to move to Labs (because of no WMF database replication being the biggest
complaint) 2) Understand labs enough to work it (I don't, someone has
offered to teach me, but I have to find the time) 3) It takes a lot of time
to migrate and test. We simply just don't have enough man hours with our
real jobs/school to make this move over, especially for single person tools.

DaB, I honestly support your decision if what I sad above is correct about
the attitude, because I wouldn't want to work at toolserver either. No
sysadmin wants to work with people constantly looking at them for why
things will not work and then management who won't give the tools to do it.

I'm personally starting to get my tools and talking with my MMP Co-Devs
about moving over to labs, but one project needs the Antispoof table from
WMF databases which labs doesn't have. So are we ending all support for it
and scrapping the project because of issues with the toolserver not being
funded enough to work? Never have I seen a more crazy proposal to end
support for services that aren't working already for a volunteer community.

DeltaQuad
English Wikipedia Administrator and CheckUser
ACC and UTRS Developer

On Tue, Sep 25, 2012 at 5:10 AM, Platonides  wrote:

> On 25/09/12 00:51, DaB. wrote:
> > Hello all,
> >
> > in these days WMDE (the chapter that finance the toolserver) is
> discussing the
> > budget for the next year (2013); you can find it at [1]. At the moment
> there is
> > no money for new toolserver-hardware in this budget and the CEO Pavel
> Richter
> > is unwilling to change this ([2] in german) – because he fears that
> there will
> > be a Wikilabs in 2014. It is not possible for me to run the toolserver
> for
> > another year with the current hardware – you all know why. For this
> reason I
> > will request a change of the budget at the general meeting at November,
> so
> > there will be a vote about. If this vote should fail (and we get no
> money for
> > new hardware), I am going to retire from my job as root at 30. December
> 2012.
> > I'm not longer able to tolerant the behavior of the german chapter and
> the WMF
> > in matter of the toolserver; I do this for free and for fun, and it is
> not
> > longer fun.
> >
> > Sincerely,
> > DaB.
>
>
> So, out of fear that there may be a better alternative in two years
> time, he decides to stop supporting it now?
>
> I know labs, and I'm not sure it will really be a better alternative.
> Currently, it isn't. The toolserver is much more reliable and flexible.
> I don't think labs will "win" in the near future, either. Although it
> might change in two years. Specially if no attention is payed to the
> toolserver for that time.
>
> I see two big problems:
> 1) If labs really becomes the "perfect tool hosting" in 2014, What
> happens before we reach that? "Yes, your tool doesn't work, migrate to
> labs next year"
>
> 2) We risk ending up with no good alternative at that time. labs is
> still not good enough, toolserver has degraded so much it's unusable.
>
>
>
> Not to mention the migration cost that would need to be payed by tool
> authors if forced to move. Although perhaps he doesn't care.
>
>
>
>
>
>
> > P.S: If you are in a board of a chapter that gives money to WMDE for the
> > toolserver: Make sure that it will be spend for hardware.
> >
> > [1]
> >
> http://meta.wikimedia.org/wiki/Wikimedia_Deutschland/2013_annual_plan_draft/en
> > [2]
> >
> meta.wikimedia.org/wiki/Talk:Wikimedia_Deutschland/2013_annual_plan_draft/de#Toolserver
> >
> >
> >
> >
> > ___
> > Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> > https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> > Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list:
> https://wiki.toolserver.org/view/Mailing_list_etiquette
>
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 25/09/12 00:51, DaB. wrote:
> Hello all,
> 
> in these days WMDE (the chapter that finance the toolserver) is discussing 
> the 
> budget for the next year (2013); you can find it at [1]. At the moment there 
> is 
> no money for new toolserver-hardware in this budget and the CEO Pavel Richter 
> is unwilling to change this ([2] in german) – because he fears that there 
> will 
> be a Wikilabs in 2014. It is not possible for me to run the toolserver for 
> another year with the current hardware – you all know why. For this reason I 
> will request a change of the budget at the general meeting at November, so 
> there will be a vote about. If this vote should fail (and we get no money for 
> new hardware), I am going to retire from my job as root at 30. December 2012.
> I'm not longer able to tolerant the behavior of the german chapter and the 
> WMF 
> in matter of the toolserver; I do this for free and for fun, and it is not 
> longer fun.
> 
> Sincerely,
> DaB.


So, out of fear that there may be a better alternative in two years
time, he decides to stop supporting it now?

I know labs, and I'm not sure it will really be a better alternative.
Currently, it isn't. The toolserver is much more reliable and flexible.
I don't think labs will "win" in the near future, either. Although it
might change in two years. Specially if no attention is payed to the
toolserver for that time.

I see two big problems:
1) If labs really becomes the "perfect tool hosting" in 2014, What
happens before we reach that? "Yes, your tool doesn't work, migrate to
labs next year"

2) We risk ending up with no good alternative at that time. labs is
still not good enough, toolserver has degraded so much it's unusable.



Not to mention the migration cost that would need to be payed by tool
authors if forced to move. Although perhaps he doesn't care.






> P.S: If you are in a board of a chapter that gives money to WMDE for the 
> toolserver: Make sure that it will be spend for hardware. 
> 
> [1] 
> http://meta.wikimedia.org/wiki/Wikimedia_Deutschland/2013_annual_plan_draft/en
> [2] 
> meta.wikimedia.org/wiki/Talk:Wikimedia_Deutschland/2013_annual_plan_draft/de#Toolserver
> 
> 
> 
> 
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[ralf]

On Tue, Sep 25, 2012 at 08:42:19AM +0200, Mike Dupont wrote:
> > , I cloned my applet page on github as backup
> also who is backing up github? please share important urls.

Only important for WP chemistry authors (they were already informed):
http://toolserver.org/~ayacop/EditorApplet.html =
http://jchempaint.github.com/EditorApplet.html



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette