Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Tim Landscheidt]

(anonymous) wrote:

> [...]
> Ryan Lane wrote:
>> If WMF becomes evil, fork the entire infrastructure into EC2,
>> Rackspace cloud, HP cloud, etc. and bring the community operations
>> people along for the ride. Hell, use the replicated databases in Labs
>> to populate your database in the cloud.

> Tim Landscheidt wrote:
>> But the nice thing about Labs is that you can try out (re-
>> plicable :-)) replication setups at no cost, and don't have
>> to upfront investments on hardware, etc., so when time
>> comes, you can just upload your setup to EC2 or whatever and
>> have a working Wikipedia clone running in a manageable time-
>> frame.

> This is not an easy task. Replicating the databases is enormously
> challenging (they're huge datasets in the cases of the big wikis) and
> they're constantly changing. If you tried to rely on dumps alone, you'd
> always be out of date by at least two weeks (assuming dumps are working
> properly). Two weeks on the Internet is a lot of time.

I don't know if this is not an easy task, but you are proba-
bly right.  So what?  If a scenario of WMF turning rogue
couldn't bear losing two weeks of edits while saving almost
a decade, we should work on ways to incremental dumps.

> But more to the point, even if you suddenly had a lot of infrastructure
> (bandwidth for constantly retrieving the data, space to store it all, and
> extra memory and CPU to allow users to, y'know, do something with it) and
> even if you suddenly had staff capable of managing these databases, not
> every table is in even available currently. As far as I'm aware,
> http://dumps.wikimedia.org doesn't include tables such as "user",
> "ipblocks", "archive", "watchlist", any tables related to global images or
> global user accounts, and probably many others. I'm not sure a full audit
> has ever been done, but this is partially tracked by
> .

The first part is easy: You go to some supplier and buy
bandwith, space, memory and CPU.  There is even staff for
hire.

  The second part is simple as well: What do you need
"ipblocks" or "watchlist" in a Wikipedia clone for?  It cer-
tainly is neither free content nor the content users use Wi-
kipedia for.

> So beyond the silly simplicity of the suggestion that one could simply "move
> to the cloud!", there are currently technical impossibilities to doing so.

And it would be far more helpful if you could stop spreading
FUD and instead show what actual impediments there are, for
example in a Labs project.

Tim


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Ryan Lane]

>> We currently have no plans for having the user databases on the same
>> servers as the replicated databases. Direct joins will not be
>> possible, so tools will need to be modified.
>
> -50
>
> It's such a useful feature, that it would be worth making a local mysql
> slaves for having them.
> I know, the all-powerful labs environment is unable to run a mysql
> instance, but we could use MySQL cluster, trading memory (available) to
> get joins (denied).
>

I'm not the one setting up the databases. If you want information
about why this won't be available, talk to Asher (binasher in
#wikimedia-operations on Freenode). Maybe he can be convinced
otherwise.

Of course, in the production cluster we don't do joins this way. We
handle the joins in the app logic, which is a more appropriate way of
doing this.

> SGE is a strong queue system. We have people and tools already trained
> to use it. It would be my first option.
> That said, if the presented alternative has the same user interface, it
> shouldn't be a problem. For instance, I don't have an opinion about
> which of the SGE forks would be preferable.
>

In general in Labs we don't have a large need for a queuing system
right now. If Toolserver folks need it very badly, it's possible to
add, someone just needs to put the effort into it. It likely wouldn't
be amazingly hard to puppetize this to run in a single project. Making
things multi-project is difficult and takes effort. Anyone can do the
single-project version in a project, multi-project will likely take
engineering effort.

>
>>> * no support for servlets
>>
>> I'm not sure what you mean by servlet?
>
> J2EE, I guess.
>

Well, if it's available in the ubuntu repos, or if it's open source,
then it's available in Labs.

>> I'd love DaB to help us improve Labs.
>>
>> Everything about Labs is fully open. Anyone can help build it, even
>> the production portions.
>>
> Would it be worth our efforts? I sometimes wonder why we should work on
> that (yes, I'm pessimistic right now).
> For instance the squid in front of *.beta.wmflabs.org. It was configured
> by Petan and me. We had absolutely no support from the WMF. The squid
> wasn't purging correctly. It worked on production, so there was a config
> error somewhere.
> We begged to see the squid config for months. But as it was in the
> private repository, no, it can't be shown, just in case it has something
> secret (very unlikely for squid config). Yes, we will clean them up and
> publish, eventually. Months passed (not to mention how publishing the
> config had been requested years ago). It could have been quickly
> reviewed before handing out, and we weren't going to abuse it if there
> really something weird was there. Replicating the WMF setup was done
> without viewing that same setup. I finally fixed it. I was quite proud
> of having solved it.

And you should be. Your changes kept that project moving along for
months until I broke it.

> Where is that file right now? It vanished. The file was lost in one of
> the multiple corruptions of labs instances. It was replaced with a copy
> of the cluster config (which was finally published in the meantime).
> So it feels like wasted effort now. I'd have liked to save a local copy
> at least.
>

To be fair, there's only been a single occurrence of instance
corruption, which was due to a bug in KVM.

Also, yes, the squid configuration was finally published because ones
of the devs spent the time to do so. I was working on stabilizing
things most of that time.

Does this mean your efforts were wasted? Of course not. Your efforts
helped keep the project running, which is important. Just because your
file was replaced with the production copy doesn't mean the work put
into it was for nothing.

> It's not enough to leave tools there and say "It is fully open. Anyone
> can help build it"
>

We're also putting effort into making the migration happen, but we're
focusing our efforts in different places. We can't do everything,
which is why I'm trying to encourage others to help out. If we work on
separate pieces of the work it'll go much quicker.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Ryan Lane]

On Wed, Sep 26, 2012 at 6:29 PM, Hersfold  wrote:
> You may not have meant for it to lead to the end of the Toolserver, but
> apparently that's how WMDE is taking it, and it sounds like that's going to
> be the inevitable result. To say otherwise is rather naive at this point,
> given the size of the threads talking about this.
>

I'll be honest, I don't really care about the politics behind any of
this, and I'm going to ignore anything more related to that. WMDE
dropping Toolserver is their decision and it doesn't affect how Labs
will operate in the future.

Labs is adding infrastructure needed to support Toolserver users. If
there's anything the Toolserver community needs that isn't in our
current roadmap, I'm more than happy to work those issues with the
community. The environment isn't going to be exactly the same, so
tools and bots may need to be modified. We can provide the necessary
resources, access, and training to integrate into the new environment.
WMDE will be providing resources to help with migrations.

Overall the environment provided by Labs has the ability to be much
more flexible and much more powerful than Toolserver. I hope everyone
migrates over, but I'll understand if anyone feels like it's too much
work.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Platonides]

On 26/09/12 20:25, Ryan Lane wrote:
>> temporary blockers
>> * no replication of wikimedia wiki databases
>> ** joining of user databases with wiki databases
> 
> We currently have no plans for having the user databases on the same
> servers as the replicated databases. Direct joins will not be
> possible, so tools will need to be modified.

-50

It's such a useful feature, that it would be worth making a local mysql
slaves for having them.
I know, the all-powerful labs environment is unable to run a mysql
instance, but we could use MySQL cluster, trading memory (available) to
get joins (denied).




>> * no support for script execution dependency (on ts: currently done by sge)
> 
> There's less of a need for this in Labs. If whatever you are running
> is really expensive, you can have your own instance. That said, I was
> looking at integrating a global queuing system. It won't be SGE,
> though.
> 
> If someone is really keen on SGE, then I recommend they work with us
> to puppetize it. Thankfully, open grid engine is already packaged in
> ubuntu, which should make that much easier.

SGE is a strong queue system. We have people and tools already trained
to use it. It would be my first option.
That said, if the presented alternative has the same user interface, it
shouldn't be a problem. For instance, I don't have an opinion about
which of the SGE forks would be preferable.


>> * no support for servlets
> 
> I'm not sure what you mean by servlet?

J2EE, I guess.




>> * no DaB.
>>
> 
> I'd love DaB to help us improve Labs.
> 
> Everything about Labs is fully open. Anyone can help build it, even
> the production portions.
> 
> - Ryan

Would it be worth our efforts? I sometimes wonder why we should work on
that (yes, I'm pessimistic right now).
For instance the squid in front of *.beta.wmflabs.org. It was configured
by Petan and me. We had absolutely no support from the WMF. The squid
wasn't purging correctly. It worked on production, so there was a config
error somewhere.
We begged to see the squid config for months. But as it was in the
private repository, no, it can't be shown, just in case it has something
secret (very unlikely for squid config). Yes, we will clean them up and
publish, eventually. Months passed (not to mention how publishing the
config had been requested years ago). It could have been quickly
reviewed before handing out, and we weren't going to abuse it if there
really something weird was there. Replicating the WMF setup was done
without viewing that same setup. I finally fixed it. I was quite proud
of having solved it.
Where is that file right now? It vanished. The file was lost in one of
the multiple corruptions of labs instances. It was replaced with a copy
of the cluster config (which was finally published in the meantime).
So it feels like wasted effort now. I'd have liked to save a local copy
at least.

It's not enough to leave tools there and say "It is fully open. Anyone
can help build it"


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Hersfold]

You may not have meant for it to lead to the end of the Toolserver, but 
apparently that's how WMDE is taking it, and it sounds like that's going 
to be the inevitable result. To say otherwise is rather naive at this 
point, given the size of the threads talking about this.



User:Hersfold
hersfoldw...@gmail.com

On 9/26/2012 6:06 PM, Ryan Lane wrote:

Yes, there's a difference. But in this case, as far as I understand it, a
direct cost (or casualty) of setting up Wikimedia Labs is the existence of
the Toolserver. Does Wikimedia need a great testing infrastructure? Yes, of
course. (And it's not as though the Toolserver has ever been without its
share of issues; I'm not trying to white-wash the past here.) But the
question is: if such a Wikimedia testing infrastructure comes at the cost of
losing the Toolserver, is that acceptable?


This is a scarecrow argument. The mere existence of Labs doesn't mean
the loss of Toolserver.

Labs is more than just a testing infrastructure. It's an
infrastructure for creating things, for enable volunteer operations,
for bringing operations and development together, for integrating
other projects, and for providing free hosting to projects that may
not have it otherwise. Labs just also happens to need some of the same
features as Toolserver.

Again, as I've mentioned, Labs purpose isn't a Toolserver replacement.
It's vision is much, much larger than what the Toolserver can do.


Ryan Lane wrote:

If WMF becomes evil, fork the entire infrastructure into EC2,
Rackspace cloud, HP cloud, etc. and bring the community operations
people along for the ride. Hell, use the replicated databases in Labs
to populate your database in the cloud.

Tim Landscheidt wrote:

But the nice thing about Labs is that you can try out (re-
plicable :-)) replication setups at no cost, and don't have
to upfront investments on hardware, etc., so when time
comes, you can just upload your setup to EC2 or whatever and
have a working Wikipedia clone running in a manageable time-
frame.

This is not an easy task. Replicating the databases is enormously
challenging (they're huge datasets in the cases of the big wikis) and
they're constantly changing. If you tried to rely on dumps alone, you'd
always be out of date by at least two weeks (assuming dumps are working
properly). Two weeks on the Internet is a lot of time.

But more to the point, even if you suddenly had a lot of infrastructure
(bandwidth for constantly retrieving the data, space to store it all, and
extra memory and CPU to allow users to, y'know, do something with it) and
even if you suddenly had staff capable of managing these databases, not
every table is in even available currently. As far as I'm aware,
http://dumps.wikimedia.org doesn't include tables such as "user",
"ipblocks", "archive", "watchlist", any tables related to global images or
global user accounts, and probably many others. I'm not sure a full audit
has ever been done, but this is partially tracked by
.

So beyond the silly simplicity of the suggestion that one could simply "move
to the cloud!", there are currently technical impossibilities to doing so.


It's the same impossibilities for forking any single CC project
online. We're not allowed by our privacy policy (and very likely by
law) to provide that information. It's absurd to fault us on this. I
guess we're being evil by not being evil.

We've providing every single other needed piece of the puzzle required
for forking.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Ryan Lane]

> Yes, there's a difference. But in this case, as far as I understand it, a
> direct cost (or casualty) of setting up Wikimedia Labs is the existence of
> the Toolserver. Does Wikimedia need a great testing infrastructure? Yes, of
> course. (And it's not as though the Toolserver has ever been without its
> share of issues; I'm not trying to white-wash the past here.) But the
> question is: if such a Wikimedia testing infrastructure comes at the cost of
> losing the Toolserver, is that acceptable?
>

This is a scarecrow argument. The mere existence of Labs doesn't mean
the loss of Toolserver.

Labs is more than just a testing infrastructure. It's an
infrastructure for creating things, for enable volunteer operations,
for bringing operations and development together, for integrating
other projects, and for providing free hosting to projects that may
not have it otherwise. Labs just also happens to need some of the same
features as Toolserver.

Again, as I've mentioned, Labs purpose isn't a Toolserver replacement.
It's vision is much, much larger than what the Toolserver can do.

> Ryan Lane wrote:
>> If WMF becomes evil, fork the entire infrastructure into EC2,
>> Rackspace cloud, HP cloud, etc. and bring the community operations
>> people along for the ride. Hell, use the replicated databases in Labs
>> to populate your database in the cloud.
>
> Tim Landscheidt wrote:
>> But the nice thing about Labs is that you can try out (re-
>> plicable :-)) replication setups at no cost, and don't have
>> to upfront investments on hardware, etc., so when time
>> comes, you can just upload your setup to EC2 or whatever and
>> have a working Wikipedia clone running in a manageable time-
>> frame.
>
> This is not an easy task. Replicating the databases is enormously
> challenging (they're huge datasets in the cases of the big wikis) and
> they're constantly changing. If you tried to rely on dumps alone, you'd
> always be out of date by at least two weeks (assuming dumps are working
> properly). Two weeks on the Internet is a lot of time.
>
> But more to the point, even if you suddenly had a lot of infrastructure
> (bandwidth for constantly retrieving the data, space to store it all, and
> extra memory and CPU to allow users to, y'know, do something with it) and
> even if you suddenly had staff capable of managing these databases, not
> every table is in even available currently. As far as I'm aware,
> http://dumps.wikimedia.org doesn't include tables such as "user",
> "ipblocks", "archive", "watchlist", any tables related to global images or
> global user accounts, and probably many others. I'm not sure a full audit
> has ever been done, but this is partially tracked by
> .
>
> So beyond the silly simplicity of the suggestion that one could simply "move
> to the cloud!", there are currently technical impossibilities to doing so.
>

It's the same impossibilities for forking any single CC project
online. We're not allowed by our privacy policy (and very likely by
law) to provide that information. It's absurd to fault us on this. I
guess we're being evil by not being evil.

We've providing every single other needed piece of the puzzle required
for forking.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ryan Lane]

> I thought the jenkins changes were ready, after Antoine wrote:
>> I have finished the preparation work in Jenkins. It will be deployed as
>> soon as we agree on the new workflow, which I still have to write down.
>

Nope. More issues were run into. We're now deploying OpenStack's zuul
framework to get this going. It needs backported packages, etc..
Either way, we're getting a little off topic :)

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 26/09/12 21:20, Ryan Lane wrote:
> Labs is an infrastructure for building infrastructure. It isn't a
> Toolserver replacement.
> 
> - Ryan

Yet at high level it has been treated as such replacement.
We could also replace the Wikimedia servers with turing machines.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 26/09/12 20:55, Ryan Lane wrote:
> They can be created in LDAP by making a labsconsole account.
> Additionally, unless the account needs to directly log in via ssh,
> there's no request process needed for the user to be used.
> 
> Of course right now there isn't open registration in labsconsole.
> We're waiting on some Jenkins changes for that to happen. Should be
> any day now.
> 
> Alternatively, the user could be created as a system account via puppet.
> 
> - Ryan

I thought the jenkins changes were ready, after Antoine wrote:
> I have finished the preparation work in Jenkins. It will be deployed as
> soon as we agree on the new workflow, which I still have to write down.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[MZMcBride]

Erik Moeller wrote:
> As others have noted, there's a difference between offering data
> (which we do - we've spent a lot of time, money and effort to ensure
> that stuff like dumps.wikimedia.org works reliably even at enwiki
> scale) and providing a working environment for the dev community.
> 
> Having a primary working environment like Labs makes sense in much the
> same way that it makes sense to have a primary multimedia repository
> like Commons (and Wikidata, and in future probably a gadget
> repository, a Lua script repository, etc.). It enables community
> network effects and economies of scale that can't easily be replicated
> and reduces wasteful duplication of effort.

Yes, there's a difference. But in this case, as far as I understand it, a
direct cost (or casualty) of setting up Wikimedia Labs is the existence of
the Toolserver. Does Wikimedia need a great testing infrastructure? Yes, of
course. (And it's not as though the Toolserver has ever been without its
share of issues; I'm not trying to white-wash the past here.) But the
question is: if such a Wikimedia testing infrastructure comes at the cost of
losing the Toolserver, is that acceptable?

Ryan Lane wrote:
> If WMF becomes evil, fork the entire infrastructure into EC2,
> Rackspace cloud, HP cloud, etc. and bring the community operations
> people along for the ride. Hell, use the replicated databases in Labs
> to populate your database in the cloud.

Tim Landscheidt wrote:
> But the nice thing about Labs is that you can try out (re-
> plicable :-)) replication setups at no cost, and don't have
> to upfront investments on hardware, etc., so when time
> comes, you can just upload your setup to EC2 or whatever and
> have a working Wikipedia clone running in a manageable time-
> frame.

This is not an easy task. Replicating the databases is enormously
challenging (they're huge datasets in the cases of the big wikis) and
they're constantly changing. If you tried to rely on dumps alone, you'd
always be out of date by at least two weeks (assuming dumps are working
properly). Two weeks on the Internet is a lot of time.

But more to the point, even if you suddenly had a lot of infrastructure
(bandwidth for constantly retrieving the data, space to store it all, and
extra memory and CPU to allow users to, y'know, do something with it) and
even if you suddenly had staff capable of managing these databases, not
every table is in even available currently. As far as I'm aware,
http://dumps.wikimedia.org doesn't include tables such as "user",
"ipblocks", "archive", "watchlist", any tables related to global images or
global user accounts, and probably many others. I'm not sure a full audit
has ever been done, but this is partially tracked by
.

So beyond the silly simplicity of the suggestion that one could simply "move
to the cloud!", there are currently technical impossibilities to doing so.

MZMcBride



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ryan Lane]

>> The database servers (replicated and user-databases) are going to be
>> provided by the Labs infrastructure. That's the majority of
>> Toolserver's servers. We have shared storage provided already.
>
> Let me count:
> -2 userland-server (linux)
> -1 userland-server (solaris)
> -2 web-servers (solaris)
> -1 web-serever (linux) (for testing)
> -2 HA-nodes
> -2 aux-servers
> -1 server for the roots
> -1 user-db-server
> -5 replicated db-servers
>
> 6 vs 11 – clearly the majority ;-)
>

This is more of a discussion of capacity. Excluding the databases, the
load of all of those servers would fit on one of our compute nodes as
virtual machines.

Basically what I'm saying is that based on our resources, we should
easily be able to handle the load from the TS community. If it turns
out that we can't, we'll add more capacity.

>>
>> The capacity of Labs is far greater than Toolserver's. We have a
>> cluster in the pmtpa and eqiad datacenters. Each cluster has 7 compute
>> nodes for virtual machines, 2 database servers for replicas, 1
>> database server for user databases and 4 storage nodes for shared
>> storage access.
>
> So you have 2 database servers for 7 wmf-clusters? You are poorer than we are.
>

Well, it's 2 per datacenter for replicas, and 1 per datacenter for
user-databases. That's 6. Also, these are new (and very beefy) servers
with SSDs. Moore's law and all. It could be that we need to add more
servers. If that's the case, we'll do so.

>> It's definitely possible to re-create Toolserver inside of Labs. I'm
>> not suggesting that's what should be done for a migration, but it's an
>> option.
>
> So who is helping me to install solaris? Who is helping me to port our self-
> written software over? Who is helping me to change the self-written scripts?
> Who is helping me to convert our jira-bugs into your bugzilla? Who will update
> all the wiki-pages?
>
> The toolserver is not just a simple Debian with a puppet-system running; its
> much more complex. And while I'm sure that I could re-build something similar
> in labs in a great time-window, it would just be something similar, not the
> same; many tools would break (if you have time, take a look in our JIRA for
> things that are not working on Debian, but on work on Solaris).
>
> Sometimes I have the feeling WMF and WMDE thinks that we can shutdown the TS
> on a Friday, rsync the stuff at the weekend, power-up labs at Monday and
> everything will work….
>

I understand there's a ton of work involved. I'm not suggesting that
it would be a simple task. I'm saying that it's technically possible
to build the same exact environment inside of Labs.

Labs is an infrastructure for building infrastructure. It isn't a
Toolserver replacement.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Ryan Lane]

>> MMPs aren't a very elegant solution. I'd prefer not to
>> force an inelegant solution into a system that allows much more
>> elegant approaches.
>
> Actually, I'm unsure on how to replicate MMP accounts there. It
> shouldn't need to manually create them everywhere. The elegant solution
> is probably to have those accounts in LDAP... Any ideas?
>

They can be created in LDAP by making a labsconsole account.
Additionally, unless the account needs to directly log in via ssh,
there's no request process needed for the user to be used.

Of course right now there isn't open registration in labsconsole.
We're waiting on some Jenkins changes for that to happen. Should be
any day now.

Alternatively, the user could be created as a system account via puppet.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Ryan Lane]

> As others have noted, there's a difference between offering data
> (which we do - we've spent a lot of time, money and effort to ensure
> that stuff like dumps.wikimedia.org works reliably even at enwiki
> scale) and providing a working environment for the dev community.
>
> Having a primary working environment like Labs makes sense in much the
> same way that it makes sense to have a primary multimedia repository
> like Commons (and Wikidata, and in future probably a gadget
> repository, a Lua script repository, etc.). It enables community
> network effects and economies of scale that can't easily be replicated
> and reduces wasteful duplication of effort.
>

I'd like to go a little further on this point.

One of the goals of Labs is to have a fully virtualized clone of our
entire infrastructure that is also completely puppetized in a way
that's reusable by third parties. If you're worried about WMF, then
you should participate in Labs. You should help puppetize and should
help make everything usable by non-WMF entities.

Bringing community operations members back into the operations of the
site is another one of the goals of Labs. If we have enough community
operations people, then the projects aren't dependent on the knowledge
of the staff to survive.

If WMF becomes evil, fork the entire infrastructure into EC2,
Rackspace cloud, HP cloud, etc. and bring the community operations
people along for the ride. Hell, use the replicated databases in Labs
to populate your database in the cloud.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Erik Moeller]

On Wed, Sep 26, 2012 at 10:15 AM, MZMcBride  wrote:
> I think I'd add "general direction of centralizing everything under a single
> Wikimedia Foundation is a bad idea" as a permanent blocker.

As others have noted, there's a difference between offering data
(which we do - we've spent a lot of time, money and effort to ensure
that stuff like dumps.wikimedia.org works reliably even at enwiki
scale) and providing a working environment for the dev community.

Having a primary working environment like Labs makes sense in much the
same way that it makes sense to have a primary multimedia repository
like Commons (and Wikidata, and in future probably a gadget
repository, a Lua script repository, etc.). It enables community
network effects and economies of scale that can't easily be replicated
and reduces wasteful duplication of effort.

That said, I'd love to make more real-time data feeds available for
third parties in general. The analytics team is currently looking into
offering a sensible alternative to the IRC feed for edit metadata, for
example.

Erik
-- 
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation

Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Daniel Schwen]

At the risk of outing myself as "naive": I do not see this as a
problem like MZMcBride does. I think the foundation should have earned
our trust by now and them locking down the data does not seem like a
credible threat to me.
In any case:

a) you can download dumps to access the data independently from WMF
b) the replication to the TS is already "at the mercy" of WMF. The TS
does not make the data any free-er.

Best,
Dschwen


> I think I'd add "general direction of centralizing everything under a single
> Wikimedia Foundation is a bad idea" as a permanent blocker. Maybe there's a
> reasonable case for why deprecating the Toolserver and creating Wikimedia
> Labs is a great idea, but I don't see it yet.
>
> I don't see why each (Wikimedia) chapter shouldn't have its own replica of
> the databases. We want free content to be free (and re-used and re-mixed and
> whatever else). If you're going to invest in infrastructure, I think it
> makes more sense to bolster replication support than try to compete with the
> Toolserver.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Ryan Lane]

> temporary blockers
> * no replication of wikimedia wiki databases
> ** joining of user databases with wiki databases

We currently have no plans for having the user databases on the same
servers as the replicated databases. Direct joins will not be
possible, so tools will need to be modified.

> * no support for script execution dependency (on ts: currently done by sge)

There's less of a need for this in Labs. If whatever you are running
is really expensive, you can have your own instance. That said, I was
looking at integrating a global queuing system. It won't be SGE,
though.

If someone is really keen on SGE, then I recommend they work with us
to puppetize it. Thankfully, open grid engine is already packaged in
ubuntu, which should make that much easier.

> * no support for servlets
>

I'm not sure what you mean by servlet?

> missing support blockers
> * no support for new users not familar with unix based systems

Can you describe how this is handled in Toolserver currently?

> * no transparent updating of packages with security problems/bug
>

Ubuntu has unattended-upgrades. It's generally enabled on instances.

> permanent blockers
> * license problems (i wrote code at work for my company and reuse parts for
> my bot framework. I have not the right to declare this code as open source
> which is needed by labs policy.)

This will continue to be a permanent blocker.

You can't decide that on your own, but you can ask your employer if
you can open source the code.

> * no DaB.
>

I'd love DaB to help us improve Labs.

Everything about Labs is fully open. Anyone can help build it, even
the production portions.

- Ryan

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[Tim Landscheidt]

(anonymous) wrote:

> [...]
> I think I'd add "general direction of centralizing everything under a single
> Wikimedia Foundation is a bad idea" as a permanent blocker. Maybe there's a
> reasonable case for why deprecating the Toolserver and creating Wikimedia
> Labs is a great idea, but I don't see it yet.

> I don't see why each (Wikimedia) chapter shouldn't have its own replica of
> the databases. We want free content to be free (and re-used and re-mixed and
> whatever else). If you're going to invest in infrastructure, I think it
> makes more sense to bolster replication support than try to compete with the
> Toolserver.

> That said, pooled resources can sometimes be a smart move to save on
> investments such as hardware. Chapters working together is not a bad thing
> (I believe some chapters donated to Wikimedia Deutschland for Toolserver
> support in the past). But the broader point is that users should be very
> cautious of the general direction that a Wikimedia (Foundation) Labs is
> headed and ask whether it's really a good idea iff it means the destruction
> of free-standing projects such as the Toolserver.

IMHO you have to differentiate between data and function.
It makes no sense to build artificial obstacles when setting
up some tool that can only be reasonably used with the live
dataset.  On the other hand, preparing for a day where WMF
turns rogue is never wrong.

  But the nice thing about Labs is that you can try out (re-
plicable :-)) replication setups at no cost, and don't have
to upfront investments on hardware, etc., so when time
comes, you can just upload your setup to EC2 or whatever and
have a working Wikipedia clone running in a manageable time-
frame.

Tim


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Reasons for not migrating to Tool Lab

[MZMcBride]

Merlissimo wrote:
> Perhaps it is useful to summarize reasons why toolserver users are not
> able to change to tool/bot labs. I added my main reasons. Perhaps other
> can add their reasons, too? (Mabe we should also add this list to the
> wiki page)
> 
> temporary blockers
> * no replication of wikimedia wiki databases
> ** joining of user databases with wiki databases
> * no support for script execution dependency (on ts: currently done by sge)
> * no support for servlets
> 
> missing support blockers
> * no support for new users not familar with unix based systems
> * no transparent updating of packages with security problems/bug
> 
> permanent blockers
> * license problems (i wrote code at work for my company and reuse parts
> for my bot framework. I have not the right to declare this code as open
> source which is needed by labs policy.)
> * no DaB.

I think I'd add "general direction of centralizing everything under a single
Wikimedia Foundation is a bad idea" as a permanent blocker. Maybe there's a
reasonable case for why deprecating the Toolserver and creating Wikimedia
Labs is a great idea, but I don't see it yet.

I don't see why each (Wikimedia) chapter shouldn't have its own replica of
the databases. We want free content to be free (and re-used and re-mixed and
whatever else). If you're going to invest in infrastructure, I think it
makes more sense to bolster replication support than try to compete with the
Toolserver.

That said, pooled resources can sometimes be a smart move to save on
investments such as hardware. Chapters working together is not a bad thing
(I believe some chapters donated to Wikimedia Deutschland for Toolserver
support in the past). But the broader point is that users should be very
cautious of the general direction that a Wikimedia (Foundation) Labs is
headed and ask whether it's really a good idea iff it means the destruction
of free-standing projects such as the Toolserver.

MZMcBride



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Daniel Schwen]

Another point to consider is that for some projects there is
considerable interdependence (this may not be the norm).
My WikiMiniAtlas for example depends on

1. the OSM database mirror being available
2. the WIWOSM project (although I could proxy that from the TS)
3. Dispensers coordinate extraction database (GHEL)

These dependencies will increase the viscosity of the migration process.
In theory database connections could be ssh tunneled although

1. that may not be a very stable or fast solution
2. someone/somewhat is blocking ssh traffic from labs to the
toolserver cluster (other ssh connections work fine) what is going on
here?!

Dschwen

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 26/09/12 12:40, Sumurai8 (DD) wrote:
> 2012/9/26 Platonides:
>> On 26/09/12 08:27, Federico Leva (Nemo) wrote:
 Yes, I know about MMP. It was introduced much later, after people were
 already very used to doing things in an individualized manner. From
 what I'm told MMPs aren't used much.
>>>
>>> So you think that even new users (I believe many users arrived after MMP
>>> were introduced) are not using MMP due to bad old habits of the other
>>> users?
>>> I've had a hard time trying to convince some users to use a MMP; I think
>>> it's not trivial to understand what models actually work and are liked.
>>
>> The reason is as simple as being much more easier, run mkdir and start
>> coding vs open a jira request to get a new MMP for a project which may
>> or may not be interesting (heh, most tools were probably born after
>> coding for a couple of hours after having a good idea) and which nobody
>> is wanting to maintain with you, probably.
>>
> 
> It is not hard to start coding in your own workspace and move code
> from your newly created directory to a MMP. A MMP doesn't necessarely
> have to be maintained by multiple users. While it's convenient having
> someone in the project that can jump in in case your tool breaks badly
> and you are not around to fix it, it is not by any means necessary.

Right. I was explaining why there aren't so many MMPs out there.


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

[Toolserver-l] Reasons for not migrating to Tool Lab

[Merlissimo]

Perhaps it is useful to summarize reasons why toolserver users are not 
able to change to tool/bot labs. I added my main reasons. Perhaps other 
can add their reasons, too? (Mabe we should also add this list to the 
wiki page)


temporary blockers
* no replication of wikimedia wiki databases
** joining of user databases with wiki databases
* no support for script execution dependency (on ts: currently done by sge)
* no support for servlets

missing support blockers
* no support for new users not familar with unix based systems
* no transparent updating of packages with security problems/bug

permanent blockers
* license problems (i wrote code at work for my company and reuse parts 
for my bot framework. I have not the right to declare this code as open 
source which is needed by labs policy.)

* no DaB.

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[MZMcBride]

Platonides wrote:
> On 25/09/12 20:48, Erik Moeller wrote:
>> 1) WMF is a technology organization. Hosting the core infrastructure
>> for Wikimedia projects is very much what we do. This includes data
>> center operation, monitoring and backups, software deployments,
>> software/service upgrades, code versioning infrastructure, bug
>> tracking infrastructure, additional support systems and services (like
>> this mailing list), etc.
>> 
>> Toolserver is in fact hosted by the Wikimedia Foundation today, in our
>> Amsterdam data-center. We provide space, power and racks for the
>> toolserver cluster, at a cost of about $65,000/year to WMF according
>> to our Director of TechOps.
> 
> Something we should all be grateful for.

I think the current Toolserver setup is less than ideal and I think the
future proposed setup (Tool Labs) is even worse. Right now there's already a
heavy reliance on the goodwill of the Wikimedia Foundation to keep the
Toolserver running. Without database replication, the Toolserver is just
mediocre shared hosting.

Going forward, the situation will worsen, as the Wikimedia Foundation is
basically creating a walled garden. We're watching as the Wikimedia
Foundation puts all of the data, tools, and infrastructure behind the same
organization and then will be able to determine who does and does not have
access to this and under what terms, a step backward as far as I'm
concerned.

Redundancy and duplication in this case is a very good thing, not a bad
thing. If we had ten Toolservers (hosted by Wikimedia chapters, Amazon,
LeaseWeb, or anyone else), it wouldn't be such an issue when database
replication stopped on one of them. It might not be as efficient, but it'd
be a much safer long-term strategy, rather than putting all of the
high-speed access to data (and in some cases, the _only_ access to data
[watchlist, anonymized user preferences, etc.]) behind the Wikimedia
Foundation's control.

MZMcBride



___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Andre Koopal]

On Wed, Sep 26, 2012 at 10:37:11AM +0200, Pavel Richter wrote:
> Hi Andre,
> 
> 
> 2012/9/26 Andre Koopal :
> > Just replying to the original mail here.
> >
> > While the discussion what the labs project should look like is usefull, the
> > immediate problem at hand is a bit forgotten, that toolserver might end
> > early next year if there is less support.
> 
> Toolserver will not end early next year. Period.
> 
Hi Pavel,

With all due respect, and I do believe your intentions are good, but if you
can't convince DaB about this and he does quit, toolserver will break and
stop. Although Nosy is still there, afaik she is still not up to date about
all the details around toolserver, so there will be a lot of knowledge lost
effectively ending toolserver.

Regards,

Andre

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[DaB.]

Hello Pavel,
At Wednesday 26 September 2012 12:31:52 DaB. wrote:
> With regards to new hardware: Our committment here dos not mean that
> we will simply replace broken parts only; it also means that we will
> buy new stuff, so that the Toolserver can handle increased demand. We
> do have the ressources to do so, and the money is available within our
> annual budget.

so you told me last year and how many servers did we bought this year (quick 
help: less than 1)? All new hardware (with the exception of a simple switch 
and some memory) that was installed this year was bought in 2011. And I told 
WMDE again and again that we are short on hardware. 
Why is it so hard to just add a position to your budget-plan

Toolserver-Hardware xx,000€

? If we will not need all of it in 2013: fine. I will not point at the number 
and demand that all will be spend. The TS will not need much money (2 or 3 new  
database-server would be enough), compared with other projects (like Wikidata, 
Render oder the community-project-budget) or the personal-cost (and you know I 
ALWAYS said that we need personal and it is ok to spend money there!) the 
position would be very small. I just need to be sure that there is money at 
all.

Sincerely,
DaB.


-- 
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885


signature.asc
Description: This is a digitally signed message part.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Sumurai8 (DD)]

2012/9/26 Platonides :
> On 26/09/12 08:27, Federico Leva (Nemo) wrote:
>>> Yes, I know about MMP. It was introduced much later, after people were
>>> already very used to doing things in an individualized manner. From
>>> what I'm told MMPs aren't used much.
>>
>> So you think that even new users (I believe many users arrived after MMP
>> were introduced) are not using MMP due to bad old habits of the other
>> users?
>> I've had a hard time trying to convince some users to use a MMP; I think
>> it's not trivial to understand what models actually work and are liked.
>
> The reason is as simple as being much more easier, run mkdir and start
> coding vs open a jira request to get a new MMP for a project which may
> or may not be interesting (heh, most tools were probably born after
> coding for a couple of hours after having a good idea) and which nobody
> is wanting to maintain with you, probably.
>

It is not hard to start coding in your own workspace and move code
from your newly created directory to a MMP. A MMP doesn't necessarely
have to be maintained by multiple users. While it's convenient having
someone in the project that can jump in in case your tool breaks badly
and you are not around to fix it, it is not by any means necessary.

>
>
> Ryan wrote:
>> Toolserver's model is fundamentally different. It's based on an old
>> concept of shared hosting. Labs is built on a model more like a VPS
>> (really more like EC2). Due to that, it's possible to give users far
>> more rights.
>
> labs model didn't exist when the toolserver started. This route was the
> only 'normal' one to follow, specially with a single server.
> It even looks too new for labs right now.
>
>
>
>> If you guys want to build the exact same Toolserver
>> environment as a Labs project, go for it. I have a good feeling you'll
>> start doing things differently when you see the affordances given by
>> having more rights, though.
>
> I have a few ideas on how to improve it on webtools, based on toolserver
> experience, but without big changes.
>
>> MMPs aren't a very elegant solution. I'd prefer not to
>> force an inelegant solution into a system that allows much more
>> elegant approaches.
>
> Actually, I'm unsure on how to replicate MMP accounts there. It
> shouldn't need to manually create them everywhere. The elegant solution
> is probably to have those accounts in LDAP... Any ideas?
>
>
> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[DaB.]

Hello,
At Wednesday 26 September 2012 12:07:17 DaB. wrote:
> > Even if
> > I would begin this very instance I doubt I would be finish in December
> > 2013; and it would need the same 18 servers (or better: more).
> 
> The database servers (replicated and user-databases) are going to be
> provided by the Labs infrastructure. That's the majority of
> Toolserver's servers. We have shared storage provided already.

Let me count:
-2 userland-server (linux)
-1 userland-server (solaris)
-2 web-servers (solaris)
-1 web-serever (linux) (for testing)
-2 HA-nodes
-2 aux-servers
-1 server for the roots
-1 user-db-server
-5 replicated db-servers

6 vs 11 – clearly the majority ;-)

> 
> The capacity of Labs is far greater than Toolserver's. We have a
> cluster in the pmtpa and eqiad datacenters. Each cluster has 7 compute
> nodes for virtual machines, 2 database servers for replicas, 1
> database server for user databases and 4 storage nodes for shared
> storage access.

So you have 2 database servers for 7 wmf-clusters? You are poorer than we are.


> 
> It's definitely possible to re-create Toolserver inside of Labs. I'm
> not suggesting that's what should be done for a migration, but it's an
> option.

So who is helping me to install solaris? Who is helping me to port our self-
written software over? Who is helping me to change the self-written scripts? 
Who is helping me to convert our jira-bugs into your bugzilla? Who will update 
all the wiki-pages?

The toolserver is not just a simple Debian with a puppet-system running; its 
much more complex. And while I'm sure that I could re-build something similar 
in labs in a great time-window, it would just be something similar, not the 
same; many tools would break (if you have time, take a look in our JIRA for 
things that are not working on Debian, but on work on Solaris).

Sometimes I have the feeling WMF and WMDE thinks that we can shutdown the TS 
on a Friday, rsync the stuff at the weekend, power-up labs at Monday and 
everything will work….

> 
> - Ryan

Sincerely,
DaB.


-- 
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885


signature.asc
Description: This is a digitally signed message part.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[DaB.]

Hello,
At Wednesday 26 September 2012 12:18:12 DaB. wrote:
> (users LOVES joins)

while I'm at joing: We also need a live copy of s4 (commons) on every wmf-db-
cluster (for the image- and commons-people).

Sincerely,
DaB.

-- 
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885


signature.asc
Description: This is a digitally signed message part.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[DaB.]

Hello,
At Wednesday 26 September 2012 11:54:20 DaB. wrote:
> Access to the wmf wiki replicated dbs is public across the entire wmflabs
> network so that's a given within the toollabs project as well.

so the entire labs-cluster use the same database-servers? Do you know that we 
have users that run queries that took hours/days to complete (because there 
are complex, for academic research or just because there is bug in the query)?
And while I read about user-databases in another mail: While the toolserver 
has a dedicated sql-server for that (which REALLY needs 
replacement/extension), many user-databases are on the replicated-db-servers 
(users LOVES joins); some are hundreds of MB big (you can imagine how many 
rows they have). Does labs support that too? 

Sincerely,
DaB.


-- 
Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885


signature.asc
Description: This is a digitally signed message part.
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Pavel Richter]

Hi Andre,


2012/9/26 Andre Koopal :
> Just replying to the original mail here.
>
> While the discussion what the labs project should look like is usefull, the
> immediate problem at hand is a bit forgotten, that toolserver might end
> early next year if there is less support.

Toolserver will not end early next year. Period.

Wikimedia Deutschland will make all necessary investments to keep the
Toolserver up and running, and we will use the next year to discuss
together with the Toolserver users and community a way to transition
into WikiLabs. And I can only urge you all to take active part in the
design and development of Wikilabs, so that it will provide for your
needs.

With regards to new hardware: Our committment here dos not mean that
we will simply replace broken parts only; it also means that we will
buy new stuff, so that the Toolserver can handle increased demand. We
do have the ressources to do so, and the money is available within our
annual budget.
Please support DaB and Sebastian Sooth (who is handling Toolserver
matters from Wikimedia Deutschland side) in putting together a list of
necessary items to be purchased.

What we will not do, however, is ignoring the fact that the Toolserver
will be replaced by Wikilabs sometime in the future. So every
investment we make will have to take this into account.

Pavel

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Platonides]

On 26/09/12 08:27, Federico Leva (Nemo) wrote:
>> Yes, I know about MMP. It was introduced much later, after people were
>> already very used to doing things in an individualized manner. From
>> what I'm told MMPs aren't used much.
> 
> So you think that even new users (I believe many users arrived after MMP
> were introduced) are not using MMP due to bad old habits of the other
> users?
> I've had a hard time trying to convince some users to use a MMP; I think
> it's not trivial to understand what models actually work and are liked.

The reason is as simple as being much more easier, run mkdir and start
coding vs open a jira request to get a new MMP for a project which may
or may not be interesting (heh, most tools were probably born after
coding for a couple of hours after having a good idea) and which nobody
is wanting to maintain with you, probably.



Ryan wrote:
> Toolserver's model is fundamentally different. It's based on an old
> concept of shared hosting. Labs is built on a model more like a VPS
> (really more like EC2). Due to that, it's possible to give users far
> more rights. 

labs model didn't exist when the toolserver started. This route was the
only 'normal' one to follow, specially with a single server.
It even looks too new for labs right now.



> If you guys want to build the exact same Toolserver
> environment as a Labs project, go for it. I have a good feeling you'll
> start doing things differently when you see the affordances given by
> having more rights, though. 

I have a few ideas on how to improve it on webtools, based on toolserver
experience, but without big changes.

> MMPs aren't a very elegant solution. I'd prefer not to
> force an inelegant solution into a system that allows much more
> elegant approaches.

Actually, I'm unsure on how to replicate MMP accounts there. It
shouldn't need to manually create them everywhere. The elegant solution
is probably to have those accounts in LDAP... Any ideas?


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Andre Koopal]

Just replying to the original mail here.

While the discussion what the labs project should look like is usefull, the
immediate problem at hand is a bit forgotten, that toolserver might end
early next year if there is less support.

As others stated, as some of the needed functionality is not present in
labs, having an environment usefull for the current toolserver community
will likely take till end of the year. To then have the community migrate
there, setup the things there, perhaps migrate data, and perhaps change
urls all over the place, will take some time already, so that is likely
middle till end of 2014.

So we will need toolserver up and running for a while, and that will take
investments. And that is the problem that will need immediate attention
at the moment.

Regards,

Andre

On Tue, Sep 25, 2012 at 12:51:50AM +0200, DaB. wrote:
> Hello all,
> 
> in these days WMDE (the chapter that finance the toolserver) is discussing 
> the 
> budget for the next year (2013); you can find it at [1]. At the moment there 
> is 
> no money for new toolserver-hardware in this budget and the CEO Pavel Richter 
> is unwilling to change this ([2] in german) – because he fears that there 
> will 
> be a Wikilabs in 2014. It is not possible for me to run the toolserver for 
> another year with the current hardware – you all know why. For this reason I 
> will request a change of the budget at the general meeting at November, so 
> there will be a vote about. If this vote should fail (and we get no money for 
> new hardware), I am going to retire from my job as root at 30. December 2012.
> I'm not longer able to tolerant the behavior of the german chapter and the 
> WMF 
> in matter of the toolserver; I do this for free and for fun, and it is not 
> longer fun.
> 
> Sincerely,
> DaB.
> 
> P.S: If you are in a board of a chapter that gives money to WMDE for the 
> toolserver: Make sure that it will be spend for hardware. 
> 
> [1] 
> http://meta.wikimedia.org/wiki/Wikimedia_Deutschland/2013_annual_plan_draft/en
> [2] 
> meta.wikimedia.org/wiki/Talk:Wikimedia_Deutschland/2013_annual_plan_draft/de#Toolserver
> 
> -- 
> Userpage: [[:w:de:User:DaB.]] — PGP: 2B255885



> ___
> Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/toolserver-l
> Posting guidelines for this list: 
> https://wiki.toolserver.org/view/Mailing_list_etiquette


___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette

Re: [Toolserver-l] Future of the toolserver

[Andre Koopal]

On Wed, Sep 26, 2012 at 01:59:51AM -0400, Ryan Lane wrote:
> 
> Of course, as I mentioned before, there's nothing forcing this new
> model at all. If you guys want to build the exact same Toolserver
> environment as a Labs project, go for it. I have a good feeling you'll
> start doing things differently when you see the affordances given by
> having more rights, though. Either path chosen, we'll help you with
> infrastructure issues along the way.
> 
> - Ryan
> 
Hi Ryan,

This statement has been made before, but then you put all responsibility
with the community, while the foundation can benefit as well. What people
want and has appeared usefull, is a *supported* environment like toolserver
where people can just get an account, run bots to support a wiki, experiment
with simple tools, if needed work and support a tool together. This can
then be a breeding bed for tools, and if needed brought a stage up, be
documented and pacakged, so it can be rolled out as a wmf supported tool.

That the toolserver environment is supported is imho an essential. While
volunteer admins can certainly help and do a lot of work, including helping
non-unix persons to startup, if there are really problems with the
environment, it better guaranties there is somebody available to acct.

Regards,

Andre Koopal

___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list: 
https://wiki.toolserver.org/view/Mailing_list_etiquette