Re: [Toolserver-l] Toolserver limitation to come?
Hey, On Wed, 1 May 2013, Ryan Lane wrote: On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt t...@tim-landscheidt.dewrote: | [...] There were never answers to this, so I bring it up here again: 1. How were almost /all/ of the problems the TS have had with replication caused by that redundancy and trying to keep it synced? Thanks for this question :) - I also want to know. From my perspective it does not look like this and even the data inconsistencies appear when we have no commons copy on a mysql instance. And: DaB experimented with federated tables for commons too and we decided to not do this since it does not perform from the start. Probably nowadays when I planned something new in this area (which does not seem to make sense for TS) I'd really give Galera a try - http://codership.com/content/using-galera-cluster 2. What limitation will the Toolserver have at some point? As to #2: From what I've been told this has to do with future sharding plans for the databases, and due to a change in how we'll be doing replication. Of course, I've heard this in passing. For answers to both of these questions you'll need to talk to binasher and/or notpeter on IRC, as they are the ones doing the database work. Thanks for telling... Cheers Marlen/nosy ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver -- Labs
(anonymous) wrote: [...] Database replication came up yet again in the office hours. Many developers (myself included) seem to be holding off on Labs until database replication is up and running. The sooner this can happen, the better. But the remaining sticking point seems to be cross-database joins, which people in the office hours suggested using federated tables or application logic to replace. It would help if the Labs folks could better explain _why_ cross-database joins won't be supported (I think most developers would agree with the reasoning) and offer better guidance and documentation for how to work around this hurdle. (For example, what is a federated table?) [...] The problem with this decision is the effort spent and the insincerity. If database replication for Labs would have meant moving some dbxxx servers to labsdbxxx, adding the views existing on Toolserver and tightening some firewall rules, it could have been set up in a month, and any moaning about having to use federated tables would have been si- lenced by the minutes it would take to add another server to the cluster to increase performance compared to the years it takes at Toolserver. Or there could have been some new concept like Galera men- tioned by Nosy that eases maintenance because it is not some sparsely documented Solaris thingy in River style. But now the plan is to have two clusters (PreLabsDBDBS and LabsDB), use triggers to remove data and then (addition- ally!) views, end up with less functionality than the Toolserver while gaining nothing, and all that takes half a year to set up in a cloak-and-dagger way while publicly the need for cross-database JOINs is acknowledged. Tim ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Toolserver limitation to come?
On 02/05/13 10:03, Marlen Caemmerer wrote:
Hey,
On Wed, 1 May 2013, Ryan Lane wrote:
On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt
t...@tim-landscheidt.dewrote:
There were never answers to this, so I bring it up here
again:
1. How were almost /all/ of the problems the TS have had
with replication caused by that redundancy and trying
to keep it synced?
Thanks for this question :) - I also want to know.
From my perspective it does not look like this and even the data
inconsistencies appear when we have no commons copy on a mysql instance.
And: DaB experimented with federated tables for commons too and we
decided to not do this since it does not perform from the start.
Probably nowadays when I planned something new in this area (which does
not seem to make sense for TS) I'd really give Galera a try -
http://codership.com/content/using-galera-cluster
FWIW, my {{ref needed}} phrase in the log was also intended to that
statement by Coren, not to multichill reply.
Additionally, I rembember you data inconsistencies happen even with
native mysql replication (as informed by Nosy earlier).
___
Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/toolserver-l
Posting guidelines for this list:
https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Survey: Moving to Labs
Hello, At Thursday 02 May 2013 15:29:07 DaB. wrote: I'm confused. I thought we were all here to support the readers, editors, researchers and developers of the Wikimedia projects? If the toolserver is empty because Labs is accomplishing the goal, isn't that a good thing? I've asked this before: why not help with Labs, rather than fighting everyone? Let's work as a team do not forget who started the fighting: The WMF. The WMF announced to WMDE that the database-replication is going to end in the near future, what caused that WMDE stopped to support the Toolserver properly. The very goal with this was to let (Tool-)Labs be the only alternative. A fair approach would have been to create Labs as an alternative to the Toolserver, letting the users (new and old) decide which system they want to use. Toolserver and Labs could have existed in coexistence, exchanging knowledge, and maybe specially in different fields after a while. But that was not what happened. Instead the WMF decided because the are bigger, have more money, servers and personal, and control the replication-data, that they just could put the toolserver to an end – what didn't work as well as expected. And now we are sitting here with confused tool-authors, annoyed tool-users and a angry root. I didn't start the fight and I am not interested in teaming-up with a party which was not interested to build a team in the very beginning when it counted. Switching or helping with Labs would signal that I'm fine with all what the WMF did – and I'm not. Sincerely, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 0x2d3ee2d42b255885 signature.asc Description: This is a digitally signed message part. ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] Cronie error
I have a cronietab job that now gives this error message: error: commlib error: can't connect to service (Connection refused) Unable to run job: unable to send message to qmaster using port 444 on host damiana: got send error. Exiting. What does that mean, and how can I fix it? I have no idea what commlib or damiana are, but perhaps they are related to qsub? The command I tried to run (from cronie) was: qsub -N links-s7 -l sql-s7-user-readonly=1 -l h_rt=11:00:00 -l virtual_free=250M -l user_slot=5 linkstats.sh 7 Where linkstats.sh is a shell script in ~la2/ -- Lars Aronsson (l...@aronsson.se) Aronsson Datateknik - http://aronsson.se ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Survey: Moving to Labs
Hey folks, I usually steer clear of these sort of battles, but it looks like it's time to state the obvious: *we need to work together better*. We're wiki people, damn it. We're the people[1] who figured out how to build an encyclopedia through (effectively) an anonymous system when those with less imagination were skeptical what it could even work at all. Now, we're fighting against ourselves about technology to support our wiki work and it is only wasting time, energy and social capitol. DaB, I don't follow toolserver-l as well as I should. What can I do to help make sure that the Toolserver cluster is well supplied *at least* until labs meets 99.9% of tool developers needs. Do I need to lobby the WMF? WMDE? Ryan, I'm sure it was not out of some sort of malicious intent, but a large number of toolserver users and especially DaB are getting a raw deal. At some point, someone seems to have suggested that WMF Labs ought to replace the Toolserver. This is painful because, while Labs is not yet ready for us, the Toolserver is already being phased out. It's not fair to just say, Come on over to Labs and help us. I don't see how jumping ship before the next one shows up is a good idea. The majority of us are doing our work as volunteers. We can't just manifest extra maintainer hours in order to spend developer time on Labs. We're already spending more time dealing with Toolserver issues than we normally would. Finally, the Toolserver isn't just a resource. It's our community. A community is far more valuable than technology. If we don't preserve our community, we'll all lose. So please, when we're fighting each other, our first thought should be how to not need to fight anymore. So here we are. Today was wasted arguing about who was wronged. How do we work together better tomorrow? -Aaron On Thu, May 2, 2013 at 9:05 AM, DaB. w...@daniel.baur4.info wrote: Hello, At Thursday 02 May 2013 15:29:07 DaB. wrote: I'm confused. I thought we were all here to support the readers, editors, researchers and developers of the Wikimedia projects? If the toolserver is empty because Labs is accomplishing the goal, isn't that a good thing? I've asked this before: why not help with Labs, rather than fighting everyone? Let's work as a team do not forget who started the fighting: The WMF. The WMF announced to WMDE that the database-replication is going to end in the near future, what caused that WMDE stopped to support the Toolserver properly. The very goal with this was to let (Tool-)Labs be the only alternative. A fair approach would have been to create Labs as an alternative to the Toolserver, letting the users (new and old) decide which system they want to use. Toolserver and Labs could have existed in coexistence, exchanging knowledge, and maybe specially in different fields after a while. But that was not what happened. Instead the WMF decided because the are bigger, have more money, servers and personal, and control the replication-data, that they just could put the toolserver to an end – what didn't work as well as expected. And now we are sitting here with confused tool-authors, annoyed tool-users and a angry root. I didn't start the fight and I am not interested in teaming-up with a party which was not interested to build a team in the very beginning when it counted. Switching or helping with Labs would signal that I'm fine with all what the WMF did – and I'm not. Sincerely, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 0x2d3ee2d42b255885 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] OFFLIST Re: Survey: Moving to Labs
Thank you, more than I can say. -Sumana On 05/02/2013 11:20 AM, Aaron Halfaker wrote: Hey folks, I usually steer clear of these sort of battles, but it looks like it's time to state the obvious: *we need to work together better*. We're wiki people, damn it. We're the people[1] who figured out how to build an encyclopedia through (effectively) an anonymous system when those with less imagination were skeptical what it could even work at all. Now, we're fighting against ourselves about technology to support our wiki work and it is only wasting time, energy and social capitol. DaB, I don't follow toolserver-l as well as I should. What can I do to help make sure that the Toolserver cluster is well supplied *at least* until labs meets 99.9% of tool developers needs. Do I need to lobby the WMF? WMDE? Ryan, I'm sure it was not out of some sort of malicious intent, but a large number of toolserver users and especially DaB are getting a raw deal. At some point, someone seems to have suggested that WMF Labs ought to replace the Toolserver. This is painful because, while Labs is not yet ready for us, the Toolserver is already being phased out. It's not fair to just say, Come on over to Labs and help us. I don't see how jumping ship before the next one shows up is a good idea. The majority of us are doing our work as volunteers. We can't just manifest extra maintainer hours in order to spend developer time on Labs. We're already spending more time dealing with Toolserver issues than we normally would. Finally, the Toolserver isn't just a resource. It's our community. A community is far more valuable than technology. If we don't preserve our community, we'll all lose. So please, when we're fighting each other, our first thought should be how to not need to fight anymore. So here we are. Today was wasted arguing about who was wronged. How do we work together better tomorrow? -Aaron On Thu, May 2, 2013 at 9:05 AM, DaB. w...@daniel.baur4.info wrote: Hello, At Thursday 02 May 2013 15:29:07 DaB. wrote: I'm confused. I thought we were all here to support the readers, editors, researchers and developers of the Wikimedia projects? If the toolserver is empty because Labs is accomplishing the goal, isn't that a good thing? I've asked this before: why not help with Labs, rather than fighting everyone? Let's work as a team do not forget who started the fighting: The WMF. The WMF announced to WMDE that the database-replication is going to end in the near future, what caused that WMDE stopped to support the Toolserver properly. The very goal with this was to let (Tool-)Labs be the only alternative. A fair approach would have been to create Labs as an alternative to the Toolserver, letting the users (new and old) decide which system they want to use. Toolserver and Labs could have existed in coexistence, exchanging knowledge, and maybe specially in different fields after a while. But that was not what happened. Instead the WMF decided because the are bigger, have more money, servers and personal, and control the replication-data, that they just could put the toolserver to an end – what didn't work as well as expected. And now we are sitting here with confused tool-authors, annoyed tool-users and a angry root. I didn't start the fight and I am not interested in teaming-up with a party which was not interested to build a team in the very beginning when it counted. Switching or helping with Labs would signal that I'm fine with all what the WMF did – and I'm not. Sincerely, DaB. -- Userpage: [[:w:de:User:DaB.]] — PGP: 0x2d3ee2d42b255885 ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Survey: Moving to Labs
Since I evidently fail at trying to keep my thank-yous offlist: Aaron, thanks for the note. I regret that DaB. and other members of the Toolserver community have gotten changed timelines and confused and changing messages around tools support. It sounds like DaB. and other Toolserver community members are still smarting from some past miscommunications and the feeling of having something taken away from them. I am sorry for those past problems. What we all want to do is work to provide strong, well-supported places for our community to make and host bots and tools -- and WMF and WMDE have put a bunch more effort into that goal in the last half-year or so, to avoid a repeat of past problems. (I especially appreciate the work by Silke and Coren on this, just to shout out.) I'm grateful for the work DaB. has done in the past, and I think the survey data is pretty useful to help us see how to move forward -- it sounds like we'll have to contact Mono once more stuff is set up on Labs to help with the move. :) DaB. said: Switching or helping with Labs would signal that I'm fine with all what the WMF did – and I'm not. I don't think other people would read cooperation that way; I think most of us collaborate on projects where we aren't 100% in agreement with all the decisions our colleagues made, and we can balance disagreeing and working together. I hope to work with you. with regards, Sumana -- Sumana Harihareswara Engineering Community Manager Wikimedia Foundation On 05/02/2013 11:24 AM, Sumana Harihareswara wrote: Thank you, more than I can say. -Sumana On 05/02/2013 11:20 AM, Aaron Halfaker wrote: Hey folks, I usually steer clear of these sort of battles, but it looks like it's time to state the obvious: *we need to work together better*. We're wiki people, damn it. We're the people[1] who figured out how to build an encyclopedia through (effectively) an anonymous system when those with less imagination were skeptical what it could even work at all. Now, we're fighting against ourselves about technology to support our wiki work and it is only wasting time, energy and social capitol. DaB, I don't follow toolserver-l as well as I should. What can I do to help make sure that the Toolserver cluster is well supplied *at least* until labs meets 99.9% of tool developers needs. Do I need to lobby the WMF? WMDE? Ryan, I'm sure it was not out of some sort of malicious intent, but a large number of toolserver users and especially DaB are getting a raw deal. At some point, someone seems to have suggested that WMF Labs ought to replace the Toolserver. This is painful because, while Labs is not yet ready for us, the Toolserver is already being phased out. It's not fair to just say, Come on over to Labs and help us. I don't see how jumping ship before the next one shows up is a good idea. The majority of us are doing our work as volunteers. We can't just manifest extra maintainer hours in order to spend developer time on Labs. We're already spending more time dealing with Toolserver issues than we normally would. Finally, the Toolserver isn't just a resource. It's our community. A community is far more valuable than technology. If we don't preserve our community, we'll all lose. So please, when we're fighting each other, our first thought should be how to not need to fight anymore. So here we are. Today was wasted arguing about who was wronged. How do we work together better tomorrow? -Aaron On Thu, May 2, 2013 at 9:05 AM, DaB. w...@daniel.baur4.info wrote: Hello, At Thursday 02 May 2013 15:29:07 DaB. wrote: I'm confused. I thought we were all here to support the readers, editors, researchers and developers of the Wikimedia projects? If the toolserver is empty because Labs is accomplishing the goal, isn't that a good thing? I've asked this before: why not help with Labs, rather than fighting everyone? Let's work as a team do not forget who started the fighting: The WMF. The WMF announced to WMDE that the database-replication is going to end in the near future, what caused that WMDE stopped to support the Toolserver properly. The very goal with this was to let (Tool-)Labs be the only alternative. A fair approach would have been to create Labs as an alternative to the Toolserver, letting the users (new and old) decide which system they want to use. Toolserver and Labs could have existed in coexistence, exchanging knowledge, and maybe specially in different fields after a while. But that was not what happened. Instead the WMF decided because the are bigger, have more money, servers and personal, and control the replication-data, that they just could put the toolserver to an end – what didn't work as well as expected. And now we are sitting here with confused tool-authors, annoyed tool-users and a angry root. I didn't start the fight and I am not interested in teaming-up with a party which was not interested to build a
Re: [Toolserver-l] Toolserver limitation to come?
On May 2, 2013 3:08 AM, Marlen Caemmerer marlen.caemme...@wikimedia.de wrote: Hey, On Wed, 1 May 2013, Ryan Lane wrote: On Wed, May 1, 2013 at 5:03 PM, Tim Landscheidt t...@tim-landscheidt.de wrote: | [...] There were never answers to this, so I bring it up here again: 1. How were almost /all/ of the problems the TS have had with replication caused by that redundancy and trying to keep it synced? Ryan, repeaters are from the root of a program inwhich start the initial setup. Thanks for this question :) - I also want to know. From my perspective it does not look like this and even the data inconsistencies appear when we have no commons copy on a mysql instance. And: DaB experimented with federated tables for commons too and we decided to not do this since it does not perform from the start. Probably nowadays when I planned something new in this area (which does not seem to make sense for TS) I'd really give Galera a try - http://codership.com/content/using-galera-cluster 2. What limitation will the Toolserver have at some point? As to #2: From what I've been told this has to do with future sharding plans for the databases, and due to a change in how we'll be doing replication. Of course, I've heard this in passing. For answers to both of these questions you'll need to talk to binasher and/or notpeter on IRC, as they are the ones doing the database work. Thanks for telling... Cheers Marlen/nosy ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
Re: [Toolserver-l] Survey: Moving to Labs
On Thu, May 2, 2013 at 8:20 AM, Aaron Halfaker aaron.halfa...@gmail.comwrote: Hey folks, I usually steer clear of these sort of battles, but it looks like it's time to state the obvious: *we need to work together better*. We're wiki people, damn it. We're the people[1] who figured out how to build an encyclopedia through (effectively) an anonymous system when those with less imagination were skeptical what it could even work at all. Now, we're fighting against ourselves about technology to support our wiki work and it is only wasting time, energy and social capitol. DaB, I don't follow toolserver-l as well as I should. What can I do to help make sure that the Toolserver cluster is well supplied *at least* until labs meets 99.9% of tool developers needs. Do I need to lobby the WMF? WMDE? Ryan, I'm sure it was not out of some sort of malicious intent, but a large number of toolserver users and especially DaB are getting a raw deal. At some point, someone seems to have suggested that WMF Labs ought to replace the Toolserver. This is painful because, while Labs is not yet ready for us, the Toolserver is already being phased out. It's not fair to just say, Come on over to Labs and help us. I don't see how jumping ship before the next one shows up is a good idea. The majority of us are doing our work as volunteers. We can't just manifest extra maintainer hours in order to spend developer time on Labs. We're already spending more time dealing with Toolserver issues than we normally would. I don't think it's necessary to lobby anyone. WMDE has agreed to continue funding TS during the transition period, and the final decommissioning date is 2014-12-31 (see http://www.mediawiki.org/wiki/Tool_Labs/Roadmap_en). Finally, the Toolserver isn't just a resource. It's our community. A community is far more valuable than technology. If we don't preserve our community, we'll all lose. So please, when we're fighting each other, our first thought should be how to not need to fight anymore. That's what I'm encouraging as well. I'd like the community to move intact to Labs, including DaB. Fighting each other won't get us anywhere. If we all work together to build the new environment, it'll go quicker and we'll resolve issues together. - Ryan ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
[Toolserver-l] SGE job queue availability and execution
I've noticed some irregularity in job execution through SGE over the past few days. Currently it seems several queues are either disabled or in an error state. Is this expected? Is there an easy way to get an idea about how many jobs are queued and how quickly they're executed, in other words how to predict when a certain job might be run? Or maybe this is just a temporary issue that'll get resolved shortly? Cheers, Morten ___ Toolserver-l mailing list (Toolserver-l@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette
