[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 --- Comment #11 from Antoine hashar Musso has...@free.fr --- When we migrated the beta cluster from pmtpa to eqiad, Sean Pringle added a master / slave setup on beta. Apparently the slave is usually laggy. It seems to be possible to make it always lagged. Someone can reach out with Sean to figure out how to make it happen. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 --- Comment #12 from Niklas Laxström niklas.laxst...@gmail.com --- According to my latest knowledge the replication delay setting only exists in recent MySQL [1] and not in MariaDB - unless the feature has been added recently. [1] I created a three server setup with replication manually. Unfortunately it did not survive an upgrade so it was broken before we got the chance to use it. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 --- Comment #13 from Antoine hashar Musso has...@free.fr --- That might depends on the percona toolkit + some custom setup. I think production has slaves which have a 24 hours delay. Someone should talk about it with Sean Pringle. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 --- Comment #15 from Sean Pringle sprin...@wikimedia.org --- For comment #14: [1] https://git.wikimedia.org/blob/operations%2Fsoftware/ccbfe07353ce100bfe7d8557f9ceeeb28103d4be/dbtools%2Fevents_dbstore_delayed.sql#L64 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 --- Comment #14 from Sean Pringle sprin...@wikimedia.org --- Production has slaves delayed by 24h using the MariaDB event scheduler [1] to start/stop the replication threads. This is fine for a coarse lag values of a few minutes, but inaccurate for anything less. The MySQL 5.6 CHANGE MASTER TO MASTER_DELAY = N; (seconds) can be more accurate, roughly ~10s, but still highly dependent on the traffic generating the replicated events. Have also not seen it in action on our traffic, so... pinch of salt. A series of 10+ second writes such as our periodic bot update/delete traffic on recentchanges or links can confuse both methods for short delays, with lag cycling between 0 and N*2. It might be possible to achieve finer granularity on beta slave by interleaving something like FLUSH TABLES WITH READ LOCK on another thread (or another event) to ensure the slave thread does not catch up so easily. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 Andre Klapper aklap...@wikimedia.org changed: What|Removed |Added See Also||https://bugzilla.wikimedia. ||org/show_bug.cgi?id=65394 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 Nemo federicol...@tiscali.it changed: What|Removed |Added Keywords||ops, performance CC||federicol...@tiscali.it -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 Siebrand Mazeland s.mazel...@xs4all.nl changed: What|Removed |Added Blocks||1268 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 Faidon Liambotis fai...@wikimedia.org changed: What|Removed |Added CC||o...@wikimedia.org --- Comment #10 from Faidon Liambotis fai...@wikimedia.org --- I doubt it but I know little about Vagrant; adding Ori to the loop. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 --- Comment #9 from Niklas Laxström niklas.laxst...@gmail.com --- Labs would be nice, but something that allows debugging and tweaking of the code would be even nicer. I wonder if it would be possible to do this with MediaWiki-Vagrant. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 Siebrand Mazeland s.mazel...@xs4all.nl changed: What|Removed |Added Blocks||51731, 54579 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 Siebrand Mazeland s.mazel...@xs4all.nl changed: What|Removed |Added CC||amir.ahar...@mail.huji.ac.i ||l, has...@free.fr, ||sprin...@wikimedia.org --- Comment #8 from Siebrand Mazeland s.mazel...@xs4all.nl --- Adding some information from an email by Sean: Nothing built into MariaDB 5.5, but Percona Toolkit has a decent tool: http://www.percona.com/doc/percona-toolkit/2.2/pt-slave-delay.html However it will depend on how accurate a delay is needed to be useful. The tool starts and stops the replication SQL thread predictably but the minimum time granularity is one transaction, which fluctuates, obviously. Essentially a delay in the order of minutes is easy to maintain. Seconds... sort of. Oracle's MySQL 5.6 has slave delay built-in using CHANGE MASTER TO MASTER_DELAY = sconds. The next MariaDB major relase may get that port -- havn't checked -- but that doesn't help us today. - the DB are Ubuntu Lucid instances with MySQL installed manually (aka no puppet class applied) Ubuntu has percona toolkit packages in our repos. At least coredb have them installed by default. Only depends on perl. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 Niklas Laxström niklas.laxst...@gmail.com changed: What|Removed |Added Blocks||46716 -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 --- Comment #7 from Krinkle krinklem...@gmail.com --- Right, so if I understand you correctly, you're looking for an environment where you can work on fixing bugs and testing bugfixes related to replication lag. In other words, a wiki (say, lagged.wikipedia.beta.wmflabs.org) to do things with (as a human being). Not a build step for continuous integration environment. Not a test suite for MediaWiki core. If so, lets move this to as a feature request for labs. To set up a wiki there that is artificially lagged. -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 --- Comment #6 from Niklas Laxström niklas.laxst...@gmail.com --- I was more concerned about actually reproducing the issues reliably and having possibility to debug them easily to understand the causes and to also to come up and test fixes without going through gerrit. I'm doubtful that you can do that with QA tests. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
[Bug 38945] Need a way to simulate replication lag to test replag issues
https://bugzilla.wikimedia.org/show_bug.cgi?id=38945 Krinkle krinklem...@gmail.com changed: What|Removed |Added Summary|Way to simulate replication |Need a way to simulate |lag issues |replication lag to test ||replag issues --- Comment #5 from Krinkle krinklem...@gmail.com --- Filed under continuous integration for now. Depending on the kind of issues you want to test for and how it is implemented, it may be more suitable to have QA test this from the outside instead of with PHPUnit from Jenkins. -- You are receiving this mail because: You are the assignee for the bug. You are watching all bug changes. ___ Wikibugs-l mailing list Wikibugs-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
