[Analytics] s1-analytics-slave under the weather

Hi! s1-analytics-slave has been struggling recently (SUL finalization load, plus some other stuff). I've had to pause EventLogging replication there tonight in order to let S1 catch up, as well as do some table maintenance. I estimate 24 hours impact. analytics-store is not affected. BR Sean _

Re: [Analytics] s1-analytics-slave

Hi Sean, no objection on my end either. I’ll have to update a bunch of scripts that populate the EE dashboards [1] but it’s no big deal as long as we clearly communicate the ETA. [1] http://ee-dashboard.wmflabs.org/dashboards/enwiki-metrics

Re: [Analytics] s1-analytics-slave

So, bump :-) - A week's notice would be needed for Halfak to vacate s1-analytics-slave, and presumably others could meet the same target. Or make it a month since there is no desperate rush. - Geowiki needs some coordination for the switchover to analytics-store staging db. Anything else? __

Re: [Analytics] s1-analytics-slave

Hi, just to keep archives happy, it seems the below message saw an out-of-thread reply at https://lists.wikimedia.org/pipermail/analytics/2015-February/003328.html and is continued there. Have fun, Christian On Thu, Feb 05, 2015 at 11:19:20PM +0100, quelltextlich e.U. - Christian Aistleit

Re: [Analytics] s1-analytics-slave

Hi Kevin, [ To keep archives happy, it seems this thread got broken off of https://lists.wikimedia.org/pipermail/analytics/2015-February/003307.html ] On Mon, Feb 09, 2015 at 09:55:46AM -0800, Kevin Leduc wrote: > Christian, can we just point the geowiki code to a different database? Simple re

Re: [Analytics] s1-analytics-slave

I know that Erik Moller still uses geowiki a lot got look at the state of the wikis. I'm not ready to prioritize a transition of the geowiki code to use wikimetrics (that's a whole project in itself). Christian, can we just point the geowiki code to a different database? On Thu, Feb 5, 2015 at 6

Re: [Analytics] s1-analytics-slave

On Fri, Feb 6, 2015 at 12:45 AM, Aaron Halfaker wrote: > I've been slow to move some datasets off of s1-analytics-slave because it > remained available. If I were given ~ a week notice, it would be no > problem to move all datasets and work to analytics-store. > > Am I reading correctly that you

Re: [Analytics] s1-analytics-slave

Hi Kevin, Hi Sean, On Thu, Feb 05, 2015 at 11:42:26PM +1000, Sean Pringle wrote: > Who is still using s1-analytics-slave, and for what sorts of things? Geowiki is still using s1-analytics-slave for the erosen_* tables [1] in the staging database. Kevin, it was mentioned at some point that geowik

Re: [Analytics] s1-analytics-slave

On Thu, Feb 5, 2015 at 6:45 AM, Aaron Halfaker wrote: > I've been slow to move some datasets off of s1-analytics-slave because it > remained available. If I were given ~ a week notice, it would be no > problem to move all datasets and work to analytics-store. > same here. __

Re: [Analytics] s1-analytics-slave

I've been slow to move some datasets off of s1-analytics-slave because it remained available. If I were given ~ a week notice, it would be no problem to move all datasets and work to analytics-store. Am I reading correctly that you are suggesting that we might have *both* dbstore1002 and dbstore2

[Analytics] s1-analytics-slave

Who is still using s1-analytics-slave, and for what sorts of things? The analytics-store is just an alias for dbstore1002, which is about to be duplicated as dbstore2002 in CODFW with all wikis and eventlogging. We could potentially dub dbstore2002 as "analytics-slave" or similar, and reclaim the

Re: [Analytics] s1-analytics-slave impressively slow queries

Sean, What Nuria said. It seems we've missed this one. Sorry for the trouble. Leila On Mon, Nov 10, 2014 at 8:01 AM, Nuria Ruiz wrote: > cc-ing leila as we were experimenting with these some weeks back in SF, I > think they can be killed w/o problems. I did not know they were still > runnin

Re: [Analytics] s1-analytics-slave impressively slow queries

On Tue, Nov 11, 2014 at 8:44 AM, Dan Andreescu wrote: > Sean what came of your discussion with Coren about limiting time or memory > of queries? I think we should totally start enforcing those kinds of > limits as it seems any queries running longer than a few days are usually > accidents. Tha

Re: [Analytics] s1-analytics-slave impressively slow queries

Sean what came of your discussion with Coren about limiting time or memory of queries? I think we should totally start enforcing those kinds of limits as it seems any queries running longer than a few days are usually accidents. On Monday, November 10, 2014, Dario Taraborelli wrote: > Let's kil

Re: [Analytics] s1-analytics-slave impressively slow queries

Let's kill them (Leila is OoO today and tomorrow). > On Nov 10, 2014, at 08:01, Nuria Ruiz wrote: > > cc-ing leila as we were experimenting with these some weeks back in SF, I > think they can be killed w/o problems. I did not know they were still > running, we run a faster version of those qu

Re: [Analytics] s1-analytics-slave impressively slow queries

cc-ing leila as we were experimenting with these some weeks back in SF, I think they can be killed w/o problems. I did not know they were still running, we run a faster version of those queries and got the data we were interested in a while back. On Mon, Nov 10, 2014 at 1:55 AM, Sean Pringle wrot

[Analytics] s1-analytics-slave impressively slow queries

Three identical queries from the 'research_prod' user have just passed one month execution time on s1-anlytics-slave: select count(*) from staging.ourvision r where exists ( select * from staging.ourvision r1 inner join staging.ourvision r2 on r2.sha1 = r1.sha1 w

[Analytics] s1-analytics-slave schema change & replag

Running a schema change on s1-analytics-slave today. It will cause s1 replag. analytics-store is not affected. -- DBA @ WMF ___ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Re: [Analytics] s1-analytics-slave lag

Thanks for the heads up, Sean. The VE graphs for enwiki [1] are currently down, all the other ones are up (see for example [2]) so it looks like there’s a problem with s1 that is not related to EventLogging (the VE scripts pull data from the MW database). Dario [1] http://ee-dashboard.wmflabs.

[Analytics] s1-analytics-slave lag

Before someone emails me about this... :-) s1-analytics-slave eventlogging replication is starting to lag again (enwiki replication is ok). I noticed that new eventlogging tables are using InnoDB instead of TokuDB on that slave. The issue is being fixed and we should be back up to speed within th

Re: [Analytics] s1-analytics-slave enwiki replication

Btw, also unaffected is S1 on analytics-store. If desperate for enwiki, go there. -s ___ Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

[Analytics] s1-analytics-slave enwiki replication

Hi! Just a heads-up: Today s1-analytics-slave db1047 encountered an issue with a enwiki table, which in turn stopped S1 replication. The bug has been idetified and the affected table is being repaired, so replag should catch up within a few hours. Note that this did not affect M2 replication (ev