I can't find a specific ticket, Nuria may know of one. In general, this is the project that LabsDB tickets are tagged with: https://phabricator.wikimedia.org/tag/wikimedia-labs-infrastructure/
On Tue, Feb 17, 2015 at 10:53 AM, Amanda Bittaker <abitta...@wikimedia.org> wrote: > Good morning Dan, > > Thanks very much for the explanation. Is there a Phabricator task we can > upvote (award a token?) to make this issue more visible? > > As always, we really appreciate your help with this. > > Best, > Amanda > > > > On Tue, Feb 17, 2015 at 7:20 AM, Dan Andreescu <dandree...@wikimedia.org> > wrote: > >> Sorry for the trouble, Amanda. The problem is solely with the underlying >> database, which we don't maintain. It's a sanitized replica of all the >> changes being made to all the wikis so it's a fairly complicated piece of >> infrastructure that sometimes has problems. The folks who maintain it are >> aware of the issues, but we'll continue representing them until they're >> solved. >> >> On Mon, Feb 16, 2015 at 3:49 PM, Amanda Bittaker <abitta...@wikimedia.org >> > wrote: >> >>> Oop, thanks for the ping, Nuria. Wikimetrics seems to be working better >>> now. I still get failures, especially when running three or four reports >>> in one batch, but the reports work if you rerun them (sometimes a couple >>> times.) >>> >>> I'm still getting "PENDING"s that turn into "FAILURE"s sometimes, which >>> I just noticed for the first time last Thursday. Also, sometimes the >>> "FAILURE"s change position in the Current Report Inbox list, moving up or >>> down a spot. Not sure if that helps diagnose what might be happening... >>> >>> In any case, Wikimetrics is mostly functioning but seems to be having >>> recurring troubles that sometimes blow up to freeze the whole tool. It >>> would be great to resolve the troubles before the next explosion--is there >>> anything I can do to help? Dan H and I still have plenty of reports to >>> run, we can keep you updated on the reports ran and failure rate while you >>> are fixing, if that would be useful. >>> >>> Many thanks, >>> Amanda >>> >>> >>> On Mon, Feb 16, 2015 at 10:15 AM, Nuria Ruiz <nu...@wikimedia.org> >>> wrote: >>> >>>> Ping .... >>>> >>>> On Fri, Feb 13, 2015 at 2:19 PM, Nuria Ruiz <nu...@wikimedia.org> >>>> wrote: >>>> >>>>> Amanda, >>>>> >>>>> Looks like wikimetrics was able to run automatic reports last night >>>>> w/o big issues, are your reports still failing? >>>>> >>>>> Thanks, >>>>> >>>>> Nuria >>>>> >>>>> On Thu, Feb 12, 2015 at 1:42 PM, Amanda Bittaker < >>>>> abitta...@wikimedia.org> wrote: >>>>> >>>>>> Alright, thanks so much for your help once again, Nuria. >>>>>> >>>>>> If there's anything I can do or any information I can contribute, >>>>>> please don't hesitate to ping me. >>>>>> >>>>>> Best, >>>>>> Amanda >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Feb 12, 2015 at 1:36 PM, Nuria Ruiz <nu...@wikimedia.org> >>>>>> wrote: >>>>>> >>>>>>> DB connections in labs look to be failing, unfortunately I think >>>>>>> besides asking for help on the labs list there is not much we can do >>>>>>> there. >>>>>>> I will start a thread on this regard. >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Nuria >>>>>>> >>>>>>> On Thu, Feb 12, 2015 at 1:32 PM, Amanda Bittaker < >>>>>>> abitta...@wikimedia.org> wrote: >>>>>>> >>>>>>>> Thanks so much for the quick response, Nuria. >>>>>>>> >>>>>>>> I ran the exact same reports on the same cohort as one of the last >>>>>>>> batches that were failing. Last time 2/4 of the reports failed, when I >>>>>>>> reran the individually they succeeded. (But they don't always, I >>>>>>>> reran one >>>>>>>> report 3 times this morning before it worked.) This time, my failure >>>>>>>> rate >>>>>>>> got worse: 4/4 failed, although they said "PENDING" for a few seconds >>>>>>>> first, which is new. >>>>>>>> >>>>>>>> Is that useful information? Please do let me know what else I can >>>>>>>> do to help solve this. >>>>>>>> >>>>>>>> Thanks again, >>>>>>>> Amanda >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 12, 2015 at 1:09 PM, Jonathan Morgan < >>>>>>>> jmor...@wikimedia.org> wrote: >>>>>>>> >>>>>>>>> Thanks Nuria! >>>>>>>>> >>>>>>>>> On Thu, Feb 12, 2015 at 12:57 PM, Nuria Ruiz <nu...@wikimedia.org> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> If so a cohort + report to repro will be most useful. >>>>>>>>>> >>>>>>>>> >>>>>>>>> Translation:* try to run the exact same reports on the same cohort >>>>>>>>> again, to see if the same metrics fail. Let us know what you find. ;) >>>>>>>>> >>>>>>>>> Same goes for anyone else who experiences these issues: the more >>>>>>>>> details we (users) can provide the engineers, the more effective they >>>>>>>>> can >>>>>>>>> be at diagnosing and addressing the problems. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> - J >>>>>>>>> >>>>>>>>> *for anyone who is not 100% familiar with that hip, new software >>>>>>>>> engineering lingo >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> Nuria >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Feb 12, 2015 at 12:35 PM, Dan Andreescu < >>>>>>>>>> dandree...@wikimedia.org> wrote: >>>>>>>>>> >>>>>>>>>>> Recently there was a restart of the labsdb cluster. I'm sorry >>>>>>>>>>> but I don't have time to check on it, but I bet that's the problem. >>>>>>>>>>> I'm >>>>>>>>>>> off tomorrow unfortunately but I'll try to check tomorrow night :( >>>>>>>>>>> I hope >>>>>>>>>>> someone else beats me to it. >>>>>>>>>>> >>>>>>>>>>> On Thu, Feb 12, 2015 at 3:20 PM, Jonathan Morgan < >>>>>>>>>>> jmor...@wikimedia.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> (ping Kevin and Dan A.) >>>>>>>>>>>> >>>>>>>>>>>> Hi Amanda, I've had some problems with report failures recently >>>>>>>>>>>> when I ran a few test cohorts. On the same cohort, when I ran >>>>>>>>>>>> multiple >>>>>>>>>>>> concurrent reports (say, bytes added, edits, and pages created), >>>>>>>>>>>> some would >>>>>>>>>>>> fail and others succeed. It wasn't clear what the issue was. >>>>>>>>>>>> >>>>>>>>>>>> - J >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Feb 12, 2015 at 12:16 PM, Amanda Bittaker < >>>>>>>>>>>> abitta...@wikimedia.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello all, >>>>>>>>>>>>> >>>>>>>>>>>>> I am getting failures again, both when uploading cohorts and >>>>>>>>>>>>> running reports. Strangely, it seems the more reports you try to >>>>>>>>>>>>> run in >>>>>>>>>>>>> one batch the less likely it is any report will succeed. >>>>>>>>>>>>> >>>>>>>>>>>>> Is anyone else having these problems again? Wonderful >>>>>>>>>>>>> Analytics people, could you please work your magic again? >>>>>>>>>>>>> >>>>>>>>>>>>> Many thanks, >>>>>>>>>>>>> Amanda >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Wikimetrics mailing list >>>>>>>>>>>>> Wikimetrics@lists.wikimedia.org >>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Jonathan T. Morgan >>>>>>>>>>>> Community Research Lead >>>>>>>>>>>> Wikimedia Foundation >>>>>>>>>>>> User:Jmorgan (WMF) >>>>>>>>>>>> <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> >>>>>>>>>>>> jmor...@wikimedia.org >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Wikimetrics mailing list >>>>>>>>>>> Wikimetrics@lists.wikimedia.org >>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jonathan T. Morgan >>>>>>>>> Community Research Lead >>>>>>>>> Wikimedia Foundation >>>>>>>>> User:Jmorgan (WMF) >>>>>>>>> <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> >>>>>>>>> jmor...@wikimedia.org >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> Wikimetrics mailing list >>>>>>>>> Wikimetrics@lists.wikimedia.org >>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>> _______________________________________________ >>> Wikimetrics mailing list >>> Wikimetrics@lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>> >>> >> >> _______________________________________________ >> Wikimetrics mailing list >> Wikimetrics@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >> >> > > _______________________________________________ > Wikimetrics mailing list > Wikimetrics@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikimetrics > >
_______________________________________________ Wikimetrics mailing list Wikimetrics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimetrics
