Sweet, thanks Jonathan. I added a "Heartbreak" token, because at this point I am really far too emotionally attached to Wikimetrics.
On Tue, Feb 17, 2015 at 9:25 AM, Jonathan Morgan <jmor...@wikimedia.org> wrote: > Hi Amanda, > > Here's a ticket you can upvote: https://phabricator.wikimedia.org/T87596 > > I added a link to this thread to the task. I also added an "Evil Spooky > Haunted Tree" token to the task. Because... well it just felt like the > right thing to do. > > - J > > On Tue, Feb 17, 2015 at 7:58 AM, Dan Andreescu <dandree...@wikimedia.org> > wrote: > >> I can't find a specific ticket, Nuria may know of one. In general, this >> is the project that LabsDB tickets are tagged with: >> https://phabricator.wikimedia.org/tag/wikimedia-labs-infrastructure/ >> >> On Tue, Feb 17, 2015 at 10:53 AM, Amanda Bittaker < >> abitta...@wikimedia.org> wrote: >> >>> Good morning Dan, >>> >>> Thanks very much for the explanation. Is there a Phabricator task we >>> can upvote (award a token?) to make this issue more visible? >>> >>> As always, we really appreciate your help with this. >>> >>> Best, >>> Amanda >>> >>> >>> >>> On Tue, Feb 17, 2015 at 7:20 AM, Dan Andreescu <dandree...@wikimedia.org >>> > wrote: >>> >>>> Sorry for the trouble, Amanda. The problem is solely with the >>>> underlying database, which we don't maintain. It's a sanitized replica of >>>> all the changes being made to all the wikis so it's a fairly complicated >>>> piece of infrastructure that sometimes has problems. The folks who >>>> maintain it are aware of the issues, but we'll continue representing them >>>> until they're solved. >>>> >>>> On Mon, Feb 16, 2015 at 3:49 PM, Amanda Bittaker < >>>> abitta...@wikimedia.org> wrote: >>>> >>>>> Oop, thanks for the ping, Nuria. Wikimetrics seems to be working >>>>> better now. I still get failures, especially when running three or four >>>>> reports in one batch, but the reports work if you rerun them (sometimes a >>>>> couple times.) >>>>> >>>>> I'm still getting "PENDING"s that turn into "FAILURE"s sometimes, >>>>> which I just noticed for the first time last Thursday. Also, sometimes >>>>> the >>>>> "FAILURE"s change position in the Current Report Inbox list, moving up or >>>>> down a spot. Not sure if that helps diagnose what might be happening... >>>>> >>>>> In any case, Wikimetrics is mostly functioning but seems to be having >>>>> recurring troubles that sometimes blow up to freeze the whole tool. It >>>>> would be great to resolve the troubles before the next explosion--is there >>>>> anything I can do to help? Dan H and I still have plenty of reports to >>>>> run, we can keep you updated on the reports ran and failure rate while you >>>>> are fixing, if that would be useful. >>>>> >>>>> Many thanks, >>>>> Amanda >>>>> >>>>> >>>>> On Mon, Feb 16, 2015 at 10:15 AM, Nuria Ruiz <nu...@wikimedia.org> >>>>> wrote: >>>>> >>>>>> Ping .... >>>>>> >>>>>> On Fri, Feb 13, 2015 at 2:19 PM, Nuria Ruiz <nu...@wikimedia.org> >>>>>> wrote: >>>>>> >>>>>>> Amanda, >>>>>>> >>>>>>> Looks like wikimetrics was able to run automatic reports last night >>>>>>> w/o big issues, are your reports still failing? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Nuria >>>>>>> >>>>>>> On Thu, Feb 12, 2015 at 1:42 PM, Amanda Bittaker < >>>>>>> abitta...@wikimedia.org> wrote: >>>>>>> >>>>>>>> Alright, thanks so much for your help once again, Nuria. >>>>>>>> >>>>>>>> If there's anything I can do or any information I can contribute, >>>>>>>> please don't hesitate to ping me. >>>>>>>> >>>>>>>> Best, >>>>>>>> Amanda >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Feb 12, 2015 at 1:36 PM, Nuria Ruiz <nu...@wikimedia.org> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> DB connections in labs look to be failing, unfortunately I think >>>>>>>>> besides asking for help on the labs list there is not much we can do >>>>>>>>> there. >>>>>>>>> I will start a thread on this regard. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> >>>>>>>>> Nuria >>>>>>>>> >>>>>>>>> On Thu, Feb 12, 2015 at 1:32 PM, Amanda Bittaker < >>>>>>>>> abitta...@wikimedia.org> wrote: >>>>>>>>> >>>>>>>>>> Thanks so much for the quick response, Nuria. >>>>>>>>>> >>>>>>>>>> I ran the exact same reports on the same cohort as one of the >>>>>>>>>> last batches that were failing. Last time 2/4 of the reports >>>>>>>>>> failed, when >>>>>>>>>> I reran the individually they succeeded. (But they don't always, I >>>>>>>>>> reran >>>>>>>>>> one report 3 times this morning before it worked.) This time, my >>>>>>>>>> failure >>>>>>>>>> rate got worse: 4/4 failed, although they said "PENDING" for a few >>>>>>>>>> seconds >>>>>>>>>> first, which is new. >>>>>>>>>> >>>>>>>>>> Is that useful information? Please do let me know what else I >>>>>>>>>> can do to help solve this. >>>>>>>>>> >>>>>>>>>> Thanks again, >>>>>>>>>> Amanda >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Feb 12, 2015 at 1:09 PM, Jonathan Morgan < >>>>>>>>>> jmor...@wikimedia.org> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Nuria! >>>>>>>>>>> >>>>>>>>>>> On Thu, Feb 12, 2015 at 12:57 PM, Nuria Ruiz < >>>>>>>>>>> nu...@wikimedia.org> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> If so a cohort + report to repro will be most useful. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Translation:* try to run the exact same reports on the same >>>>>>>>>>> cohort again, to see if the same metrics fail. Let us know what you >>>>>>>>>>> find. ;) >>>>>>>>>>> >>>>>>>>>>> Same goes for anyone else who experiences these issues: the more >>>>>>>>>>> details we (users) can provide the engineers, the more effective >>>>>>>>>>> they can >>>>>>>>>>> be at diagnosing and addressing the problems. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> - J >>>>>>>>>>> >>>>>>>>>>> *for anyone who is not 100% familiar with that hip, new software >>>>>>>>>>> engineering lingo >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>>> >>>>>>>>>>>> Nuria >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Feb 12, 2015 at 12:35 PM, Dan Andreescu < >>>>>>>>>>>> dandree...@wikimedia.org> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Recently there was a restart of the labsdb cluster. I'm sorry >>>>>>>>>>>>> but I don't have time to check on it, but I bet that's the >>>>>>>>>>>>> problem. I'm >>>>>>>>>>>>> off tomorrow unfortunately but I'll try to check tomorrow night >>>>>>>>>>>>> :( I hope >>>>>>>>>>>>> someone else beats me to it. >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Feb 12, 2015 at 3:20 PM, Jonathan Morgan < >>>>>>>>>>>>> jmor...@wikimedia.org> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> (ping Kevin and Dan A.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Amanda, I've had some problems with report failures >>>>>>>>>>>>>> recently when I ran a few test cohorts. On the same cohort, when >>>>>>>>>>>>>> I ran >>>>>>>>>>>>>> multiple concurrent reports (say, bytes added, edits, and pages >>>>>>>>>>>>>> created), >>>>>>>>>>>>>> some would fail and others succeed. It wasn't clear what the >>>>>>>>>>>>>> issue was. >>>>>>>>>>>>>> >>>>>>>>>>>>>> - J >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Feb 12, 2015 at 12:16 PM, Amanda Bittaker < >>>>>>>>>>>>>> abitta...@wikimedia.org> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hello all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I am getting failures again, both when uploading cohorts and >>>>>>>>>>>>>>> running reports. Strangely, it seems the more reports you try >>>>>>>>>>>>>>> to run in >>>>>>>>>>>>>>> one batch the less likely it is any report will succeed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Is anyone else having these problems again? Wonderful >>>>>>>>>>>>>>> Analytics people, could you please work your magic again? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Many thanks, >>>>>>>>>>>>>>> Amanda >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>>> Wikimetrics mailing list >>>>>>>>>>>>>>> Wikimetrics@lists.wikimedia.org >>>>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Jonathan T. Morgan >>>>>>>>>>>>>> Community Research Lead >>>>>>>>>>>>>> Wikimedia Foundation >>>>>>>>>>>>>> User:Jmorgan (WMF) >>>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> >>>>>>>>>>>>>> jmor...@wikimedia.org >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Wikimetrics mailing list >>>>>>>>>>>>> Wikimetrics@lists.wikimedia.org >>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Jonathan T. Morgan >>>>>>>>>>> Community Research Lead >>>>>>>>>>> Wikimedia Foundation >>>>>>>>>>> User:Jmorgan (WMF) >>>>>>>>>>> <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> >>>>>>>>>>> jmor...@wikimedia.org >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Wikimetrics mailing list >>>>>>>>>>> Wikimetrics@lists.wikimedia.org >>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> Wikimetrics mailing list >>>>> Wikimetrics@lists.wikimedia.org >>>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> Wikimetrics mailing list >>>> Wikimetrics@lists.wikimedia.org >>>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>>> >>>> >>> >>> _______________________________________________ >>> Wikimetrics mailing list >>> Wikimetrics@lists.wikimedia.org >>> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >>> >>> >> >> _______________________________________________ >> Wikimetrics mailing list >> Wikimetrics@lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikimetrics >> >> > > > -- > Jonathan T. Morgan > Community Research Lead > Wikimedia Foundation > User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> > jmor...@wikimedia.org > > > _______________________________________________ > Wikimetrics mailing list > Wikimetrics@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikimetrics > >
_______________________________________________ Wikimetrics mailing list Wikimetrics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimetrics
