Hi Joshua, today's delay is unrelated to the problem mentioned by Andrew, it was due to a silent issue with our scheduling system. We are investigating it. In the meantime, the data is now available. Sorry for the inconvenience!
On Fri, Mar 3, 2023 at 12:38 PM Joshua Haecker <[email protected]> wrote: > Thanks for this Andrew! Sadly it appears to be down again, did we try the > patch again? Thanks in advance for any details you can provide. > > On Fri, Feb 24, 2023 at 2:50 PM Andrew Otto <[email protected]> wrote: > >> Hello! >> >> Here is what I know. >> - 2023-02-22T14:33 UTC - This patch >> <https://gerrit.wikimedia.org/r/c/operations/puppet/+/890843/3#message-dcf8af5c4e17a81ad79d16fa6cd7aad504432f87> >> was merged >> - Over the next several hours systemd timers (cron jobs) across a lot of >> the fleet stopped running. >> - 2023-02-22T17:34 UTC - This patch >> <https://gerrit.wikimedia.org/r/c/operations/puppet/+/891340/> reverted >> the change >> After this jobs began to run again. However, because the webrequest >> dataset is so huge, it took hours for ingestion of it to catch up. >> Downstream jobs that use webrequest as input (including pageviews >> computation) began to timeout while waiting for input. >> - We have been slowly restarting and recovering jobs now that webrequest >> ingestion has caught up again. >> >> I don't know exactly how long data is delayed or when it will be fully >> available, but I'd guess: soon / today? >> >> -Andrew Otto >> WMF >> >> >> On Fri, Feb 24, 2023 at 7:10 AM Joshua Haecker <[email protected]> wrote: >> >>> Another day another delay, is there a known broader issue? Just trying >>> to provide reasonable expectations of when data might be available. Thanks >>> >>> On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker <[email protected]> wrote: >>> >>>> Hi all, >>>> >>>> Just curious if there is a known cause for the multiple long delays >>>> we've had on the AQS API data being available this week? I know periodic >>>> delays are not uncommon but these seem beyond normal levels. >>>> >>>> Thanks! >>>> >>>> ~Josh >>>> >>>> -- >>> Joshua Haecker >>> CEO, Co-Founder >>> Predata, Inc. >>> >>> [email protected] >>> 609-865-2181 >>> 1201 Pennsylvania Ave NW >>> <https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g> >>> Washington, D.C. >>> <https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g> >>> >>> *This message and its contents are confidential. If you received this >>> message in error, do not use or rely upon it. Instead, please inform the >>> sender and then delete it. Thank you.* >>> >> _______________________________________________ >>> Analytics mailing list -- [email protected] >>> To unsubscribe send an email to [email protected] >>> >> _______________________________________________ >> Analytics mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> > -- > Joshua Haecker > CEO, Co-Founder > Predata, Inc. > > [email protected] > 609-865-2181 > 1201 Pennsylvania Ave NW > Washington, D.C. > > *This message and its contents are confidential. If you received this > message in error, do not use or rely upon it. Instead, please inform the > sender and then delete it. Thank you.* > _______________________________________________ > Analytics mailing list -- [email protected] > To unsubscribe send an email to [email protected] > -- *Marcel Ruiz Forns** (he/him)* Senior Software Engineer
_______________________________________________ Analytics mailing list -- [email protected] To unsubscribe send an email to [email protected]
