Thanks for this Andrew! Sadly it appears to be down again, did we try the patch again? Thanks in advance for any details you can provide.
On Fri, Feb 24, 2023 at 2:50 PM Andrew Otto <[email protected]> wrote: > Hello! > > Here is what I know. > - 2023-02-22T14:33 UTC - This patch > <https://gerrit.wikimedia.org/r/c/operations/puppet/+/890843/3#message-dcf8af5c4e17a81ad79d16fa6cd7aad504432f87> > was merged > - Over the next several hours systemd timers (cron jobs) across a lot of > the fleet stopped running. > - 2023-02-22T17:34 UTC - This patch > <https://gerrit.wikimedia.org/r/c/operations/puppet/+/891340/> reverted > the change > After this jobs began to run again. However, because the webrequest > dataset is so huge, it took hours for ingestion of it to catch up. > Downstream jobs that use webrequest as input (including pageviews > computation) began to timeout while waiting for input. > - We have been slowly restarting and recovering jobs now that webrequest > ingestion has caught up again. > > I don't know exactly how long data is delayed or when it will be fully > available, but I'd guess: soon / today? > > -Andrew Otto > WMF > > > On Fri, Feb 24, 2023 at 7:10 AM Joshua Haecker <[email protected]> wrote: > >> Another day another delay, is there a known broader issue? Just trying to >> provide reasonable expectations of when data might be available. Thanks >> >> On Thu, Feb 23, 2023 at 9:55 AM Joshua Haecker <[email protected]> wrote: >> >>> Hi all, >>> >>> Just curious if there is a known cause for the multiple long delays >>> we've had on the AQS API data being available this week? I know periodic >>> delays are not uncommon but these seem beyond normal levels. >>> >>> Thanks! >>> >>> ~Josh >>> >>> -- >> Joshua Haecker >> CEO, Co-Founder >> Predata, Inc. >> >> [email protected] >> 609-865-2181 >> 1201 Pennsylvania Ave NW >> <https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g> >> Washington, D.C. >> <https://www.google.com/maps/search/1201+Pennsylvania+Ave+NW+Washington,+D.C.?entry=gmail&source=g> >> >> *This message and its contents are confidential. If you received this >> message in error, do not use or rely upon it. Instead, please inform the >> sender and then delete it. Thank you.* >> > _______________________________________________ >> Analytics mailing list -- [email protected] >> To unsubscribe send an email to [email protected] >> > _______________________________________________ > Analytics mailing list -- [email protected] > To unsubscribe send an email to [email protected] > -- Joshua Haecker CEO, Co-Founder Predata, Inc. [email protected] 609-865-2181 1201 Pennsylvania Ave NW Washington, D.C. *This message and its contents are confidential. If you received this message in error, do not use or rely upon it. Instead, please inform the sender and then delete it. Thank you.*
_______________________________________________ Analytics mailing list -- [email protected] To unsubscribe send an email to [email protected]
