[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, Gehel Cc: HenkvD, MPhamWMF, DAbad, RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fu

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 830924 **merged** by jenkins-bot: [wikidata/query/rdf@master] Store OAuth1AccessToken fully within cookies https://gerrit.wikimedia.org/r/830924 TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedi

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 830924 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [wikidata/query/rdf@master] Store OAuth1AccessToken fully within cookies https://gerrit.wikimedia.org/r/830924 TASK DETAIL https://phabricator.wikimedia.org/T306899

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. Some quick testing makes this look successful. Using curl to perform a POST no longer 500's: curl 'https://commons-query.wikimedia.org/sparql' \ -XPOST \ -H 'cookie: wcqsOauth=; wcqsSession=' \ -d 'query=prefix%20schema:%20%3Chttp://sche

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-08-30T05:03:45Z] T306899 T316496 Deployed WCQS `0.3.115`. That should (hopefully) resolve these tickets. TASK DETAIL https://ph

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 825922 **merged** by jenkins-bot: [wikidata/query/rdf@master] mw-oauth-proxy: Update nginx test integration to work with POST https://gerrit.wikimedia.org/r/825922 TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 825434 **merged** by jenkins-bot: [wikidata/query/rdf@master] mw-oauth-proxy: Remember access token and use to refresh session https://gerrit.wikimedia.org/r/825434 TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https:/

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 825925 **merged** by Bking: [operations/puppet@production] query_service: Avoid passing content body to internal auth endpoints https://gerrit.wikimedia.org/r/825925 TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https:

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 825925 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [operations/puppet@production] query_service: Avoid passing content body to internal auth endpoints https://gerrit.wikimedia.org/r/825925 TASK DETAIL https://phabric

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 825922 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [wikidata/query/rdf@master] mw-oauth-proxy: Update nginx test integration to work with POST https://gerrit.wikimedia.org/r/825922 TASK DETAIL https://phabricator.wik

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, gerritbot Cc: HenkvD, MPhamWMF, DAbad, RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 825434 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [wikidata/query/rdf@master] mw-oauth-proxy: Remember access token and use to refresh session https://gerrit.wikimedia.org/r/825434 TASK DETAIL https://phabricator.wi

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

HenkvD added a comment. I also have had many of these errors. I sometimes work for hours with the same session, and I agree it might be time-outs. In my experience: I can continue right away if I open a new browser session. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREF

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. I've tracked down one source of 500 errors, unclear if the original report here is for same thing. Reproduction: curl -XPOST https://commons-query.wikimedia.org/any-url-doesnt-matter -d 'foo=bar' Reason: This request includes a `Content-Length`

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. Leaving the commons-query.wikimedia.org browser tab open for a few hours and re-running queries every 30-60 minutes or so reproduced a 500 after a few hours. Related js console errors. Timestamps are PDT. Unclear if the errors at 13:00 and 13:10 are directly rel

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. In T306899#8128904 , @Dominicbm wrote: > Experienced the same error today again, here is an exact timestamp (of the response): `Wed, 03 Aug 2022 17:15:19 GMT`. This lines up nicely with a message from l

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

MPhamWMF added subscribers: DAbad, MPhamWMF. MPhamWMF added a comment. @DAbad , this looks like it may be related to WCQS authentication, which may be relevant to current platform work TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.o

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm added a comment. Experienced the same error today again, here is an exact timestamp (of the response): `Wed, 03 Aug 2022 17:15:19 GMT`. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm added a comment. In T306899#7945324 , @RKemper wrote: > @Dominicbm We made `the error message is understandable by users` part of the acceptance criteria of this ticket, so I think that's why T306919

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm reopened this task as "Open". TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, Dominicbm Cc: RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, D

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm added a comment. In T306899#8029731 , @Gehel wrote: > @Dominicbm, @Fuzheado: did you see this issue again? Do you have a timestamp that we could correlate with the logs? I have been away from WCQS for a while, so the lack

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Gehel closed this task as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, Gehel Cc: RKemper, EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominic

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Gehel added a comment. Since we did not get any feedback on this error, we're assuming that it has resolved. If not, please re-open and ping us with a timestamp of when the error occurs. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedi

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Gehel added a comment. @Dominicbm, @Fuzheado: did you see this issue again? Do you have a timestamp that we could correlate with the logs? TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBern

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. Lacking better ideas on how to align the errors with some request that causes the error I've started up `tcpdump` on all the wcqs instances. They will store up to 100 1GB files per instance before starting to overwrite the initial files. The overall goal here is

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. Reviewed logs again looking for patterns. Not much, but at least logstash is now aggregating together logs from the various hosts. Can see that the `/oauth/check_auth java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 3/3

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

RKemper added a comment. @Dominicbm We made `the error message is understandable by users` part of the acceptance criteria of this ticket, so I think that's why T306919 was closed as a duplicate. Basically this ticket covers both the fixing of the

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm added a comment. In T306899#7942622 , @RKemper wrote: > @Dominicbm @Fuzheado Understood that you all are still seeing these failures. We're still having trouble reproducing it on our end (we don't have great intuition as why b

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2022-05-19T17:51:34Z] T306899 Rolled `wdqs` and `wcqs` deploys to adjust logging settings. Hoping this gives us more visibility on the 500 errors WCQS users have been experiencing

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

RKemper added a comment. @Dominicbm @Fuzheado Understood that you all are still seeing these failures. We're still having trouble reproducing it on our end (we don't have great intuition as why but it's possible that the issue might only surface based on the geoDNS routing). We've added some

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson, Maintenance_bot Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Ak

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 792266 **merged** by Ryan Kemper: [operations/puppet@production] rdf query service: Apply WARN log level only to com.bigdata https://gerrit.wikimedia.org/r/792266 TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://p

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm added a comment. Honestly, if you are seeing few errors, it may also be that heavy users (like myself) are putting off our projects until the tool is more usable. If another data point is useful, I was using WCQS again today, and only submitted a couple of simple queries before I g

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson claimed this task. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: EBernhardson Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Fernandobacasegua3

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Fernando

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 792266 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [operations/puppet@production] rdf query service: Apply WARN log level only to com.bigdata https://gerrit.wikimedia.org/r/792266 TASK DETAIL https://phabricator.wiki

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Domin

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 790775 **merged** by jenkins-bot: [wikidata/query/rdf@master] Separate logging deps into dedicated package https://gerrit.wikimedia.org/r/790775 TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedi

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: EBernhardson, FRomeo_WMF, GFontenelle_WMF, Gehel, Fuzheado, Aklapper, Dominicbm, Fernando

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

gerritbot added a comment. Change 790775 had a related patch set uploaded (by Ebernhardson; author: Ebernhardson): [wikidata/query/rdf@master] Separate logging deps into dedicated package https://gerrit.wikimedia.org/r/790775 TASK DETAIL https://phabricator.wikimedia.org/T306899

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. Following the thread of something related to auth, I've found that the application server (jetty) which hosts the app has never properly had it's logging setup. Logs only come from the embedded applications, the application server itself ends up with bare minimu

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. Not finding anything decisive yet, but will continue looking. It occured to me that if it's happening consistently for an individual user but not in general that it could somehow be related to their authentication cookie. If seems plausible clearing the auth coo

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

EBernhardson added a comment. Usually the first stop for this kind of error would be reviewing the `ATS Backends <-> Origin Servers Overview` which suggest a low rate of 5xxs, typically 1-5% of requests fail. In a quick review of the last few 500 requests on one of the servers they were all

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm added a subscriber: Gehel. Dominicbm added a comment. @Gehel My intention was to have one ticket for the reliability issues, and one for the message itself, if that makes sense. Just wanted to point out, because the other one was merged into this, but the new AC for this ticket, wh

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

MPhamWMF set the point value for this task to "8". TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: MPhamWMF Cc: Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, ma

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Gehel updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, CBogen,

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Gehel merged a task: T306919: Improve query service 500 error message. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Gehel Cc: Fuzheado, Aklapper, Dominicbm, Astuthiodit_1, karapayneWMDE, Invadib

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

MPhamWMF moved this task from Incoming to Current work on the Wikidata-Query-Service board. MPhamWMF added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T306899 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL PREFERENCES htt

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm added a comment. As @Fuzheado notes, it usually resolves itself shortly, but the tool itself feels unreliable. As an example, I've been using it heavily today. It went down for me for 20-30 minutes around 9:30 am today in my time, and then began working again, and now I am encounte

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Fuzheado added a comment. Seems to be OK now, but I agree with @Dominicbm - these intermittent errors will crop up for 10-20 minutes then go away, but it's not clear whether they are being noticed or tracked by the folks on the search team. Any insights would be most welcome. Thanks. TASK D

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T306899 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: Aklapper, Dominicbm, Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja,

[Wikidata-bugs] [Maniphest] T306899: WCQS 500 errors

Dominicbm created this task. Dominicbm added a project: Wikidata-Query-Service. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION I am experiencing `500 Internal Server Error` again in WCQS. This happens after a query is run, and the following message is returned in the outp