[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Nikerabbit]

Nikerabbit updated the task description.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: RLazarus, akosiaris, Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, 
KartikMistry, Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, 
Rileych, Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Nikerabbit]

Nikerabbit closed this task as "Resolved".
Nikerabbit claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: RLazarus, akosiaris, Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, 
KartikMistry, Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, 
Rileych, Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Addshore]

Addshore added a comment.


  Should we close this?

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: RLazarus, akosiaris, Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, 
KartikMistry, Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, 
Rileych, Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[KartikMistry]

KartikMistry added a comment.


  Moving to done. Errors are no longer happening or not frequent atleast.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: KartikMistry
Cc: RLazarus, akosiaris, Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, 
KartikMistry, Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, 
Rileych, Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[KartikMistry]

KartikMistry added a comment.


  In T285219#7220841 , 
@JMeybohm wrote:
  
  > Picking up from the IRC conversation yesterday @RLazarus figured that the 
response body looks like it is 
https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/master/errorpages/503.html
  > At the time this issue was opened (June 21) we did had some database 
issues, so increased rate of 503's from apiservers are most likely due to that.
  >
  > Looking at the last two weeks the picture has changed from June to now with 
only a hand full of requests failing for cxserver in the last two weeks, most 
of them due to upstream connection failure ("UF" in response flags field). 
Those errors might happen from time to time due to the service-proxy creating 
persistent connections which then might get closed server side or due some 
network issues. But as that is happening at a very low rate, we did not dig 
more into that by now.
  
  Thanks a lot!
  
  It seems these errors has reduced greatly or vanished now (atleast since last 
24 hours!)
  
  @Nikerabbit We can wait for a day or two and then moved it to Done.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: KartikMistry
Cc: RLazarus, akosiaris, Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, 
KartikMistry, Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, 
Rileych, Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[JMeybohm]

JMeybohm added a subscriber: RLazarus.
JMeybohm added a comment.


  Picking up from the IRC conversation yesterday @RLazarus figured that the 
response body looks like it is 
https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/master/errorpages/503.html
  At the time this issue was opened (June 21) we did had some database issues, 
so increased rate of 503's from apiservers are most likely due to that.
  
  Looking at the last two weeks the picture has changed from June to now with 
only a hand full of requests failing for cxserver in the last two weeks, most 
of them due to upstream connection failure ("UF" in response flags field). 
Those errors might happen from time to time due to the service-proxy creating 
persistent connections which then might get closed server side or due some 
network issues. But as that is happening at a very low rate, we did not dig 
more into that by now.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: JMeybohm
Cc: RLazarus, akosiaris, Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, 
KartikMistry, Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, 
Rileych, Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Nikerabbit]

Nikerabbit added a comment.


  My current understanding:
  
  - This error does not originate from cxserver
  - Usually API errors contain an API response with error codes.
  - The HTTP 503 error is hiding the underlying issue and it's not giving any 
error codes or request ids to explore further. I do not know how to find the 
underlying issue.
  - This issue does not affect CXServer only, but also other projects using the 
tls-proxy at rate of over 100 per minute: 
https://logstash.wikimedia.org/goto/67bdc723be54629059656d3034460bad
  - It is not only Wikidata, but also Commons and other wikis to which the 
requests are failing
  - One simple query that failed is 
`https://wikidata.org/w/api.php?action=wbgetentities=labels=Q1139546=en=json`
 – this should not fail due to any reason related to the query itself
  
  Is this a general issue with the tls-proxy that some amount of requests are 
failing? Or if not and this is just the way it reports all errors from 
upstream, how can we make the tls-proxy log the actual issue somewhere (or 
where is it if it is done already)?

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, 
Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, Rileych, 
Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Pginer-WMF]

Pginer-WMF edited projects, added Language-Team (Language-2021-July-September); 
removed Language-Team (Language-2021-April-June).
Pginer-WMF moved this task from Quarter Backlog to Blocked on the Language-Team 
(Language-2021-July-September) board.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

WORKBOARD
  https://phabricator.wikimedia.org/project/board/5406/

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pginer-WMF
Cc: Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, 
Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, Rileych, 
Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Nikerabbit]

Nikerabbit removed a project: Patch-For-Review.
Nikerabbit added a comment.


  Ignore above, accidentally pasted wrong bug number.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, 
Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, Rileych, 
Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn, Biggs657, 
Lalamarie69, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, joker88john, 
CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Bsandipan, Lewizho99, 
Maathavan
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[gerritbot]

gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, Biggs657, 
Invadibot, UOzurumba, Lalamarie69, PallaviPatke, maantietaja, wkandek, 
Alter-paule, Beast1978, Rileych, Un1tY, Nintendofan885, Akuckartz, 50019062, 
Hook696, Kent7301, joker88john, CucyNoiD, Nandana, jijiki, Gaboe420, 
Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, GoranSMilovanovic, 
chapulina, QZanden, Alfa80, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[gerritbot]

gerritbot added a comment.


  Change 702077 had a related patch set uploaded (by Nikerabbit; author: 
Nikerabbit):
  
  [translatewiki@master] Add systemd timer for autoexport
  
  https://gerrit.wikimedia.org/r/702077

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, 
Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, Rileych, 
Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Nikerabbit]

Nikerabbit added a comment.


  Usually there is a request id of some kind in the error message, but this has 
none. I wonder if this is some kind of rate limiting or timeout issue?

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, 
Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, Rileych, 
Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Nikerabbit]

Nikerabbit added a comment.


  See
  
  - 
https://github.com/wikimedia/mediawiki-services-cxserver/blob/master/lib/suggestion/SourceSuggester.js#L128
  - 
https://github.com/wikimedia/mediawiki-services-cxserver/blob/master/lib/suggestion/SourceSuggester.js#L151
  
  These look like simple queries to me, except if the latter one has a lot of 
languages.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, 
Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, Rileych, 
Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Addshore]

Addshore added a comment.


  I'm not sure of anything that would be causing this.
  What sort of requests is cxserver making to the wikidata api?

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Addshore
Cc: Addshore, JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, 
Invadibot, UOzurumba, PallaviPatke, maantietaja, wkandek, Rileych, 
Nintendofan885, Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, 
GoranSMilovanovic, chapulina, QZanden, Alfa80, LawExplorer, _jensen, 
rosalieper, Soum213, Taiwania_Justo, Nizil, Scott_WUaS, Ixocactus, 
Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T285219: cxserver: https://cxserver.wikimedia.org/v2/suggest/source/Paneer/ca?sourcelanguages=en occasionally fails with HTTP 503

[Nikerabbit]

Nikerabbit added a project: Wikidata.
Nikerabbit added a comment.
Restricted Application added a project: wdwb-tech.


  It's likely that the error is coming from `wbsearchentities` or  
`wbgetentities` API and cxserver's error handling just forwards the whole error 
reply through it's own API. Tagging #wikidata 
 in case they are aware of any 
issues related to these APIs.

TASK DETAIL
  https://phabricator.wikimedia.org/T285219

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Nikerabbit
Cc: JMeybohm, santhosh, Nikerabbit, Aklapper, KartikMistry, Invadibot, 
UOzurumba, PallaviPatke, maantietaja, wkandek, Rileych, Nintendofan885, 
Akuckartz, 50019062, Nandana, jijiki, Lahi, Gq86, GoranSMilovanovic, chapulina, 
QZanden, Alfa80, LawExplorer, _jensen, rosalieper, Soum213, Taiwania_Justo, 
Nizil, Scott_WUaS, Ixocactus, Wikidata-bugs, aude, Amire80, Jsahleen, Arrbee, 
Addshore, Mbch331, Dzahn
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org