[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[Pintoch]

Pintoch added a comment.


  @Spinster I'd rather keep this open since this is a problem that is likely to 
be relevant to other tools and still has not been solved as far as I can tell. 
It seems similar to T282732  which 
is current apparently.

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pintoch
Cc: Fuzheado, Spinster, wolfgang8741, Bstorm, Thadguidry, Lydia_Pintscher, 
Nemo_bis, Nintendofan885, bd808, Pintoch, Aklapper, Invadibot, Devnull, 
maantietaja, Akuckartz, Nandana, skpuneethumar, Zylc, 1978Gage2001, Lahi, 
Operator873, Gq86, Bsandipan, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, 
Nattes, QZanden, Tbscho, Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, 
Scott_WUaS, Luke081515, Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, 
Nikerabbit, scfc, coren, Mbch331, Krenair
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[Spinster]

Spinster added a comment.


  I'm doing some cleanup of OpenRefine tasks in preparation for new development 
related to #structureddataoncommons 
  (see this 
page 

 for more context) and I wonder if this issue is still relevant, or if this 
task can be closed?
  
  https://wdreconcile.toolforge.org is deprecated and 
https://wikidata.reconci.link is its replacement for a while already.

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Spinster
Cc: Spinster, wolfgang8741, Bstorm, Thadguidry, Lydia_Pintscher, Nemo_bis, 
Nintendofan885, bd808, Pintoch, Aklapper, Invadibot, Devnull, maantietaja, 
Akuckartz, Nandana, skpuneethumar, Zylc, 1978Gage2001, Lahi, Operator873, Gq86, 
Bsandipan, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, Nattes, QZanden, 
Tbscho, Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, Scott_WUaS, 
Luke081515, Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, Nikerabbit, scfc, 
coren, Mbch331, Krenair
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[bd808]

bd808 added a comment.


  In T257405#6505154 , 
@Pintoch wrote:
  
  > To restart the service, a single command is needed: `webservice --backend 
kubernetes python3.7 restart`. It is very simple!
  
  Here's a way to make it even a bit easier: create a $HOME/service.template 
file 
 
with your default settings in it and then just type `webservice 
[start|stop|restart]` as needed.
  
  name=$HOME/service.template
backend: kubernetes
type: python3.7

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: bd808
Cc: Bstorm, Thadguidry, Lydia_Pintscher, Nemo_bis, Nintendofan885, bd808, 
Pintoch, Aklapper, Akuckartz, darthmon_wmde, Nandana, skpuneethumar, Zylc, 
1978Gage2001, Lahi, Gq86, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, 
QZanden, Tbscho, Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, 
Scott_WUaS, Luke081515, Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, 
Nikerabbit, scfc, coren, Mbch331, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[Pintoch]

Pintoch added a comment.


  It does use tools-redis indeed!

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pintoch
Cc: Bstorm, Thadguidry, Lydia_Pintscher, Nemo_bis, Nintendofan885, bd808, 
Pintoch, Aklapper, Akuckartz, darthmon_wmde, Nandana, skpuneethumar, Zylc, 
1978Gage2001, Lahi, Gq86, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, 
QZanden, Tbscho, Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, 
Scott_WUaS, Luke081515, Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, 
Nikerabbit, scfc, coren, Mbch331, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[Bstorm]

Bstorm added a comment.


  In T257405#6505154 , 
@Pintoch wrote:
  
  > Yesterday the `https://wdreconcile.toolforge.org/` service returned 504 
errors for all URLs (after a waiting time - they did not appear "cached" as in 
this ticket). Restarting the service seems to temporarily mitigate the issue, 
so it's probably a different sort of outage. I do not commit to restarting the 
service when this happens - I no longer receive notifications when the service 
is down. If people want to volunteer to do it, I can give them access to the 
tool. To restart the service, a single command is needed: `webservice --backend 
kubernetes python3.7 restart`. It is very simple!
  
  Does it use tools-redis by chance? That had about 10 min of downtime during 
migrations, in case that helps anyone who decides to troubleshoot.

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Bstorm
Cc: Bstorm, Thadguidry, Lydia_Pintscher, Nemo_bis, Nintendofan885, bd808, 
Pintoch, Aklapper, Akuckartz, darthmon_wmde, Nandana, skpuneethumar, Zylc, 
1978Gage2001, Lahi, Gq86, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, 
QZanden, Tbscho, Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, 
Scott_WUaS, Luke081515, Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, 
Nikerabbit, scfc, coren, Mbch331, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[Pintoch]

Pintoch added a comment.


  Yesterday the `https://wdreconcile.toolforge.org/` service returned 504 
errors for all URLs (after a waiting time - they did not appear "cached" as in 
this ticket). Restarting the service seems to temporarily mitigate the issue, 
so it's probably a different sort of outage. I do not commit to restarting the 
service when this happens - I no longer receive notifications when the service 
is down. If people want to volunteer to do it, I can give them access to the 
tool. To restart the service, a single command is needed: `webservice --backend 
kubernetes python3.7 restart`. It is very simple!

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pintoch
Cc: Thadguidry, Lydia_Pintscher, Nemo_bis, Nintendofan885, bd808, Pintoch, 
Aklapper, Akuckartz, darthmon_wmde, Nandana, skpuneethumar, Zylc, Bstorm, 
1978Gage2001, Lahi, Gq86, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, 
QZanden, Tbscho, Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, 
Scott_WUaS, Luke081515, Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, 
Nikerabbit, scfc, coren, Mbch331, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[Pintoch]

Pintoch added a comment.


  I realized today that I have uptime statistics for this service since I have 
been monitoring it for a few years (with downnotifier.com).
  
  | Year | Nb of outages | Uptime|
  | 2018 | 67| 99.79%|
  | 2019 | 324   | 99.09%|
  | 2020 | 241 so far| 98.02% so far |
  |
  
  These are obtained by polling the service every 10 minutes with the same 
query (and retrying every minute in case of failure).
  The large majority of the outages detected by this polling strategy are 502 
errors which last for a few minutes.
  
  In comparison, a simple deployment of the service on 
https://wikidata.reconci.link/ (with just Apache as a reverse proxy to the 
tool) achieves 99.99+% availability so far (the only outage registered so far 
is when the service was initially set up). So perhaps the complexity of the 
deployment in toolforge comes at a performance price? I would not be able to 
pin down where sadly…

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pintoch
Cc: Thadguidry, Lydia_Pintscher, Nemo_bis, Nintendofan885, bd808, Pintoch, 
Aklapper, Akuckartz, darthmon_wmde, Nandana, skpuneethumar, Zylc, Bstorm, 
1978Gage2001, Lahi, Gq86, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, 
QZanden, Tbscho, Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, 
Scott_WUaS, Luke081515, Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, 
Nikerabbit, scfc, coren, Mbch331, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[Pintoch]

Pintoch added a comment.


  Thanks a lot @Lydia_Pintscher !
  
  I have also used this opportunity to optimize the service and deploy it with 
ASGI, which Toolforge does not support as far as I know. Perhaps this is an 
indication that this should rather be a Cloud VPS project, where we would be 
more in control of the deployment. But this would still live behind an HTTPS 
proxy. At the moment my priority is to make sure there is one reliable instance 
out there, where I can fix any issue directly.

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Pintoch
Cc: Lydia_Pintscher, Nemo_bis, Nintendofan885, bd808, Pintoch, Aklapper, 
Akuckartz, darthmon_wmde, Nandana, skpuneethumar, Zylc, Bstorm, 1978Gage2001, 
Lahi, Gq86, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, QZanden, Tbscho, 
Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, Scott_WUaS, Luke081515, 
Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, Nikerabbit, scfc, coren, 
Mbch331, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] T257405: wdreconcile.toolforge.org acting as though HTTP 502 Gateway errors are cached

[Lydia_Pintscher]

Lydia_Pintscher added a project: Wikidata.
Lydia_Pintscher added a comment.


  This is a rather important service for Wikidata so it'd be <3 if we could 
figure this out.

TASK DETAIL
  https://phabricator.wikimedia.org/T257405

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lydia_Pintscher
Cc: Lydia_Pintscher, Nemo_bis, Nintendofan885, bd808, Pintoch, Aklapper, 
Akuckartz, darthmon_wmde, Nandana, skpuneethumar, Zylc, Bstorm, 1978Gage2001, 
Lahi, Gq86, GoranSMilovanovic, DSquirrelGM, Chicocvenancio, QZanden, Tbscho, 
Freddy2001, LawExplorer, JJMC89, _jensen, rosalieper, Scott_WUaS, Luke081515, 
Wikidata-bugs, Jitrixis, aude, Gryllida, jayvdb, Nikerabbit, scfc, coren, 
Mbch331, Krenair
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs