[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-24 Thread taavi 
taavi closed this task as "Resolved".
taavi claimed this task.

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: taavi
Cc: TheresNoTime, Lucas_Werkmeister_WMDE, Aklapper, Michael, gonzalez.actor, 
Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, 
CBogen, ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, 
Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, Krenair, 
jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-24 Thread Maintenance_bot 
Maintenance_bot removed a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Maintenance_bot
Cc: TheresNoTime, Lucas_Werkmeister_WMDE, Aklapper, Michael, gonzalez.actor, 
Astuthiodit_1, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, 
CBogen, ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, 
Bsandipan, GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, 
rosalieper, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, Krenair, 
jeremyb, Themindcoder, Adamm71, Jersione, Hellket777, LisafBia6531, 786, 
Biggs657, Juan90264, Alter-paule, Beast1978, Un1tY, Hook696, Kent7301, 
joker88john, CucyNoiD, Gaboe420, Giuliamocci, Cpaulf30, Af420, Lewizho99, 
Maathavan, Neuronton
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-24 Thread gerritbot 
gerritbot added a comment.


  Change 910758 **merged** by Jbond:
  
  [operations/puppet@production] tlsproxy: Fix Nginx reload when cfssl certs 
get renewed
  
  https://gerrit.wikimedia.org/r/910758

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: TheresNoTime, Lucas_Werkmeister_WMDE, Aklapper, Michael, Themindcoder, 
Adamm71, gonzalez.actor, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 
786, Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, 
Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, 
Hook696, CptViraj, Kent7301, joker88john, DannyS712, CucyNoiD, Nandana, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, Neuronton, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, 
Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  Ah, I was about to create a separate task for tracking the repeating failure, 
but nevermind :)

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, Aklapper, Michael, Themindcoder, Adamm71, 
gonzalez.actor, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, 
Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, 
Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, 
Hook696, CptViraj, Kent7301, joker88john, DannyS712, CucyNoiD, Nandana, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, TheresNoTime, Neuronton, Scott_WUaS, Wikidata-bugs, aude, jayvdb, 
Mbch331, Jay8g, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread gerritbot 
gerritbot added a project: Patch-For-Review.

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Lucas_Werkmeister_WMDE, Aklapper, Michael, Themindcoder, Adamm71, 
gonzalez.actor, Jersione, Hellket777, LisafBia6531, Astuthiodit_1, 786, 
Biggs657, karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, 
Juan90264, Alter-paule, Beast1978, CBogen, ItamarWMDE, Un1tY, Akuckartz, 
Hook696, CptViraj, Kent7301, joker88john, DannyS712, CucyNoiD, Nandana, 
Gaboe420, Giuliamocci, Cpaulf30, Lahi, Gq86, Af420, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, Lewizho99, Maathavan, _jensen, 
rosalieper, TheresNoTime, Neuronton, Scott_WUaS, Wikidata-bugs, aude, jayvdb, 
Mbch331, Jay8g, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread gerritbot 
gerritbot added a comment.


  Change 910758 had a related patch set uploaded (by Majavah; author: Majavah):
  
  [operations/puppet@production] tlsproxy: Fix Nginx reload when cfssl certs 
get renewed
  
  https://gerrit.wikimedia.org/r/910758

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: Lucas_Werkmeister_WMDE, Aklapper, Michael, gonzalez.actor, Astuthiodit_1, 
karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, CBogen, 
ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
TheresNoTime, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, Krenair, 
jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  In T335181#8798253 , 
@Lucas_Werkmeister_WMDE wrote:
  
  > I think the cert expired again (already), like in T333952#8759072 
. To check this, I SSHed 
into `deployment-elastic09.deployment-prep.eqiad1.wikimedia.cloud`, ran `sudo 
lsof -iTCP -sTCP:LISTEN -n -P` to find some port numbers the server is 
listening on, and ran `openssl s_client -connect localhost:9643`; it reports 
“notAfter=Apr 19 10:33:00 2023 GMT” for the certificate, which is about two 
days ago.
  
  (An easier way to check would have been something like `curl 
https://deployment-elastic10.deployment-prep.eqiad1.wikimedia.cloud:9643` in 
T333952#8760541 , which can 
also be run on other Beta cluster servers, e.g. deployment-mwmaint02 – the main 
information I was missing was the port number.)

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, Aklapper, Michael, gonzalez.actor, Astuthiodit_1, 
karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, CBogen, 
ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
TheresNoTime, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, Krenair, 
jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  Looks like exactly the same solution worked again 路 but it’s hardly 
sustainable if the servers need to be manually reloaded every few weeks…

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, Aklapper, Michael, gonzalez.actor, Astuthiodit_1, 
karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, CBogen, 
ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
TheresNoTime, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, Krenair, 
jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread Stashbot 
Stashbot added a comment.


  Mentioned in SAL (#wikimedia-releng) [2023-04-21T10:04:50Z]  
`sudo systemctl reload nginx` on deployment-elastic09, deployment-elastic10, 
deployment-elastic11 # T335181 

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Stashbot
Cc: Lucas_Werkmeister_WMDE, Aklapper, Michael, gonzalez.actor, Astuthiodit_1, 
karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, CBogen, 
ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
TheresNoTime, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, Krenair, 
jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  Hm, but the certificate in the nginx config (compare T333952#8760541 
) looks like it’s still 
valid:
  
lucaswerkmeister-wmde@deployment-elastic09:~$ grep -rF ssl_certificate 
/etc/nginx/
/etc/nginx/snippets/snakeoil.conf:ssl_certificate 
/etc/ssl/certs/ssl-cert-snakeoil.pem;
/etc/nginx/snippets/snakeoil.conf:ssl_certificate_key 
/etc/ssl/private/ssl-cert-snakeoil.key;
/etc/nginx/sites-available/beta-search:ssl_certificate 
/etc/cfssl/ssl/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud.chained.pem;
/etc/nginx/sites-available/beta-search:ssl_certificate_key 
/etc/cfssl/ssl/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud-key.pem;
/etc/nginx/sites-available/beta-search-omega:ssl_certificate 
/etc/cfssl/ssl/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud.chained.pem;
/etc/nginx/sites-available/beta-search-omega:ssl_certificate_key 
/etc/cfssl/ssl/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud-key.pem;
/etc/nginx/sites-available/beta-search-psi:ssl_certificate 
/etc/cfssl/ssl/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud.chained.pem;
/etc/nginx/sites-available/beta-search-psi:ssl_certificate_key 
/etc/cfssl/ssl/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud-key.pem;
lucaswerkmeister-wmde@deployment-elastic09:~$ sudo cat 
/etc/cfssl/ssl/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud/discovery__deployment-elastic09_deployment-prep_eqiad1_wikimedia_cloud.chained.pem
 | openssl x509 -noout -text | grep -A2 Validity
Validity
Not Before: Apr  8 10:03:00 2023 GMT
Not After : May  6 10:03:00 2023 GMT

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, Aklapper, Michael, gonzalez.actor, Astuthiodit_1, 
karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, CBogen, 
ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
TheresNoTime, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, Krenair, 
jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread Lucas_Werkmeister_WMDE 
Lucas_Werkmeister_WMDE added a comment.


  I think the cert expired again (already), like in T333952#8759072 
. To check this, I SSHed 
into `deployment-elastic09.deployment-prep.eqiad1.wikimedia.cloud`, ran `sudo 
lsof -iTCP -sTCP:LISTEN -n -P` to find some port numbers the server is 
listening on, and ran `openssl s_client -connect localhost:9643`; it reports 
“notAfter=Apr 19 10:33:00 2023 GMT” for the certificate, which is about two 
days ago.

TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Lucas_Werkmeister_WMDE
Cc: Lucas_Werkmeister_WMDE, Aklapper, Michael, gonzalez.actor, Astuthiodit_1, 
karapayneWMDE, Invadibot, MPhamWMF, maantietaja, Wilmanbeno, CBogen, 
ItamarWMDE, Akuckartz, CptViraj, DannyS712, Nandana, Lahi, Gq86, Bsandipan, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, _jensen, rosalieper, 
TheresNoTime, Scott_WUaS, Wikidata-bugs, aude, jayvdb, Mbch331, Jay8g, Krenair, 
jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org

[Wikidata-bugs] [Maniphest] T335181: Cirrus search is broken on beta (April 2023, second occurence)

2023-04-21 Thread Michael 
Michael created this task.
Michael added projects: CirrusSearch, Wikidata, Beta-Cluster-Infrastructure.
Restricted Application added a subscriber: Aklapper.
Restricted Application added projects: Discovery-Search, wdwb-tech.

TASK DESCRIPTION
  It seems that cirrus search is broken again on beta (at least on 
betawikidata).
  
  The following API request:
  
https://wikidata.beta.wmflabs.org/w/api.php?action=wbsearchentities=string=json=plaintext=en=en=property
  
  gives me this error:
  
{
   "errors":[
  {
 "code":"cirrussearch-backend-error",
 "module":"wbsearchentities",
 "*":"We could not complete your search due to a temporary problem. 
Please try again later."
  }
   ],
   "servedby":"deployment-mediawiki11",
   "*":"See https://wikidata.beta.wmflabs.org/w/api.php for API usage. 
Subscribe to the mediawiki-api-announce mailing list at 
https://lists.wikimedia.org/postorius/lists/mediawiki-api-announce.lists.wikimedia.org/;
 for notice of API deprecations and breaking changes."
}
  
  Note that there was a similar error earlier this month, not sure if that is 
related: T333952: cirrussearch-backend-error on beta cluster 


TASK DETAIL
  https://phabricator.wikimedia.org/T335181

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Michael
Cc: Aklapper, Michael, gonzalez.actor, Astuthiodit_1, karapayneWMDE, Invadibot, 
MPhamWMF, maantietaja, Wilmanbeno, CBogen, ItamarWMDE, Akuckartz, CptViraj, 
DannyS712, Nandana, Lahi, Gq86, Bsandipan, GoranSMilovanovic, QZanden, EBjune, 
LawExplorer, _jensen, rosalieper, TheresNoTime, Scott_WUaS, Wikidata-bugs, 
aude, jayvdb, Mbch331, Jay8g, Krenair, jeremyb
___
Wikidata-bugs mailing list -- wikidata-bugs@lists.wikimedia.org
To unsubscribe send an email to wikidata-bugs-le...@lists.wikimedia.org