[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

gerritbot added a comment. Change 823766 **abandoned** by Ori: [operations/puppet@production] beta cluster: don't instantiate ::esitest Reason: I think this is obsoleted by https://gerrit.wikimedia.org/r/c/operations/puppet/+/824146 , and indeed it's no longer cherry-picked on

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

jbond closed subtask T315394: Remove two cherry-picked reverts from deployment-puppetmaster04 as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: jbond Cc: jnuche, jbond, thcipriani, or

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

Zabe closed subtask T315351: Evaluation Error on deployment-cache-text06 puppet run as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Zabe Cc: jnuche, jbond, thcipriani, ori, Zabe, A

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

RhinosF1 added a project: SRE-OnFire. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RhinosF1 Cc: jnuche, jbond, thcipriani, ori, Zabe, ArielGlenn, bking, RhinosF1, Ryasmeen, matmarex, ppelberg,

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

RhinosF1 added a project: Wikimedia-Incident. RhinosF1 added a comment. https://wikitech.wikimedia.org/wiki/Incidents/2022-08-16_Beta_Cluster_502 TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To:

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

Zabe added a comment. Ok, lemme try to quickly summarize what happened and what was done. Some cloudvirts hosts got accidentally rebooted which caused deployment-prep to go offline and it did not came back up by itself. Request from - via deployment-cache-text06.deployment-prep.eqi

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

jbond added a comment. > Correct, deployment-deploy03 is the beta deployment server. The failures started right after the Puppet changes in deployment-puppetmaster04 Ack i suspect that could be because ori fixed the merge conflicts which brought in a bunch more changes > Yes, sorry,

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

jnuche added a comment. > Just to confirm puppet was failing on deployment-deploy03 not on deployment-puppetmaster04? Correct, `deployment-deploy03` is the beta deployment server. The failures started right after the Puppet changes in `deployment-puppetmaster04` > this should have b

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

jbond added a comment. In T315350#8161180 , @jnuche wrote: > Hi there. JFYI, after the changes early this morning in `deployment-puppetmaster04` Puppet started failing on the beta deployment server: Just to confirm puppet was

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

jnuche added a comment. Hi there. JFYI, after the changes early this morning in `deployment-puppetmaster04` Puppet started failing on the beta deployment server: Aug 17 00:23:40 deployment-deploy03 puppet-agent[13622]: Could not retrieve catalog from remote server: Error 500 on SERVER

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

gerritbot added a comment. Change 824146 had a related patch set uploaded (by Jbond; author: jbond): [operations/puppet@production] P:cache::varnish::frontend::text: make esitest ensurable and absent on cloud https://gerrit.wikimedia.org/r/824146 TASK DETAIL https://phabricator.wi

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

Zabe closed subtask T315379: (Beta cluster) Running logspam-watch on deployment-mwlog01 gives repeated `Use of uninitialized value $host` errors as "Resolved". TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailprefe

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime added a subtask: T315395: Rebase & merge or re-cherry-pick 668701 on deployment-puppetmaster04. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: TheresNoTime Cc: thcipriani, ori, Zabe,

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime added a subtask: T315394: Remove two cherry-picked reverts from deployment-puppetmaster04. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: TheresNoTime Cc: thcipriani, ori, Zabe, Arie

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

ori added a comment. Follow-up items to get the Puppet repo on deployment-puppetmaster04 in good shape: - The two cherry-picked reverts should be removed (https://gerrit.wikimedia.org/r/c/operations/puppet/+/823638, https://gerrit.wikimedia.org/r/c/operations/puppet/+/823639) and the ch

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime lowered the priority of this task from "Unbreak Now!" to "Medium". TheresNoTime updated the task description. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: TheresNoTime Cc: thciprian

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime added a subscriber: thcipriani. TheresNoTime added a comment. [2022-08-17T00:49:52Z, #wikimedia-releng, @thcipriani] what I've managed to find: it seems like mediawiki is working locally on the MediaWiki hosts. But I get timeouts on the -cache-text server. Traffic server

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: gerritbot Cc: ori, Zabe, ArielGlenn, bking, RhinosF1, Ryasmeen, matmarex, ppelberg, Daimona, TheresNoTi

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

gerritbot added a comment. Change 823766 had a related patch set uploaded (by Ori; author: Ori): [operations/puppet@production] beta cluster: don't instantiate ::esitest https://gerrit.wikimedia.org/r/823766 TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime added a comment. [2022-08-17T00:43:52Z, #wikimedia-releng, @ori] OK I hacked in locally `if $::realm != 'labs' {}` around the ::esitest in profile/manifests/cache/varnish/frontend/text.pp I'll turn it into a patch shortly puppet ran successfully on deployment-cache-tex

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime closed subtask T315355: (Beta cluster) Wikimedia\RequestTimeout\RequestTimeoutException: The maximum execution time of 200 seconds was exceeded as "Invalid". TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime closed subtask T315354: (Beta Cluster) Unexpected connection error communicating with Elasticsearch. Curl code: {curl_code} as "Invalid". TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ T

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime added a subtask: T315379: (Beta cluster) Running logspam-watch on deployment-mwlog01 gives repeated `Use of uninitialized value $host` errors. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreference

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime added subscribers: Zabe, ori. TheresNoTime added a comment. [2022-08-16T23:25:49Z, #wikimedia-releng, @Zabe] I try to understand why apache is not willing to start AH00526: Syntax error on line 1 of /etc/apache2/conf-enabled/50-configmaster-port.conf Cannot define mult

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

Jdforrester-WMF removed a parent task: T312253: 502 errors on beta cluster. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Jdforrester-WMF Cc: ArielGlenn, bking, RhinosF1, Ryasmeen, matmarex, ppel

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

RhinosF1 added a subtask: T315351: Evaluation Error on deployment-cache-text06 puppet run . TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: RhinosF1 Cc: ArielGlenn, bking, RhinosF1, Ryasmeen, matm

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

Stashbot added a comment. Mentioned in SAL (#wikimedia-releng) [2022-08-16T20:51:48Z] beta: is down see wikitech-l and https://phabricator.wikimedia.org/T315350 TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/email

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

TheresNoTime added a project: Release-Engineering-Team. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: TheresNoTime Cc: bking, RhinosF1, Ryasmeen, matmarex, ppelberg, Daimona, TheresNoTime, Aklap

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

Maintenance_bot added a project: Wikidata. Restricted Application added a project: wdwb-tech. TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: bking, RhinosF1, Ryasmeen, matmarex

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

bking added a comment. Brian (Search platform SRE) here. Apologies for the confusion: to the best of my knowledge, there is no WDQS in beta cluster. I did take a look at the Elasticsearch beta cluster instances and I found that the cluster status is healthy and the SSL certificates

[Wikidata-bugs] [Maniphest] T315350: Known, Beta cluster Error: 502, Next Hop Connection Failed

RhinosF1 added projects: Wikidata-Query-Service, Discovery-Search. RhinosF1 added a comment. Hi Search Team, can you help with the WDQS errors? TASK DETAIL https://phabricator.wikimedia.org/T315350 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: R