[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

JMeybohm closed this task as "Resolved". JMeybohm added a comment. Thanks, closing then. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm Cc: JMeybohm, dcausse, Aklapper, Biggs657,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

dcausse moved this task from In Progress to Needs Reporting on the Discovery-Search (Current work) board. dcausse added a comment. seems to be fixed now by providing explicit K8S client env. TASK DETAIL https://phabricator.wikimedia.org/T287443 WORKBOARD

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708495 **merged** by jenkins-bot: [operations/deployment-charts@master] flink-session-cluster: Use the wmf.kubernetes.ApiEnv template https://gerrit.wikimedia.org/r/708495 TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm, gerritbot Cc: JMeybohm, dcausse, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708495 had a related patch set uploaded (by DCausse; author: DCausse): [operations/deployment-charts@master] flink-session-cluster: Use the wmf.kubernetes.ApiEnv template https://gerrit.wikimedia.org/r/708495 TASK DETAIL

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm, Maintenance_bot Cc: JMeybohm, dcausse, Aklapper, Invadibot, MPhamWMF, maantietaja,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708489 **merged** by JMeybohm: [operations/puppet@production] deployment_server: Add defaults for kubernetes apiserver https://gerrit.wikimedia.org/r/708489 TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708486 **abandoned** by DCausse: [operations/deployment-charts@master] rdf-streaming-updater: Declare kubernetesApi for staging Reason: superseded by https://gerrit.wikimedia.org/r/c/operations/puppet/+/708489

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708489 had a related patch set uploaded (by JMeybohm; author: JMeybohm): [operations/puppet@production] deployment_server: Add defaults for kubernetes apiserver https://gerrit.wikimedia.org/r/708489 TASK DETAIL

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708486 had a related patch set uploaded (by DCausse; author: DCausse): [operations/deployment-charts@master] rdf-streaming-updater: Declare kubernetesApi for staging https://gerrit.wikimedia.org/r/708486 TASK DETAIL

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708477 **merged** by jenkins-bot: [operations/deployment-charts@master] flink-session-cluster: use kubernetesApiEnv when available https://gerrit.wikimedia.org/r/708477 TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm, gerritbot Cc: JMeybohm, dcausse, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708477 had a related patch set uploaded (by DCausse; author: DCausse): [operations/deployment-charts@master] flink-session-cluster: use kubernetesApiEnv when available https://gerrit.wikimedia.org/r/708477 TASK DETAIL

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

JMeybohm added a comment. That is because your application is reading default kubernetes environment variables which carry the ClusterIP of `kubernetes.default.svc.cluster.local` instead of it's name. The ClusterIP we unfortunately don't have in the certificate on the actual servers.

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm, Maintenance_bot Cc: JMeybohm, dcausse, Aklapper, Invadibot, MPhamWMF, maantietaja,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708471 **merged** by jenkins-bot: [operations/deployment-charts@master] rdf-streaming-updater: Disable hostname verif from the k8s client https://gerrit.wikimedia.org/r/708471 TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm, gerritbot Cc: JMeybohm, dcausse, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708471 had a related patch set uploaded (by DCausse; author: DCausse): [operations/deployment-charts@master] rdf-streaming-updater: Disable hostname verif from the k8s client https://gerrit.wikimedia.org/r/708471 TASK DETAIL

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

dcausse added a comment. Will set `kubernetes.disable.hostname.verification` to true for the k8s client for now to unblock this. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

dcausse moved this task from To Be Deployed to In Progress on the Discovery-Search (Current work) board. dcausse added a comment. I'm now getting: {"@timestamp":"2021-07-27T16:59:20,553","log.level":"ERROR","message":"Exception occurred while acquiring lock 'ConfigMapLock:

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

Maintenance_bot removed a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm, Maintenance_bot Cc: JMeybohm, dcausse, Aklapper, Invadibot, MPhamWMF, maantietaja,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708315 **merged** by jenkins-bot: [operations/deployment-charts@master] rdf-streaming-updater: Allow egress to kubernetes api servers https://gerrit.wikimedia.org/r/708315 TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: JMeybohm, gerritbot Cc: JMeybohm, dcausse, Aklapper, Biggs657, Invadibot, Lalamarie69, MPhamWMF,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

gerritbot added a comment. Change 708315 had a related patch set uploaded (by JMeybohm; author: JMeybohm): [operations/deployment-charts@master] rdf-streaming-updater: Allow egress to kubernetes api servers https://gerrit.wikimedia.org/r/708315 TASK DETAIL

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

JMeybohm claimed this task. JMeybohm added a comment. Looking into this. Problem is that we currently do not allow Pods to access the Kubernetes API servers (Egress rule is missing) and it's not super trivial to allow that in a transparent way (e.g. without having to declare the API

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

dcausse added a project: Discovery-Search (Current work). dcausse triaged this task as "High" priority. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: dcausse Cc: JMeybohm, dcausse, Aklapper,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

Maintenance_bot added a project: Wikidata. Restricted Application added a project: wdwb-tech. TASK DETAIL https://phabricator.wikimedia.org/T287443 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: Maintenance_bot Cc: JMeybohm, dcausse, Aklapper,

[Wikidata-bugs] [Maniphest] T287443: Flink jobmanager and taskmanager cannot talk to the k8s api server

dcausse created this task. dcausse added projects: Wikidata-Query-Service, serviceops. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION Seen on k8s staging when the jobmanager tries to look up for its leader election config maps: