[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change #966921 **merged** by Bking: [operations/deployment-charts@master] dse-k8s: don't watch rdf-streaming-updater namespace https://gerrit.wikimedia.org/r/966921 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change #966902 **merged** by jenkins-bot: [operations/deployment-charts@master] dse-k8s: remove rdf-streaming-updater service https://gerrit.wikimedia.org/r/966902 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking closed this task as "Resolved". bking moved this task from In Progress to Done on the Data-Platform-SRE board. bking added a comment. Apologies for the confusion. We have already migrated the rdf-streaming-updater to production, so I'm closing this ticket (which is focused on staging)

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 967229 **merged** by jenkins-bot: [operations/deployment-charts@master] rdf-streaming-updater: update values for application mode https://gerrit.wikimedia.org/r/967229 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 978639 **merged** by Bking: [operations/puppet@production] flink-zk: Add codfw flink-zk cluster info https://gerrit.wikimedia.org/r/978639 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 978639 had a related patch set uploaded (by Bking; author: Bking): [operations/puppet@production] flink-zk: Add codfw flink-zk cluster info https://gerrit.wikimedia.org/r/978639 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 978634 **merged** by Bking: [operations/puppet@production] flink-zk: Activate codfw hosts https://gerrit.wikimedia.org/r/978634 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 978634 had a related patch set uploaded (by Bking; author: Bking): [operations/puppet@production] flink-zk: Activate codfw hosts https://gerrit.wikimedia.org/r/978634 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 978617 **merged** by jenkins-bot: [operations/deployment-charts@master] admin_ng: tell flink-operator to listen to rdf-streaming-updater ns https://gerrit.wikimedia.org/r/978617 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 978617 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] admin_ng: tell flink-operator to listen to rdf-streaming-updater ns https://gerrit.wikimedia.org/r/978617 TASK DETAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 975289 **merged** by Bking: [operations/deployment-charts@master] staging-eqiad: raise rdf-streaming-updater quota https://gerrit.wikimedia.org/r/975289 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 975289 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] staging-eqiad: raise rdf-streaming-updater quota https://gerrit.wikimedia.org/r/975289 TASK DETAIL https://phabricator.wikimedia.org/T349095

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a comment. Both apps (commons and wikidata) are stable in staging-eqiad now: bking@deploy2002:~/deployment-charts$ kubectl get flinkdeployments.flink.apache.org NAME JOB STATUS LIFECYCLE STATE flink-app-commonsRUNNING STABLE

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 973242 **merged** by jenkins-bot: [operations/deployment-charts@master] staging-eqiad: raise rdf-streaming-updater quota https://gerrit.wikimedia.org/r/973242 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 973242 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] staging-eqiad: raise rdf-streaming-updater quota https://gerrit.wikimedia.org/r/973242 TASK DETAIL https://phabricator.wikimedia.org/T349095

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 972483 **merged** by jenkins-bot: [operations/deployment-charts@master] staging-eqiad: raise rdf-streaming-updater quota https://gerrit.wikimedia.org/r/972483 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 972483 **restored** by Bking: [operations/deployment-charts@master] staging-eqiad: raise rdf-streaming-updater quota https://gerrit.wikimedia.org/r/972483 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 972483 **abandoned** by Bking: [operations/deployment-charts@master] staging-eqiad: raise rdf-streaming-updater quota Reason: superseded by changes in 967229 https://gerrit.wikimedia.org/r/972483 TASK DETAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 972483 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] staging-eqiad: raise rdf-streaming-updater quota https://gerrit.wikimedia.org/r/972483 TASK DETAIL https://phabricator.wikimedia.org/T349095

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a comment. Current status: flink-operator is listening for rdf-streaming-updater rdf-streaming-updater job deploys, but it seems like it can't connect to kafka: {"@timestamp":"2023-11-06T23:03:13.111Z","log.level": "INFO","message":"[AdminClient

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 972005 **merged** by Bking: [operations/deployment-charts@master] rbac: permit deploy-flink user to create flinkdeployments https://gerrit.wikimedia.org/r/972005 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 972005 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] rbac: permit deploy-flink user to create flinkdeployments https://gerrit.wikimedia.org/r/972005 TASK DETAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 971221 **merged** by jenkins-bot: [operations/deployment-charts@master] admin_ng: Activate flink-operator for rdf-streaming-updater https://gerrit.wikimedia.org/r/971221 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

Gehel triaged this task as "Medium" priority. TASK DETAIL https://phabricator.wikimedia.org/T349095 WORKBOARD https://phabricator.wikimedia.org/project/board/6524/ EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, Gehel Cc: pfischer,

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 971221 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] admin_ng: Activate flink-operator for rdf-streaming-updater https://gerrit.wikimedia.org/r/971221 TASK DETAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

EBernhardson changed the point value for this task from "8" to "13". TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, EBernhardson Cc: pfischer, EBernhardson, dcausse, BTullis, Aklapper,

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

EBernhardson set the point value for this task to "8". TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, EBernhardson Cc: pfischer, EBernhardson, dcausse, BTullis, Aklapper, bking,

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

EBernhardson moved this task from Incoming to Current work on the Wikidata-Query-Service board. EBernhardson added a project: Discovery-Search (Current work). TASK DETAIL https://phabricator.wikimedia.org/T349095 WORKBOARD https://phabricator.wikimedia.org/project/board/891/ EMAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 967229 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] rdf-streaming-updater: update staging values https://gerrit.wikimedia.org/r/967229 TASK DETAIL https://phabricator.wikimedia.org/T349095

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 966921 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] dse-k8s: don't watch rdf-streaming-updater namespace https://gerrit.wikimedia.org/r/966921 TASK DETAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a project: Patch-For-Review. TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, gerritbot Cc: pfischer, EBernhardson, dcausse, BTullis, Aklapper, bking, Danny_Benjafield_WMDE,

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

gerritbot added a comment. Change 966902 had a related patch set uploaded (by Bking; author: Bking): [operations/deployment-charts@master] dse-k8s: remove rdf-streaming-updater service https://gerrit.wikimedia.org/r/966902 TASK DETAIL https://phabricator.wikimedia.org/T349095

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

Stashbot added a comment. Mentioned in SAL (#wikimedia-operations) [2023-10-18T15:43:33Z] bking@deploy2002 destroy dse-k8s-services instance of rdf-streaming-updater T349095 TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a comment. Create a savepoint by incrementing the nonce value in the helmfile.d/dse-k8s-services/values.yaml and deploy Destroy the deployment on the dse-k8s cluster /srv/deployment-charts/helmfile.d/dse-k8s-services/rdf-streaming-updater/$ helmfile -e

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a comment. We could also deploy via a new namespace, but I wonder what implications that would have for our monitoring/tooling etc. Open to feedback/suggestions on this one. TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

BTullis added a comment. @bking and I have been discussing this and we think that the best course of action would be to deploy this in multiple steps. e.g. Something like: - Create a savepoint by incrementing the nonce value in the `helmfile.d/dse-k8s-services/values.yaml` and deploy

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking added a parent task: T326409: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model. TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking Cc: Aklapper,

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

Maintenance_bot added a project: Wikidata. TASK DETAIL https://phabricator.wikimedia.org/T349095 EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/ To: bking, Maintenance_bot Cc: Aklapper, bking, Danny_Benjafield_WMDE, Astuthiodit_1, AWesterinen, BTullis,

[Wikidata-bugs] [Maniphest] T349095: Migrate staging rdf-streaming-updater to flink operator

bking created this task. bking added projects: Wikidata-Query-Service, Data-Platform-SRE. Restricted Application added a subscriber: Aklapper. TASK DESCRIPTION We have tested the flink operator mode in dse-k8s . Our next step is to migrate the