[ https://issues.apache.org/jira/browse/FLINK-11457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758100#comment-16758100 ]
Oscar Westra van Holthe - Kind commented on FLINK-11457: -------------------------------------------------------- Seems sensible. Pity there's no sufficient coordination for initiialization (yet?) despite the active jobmanager coordinating the job execution by the taskmanagers. Edit: adjusted the subject & description to account for the discussion in the [PR|https://github.com/apache/flink/pull/5857] for FLINK-9187. What's left is the part where the current implementation goes wrong. > PrometheusPushGatewayReporter does not cleanup its metrics > ---------------------------------------------------------- > > Key: FLINK-11457 > URL: https://issues.apache.org/jira/browse/FLINK-11457 > Project: Flink > Issue Type: Bug > Reporter: Oscar Westra van Holthe - Kind > Priority: Major > > When cancelling a job running on a yarn based cluster and then shutting down > the cluster, metrics on the push gateway are not deleted. > My yarn-conf.yaml settings: > {code:yaml} > metrics.reporters: promgateway > metrics.reporter.promgateway.class: > org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter > metrics.reporter.promgateway.host: pushgateway.gcpstg.bolcom.net > metrics.reporter.promgateway.port: 9091 > metrics.reporter.promgateway.jobName: PSMF > metrics.reporter.promgateway.randomJobNameSuffix: true > metrics.reporter.promgateway.deleteOnShutdown: true > metrics.reporter.promgateway.interval: 30 SECONDS > {code} > What I expect to happen: > * when running, the metrics are pushed to the push gateway to a separate > label per node (jobmanager/taskmanager) > * when shutting down, the metrics are deleted from the push gateway > This last bit does not happen. > How the job is run: > {code}flink run -m yarn-cluster -yn 5 -ys 2 -yst > "$INSTALL_DIRECTORY/app/psmf.jar"{code} > How the job is stopped: > {code} > YARN_APP_ID=$(yarn application -list | grep "PSMF" | awk '{print $1}') > FLINK_JOB_ID=$(flink list -r -yid ${YARN_APP_ID} | grep "PSMF" | awk '{print > $4}') > flink cancel -s "${SAVEPOINT_DIR%/}/" -yid "${YARN_APP_ID}" "${FLINK_JOB_ID}" > echo "stop" | yarn-session.sh -id ${YARN_APP_ID} > {code} > Is there anything I'm sdoing wrong? Anything I can help to fix? -- This message was sent by Atlassian JIRA (v7.6.3#76005)