skywalker0618 opened a new pull request, #20499: URL: https://github.com/apache/hudi/pull/20499
### Describe the issue this Pull Request addresses The Hive sync metrics of the Flink write coordinator stop updating after a restart that recreates the coordinator, such as any restart or rescale under the adaptive scheduler. Flink keeps one coordinator metric group per operator for the life of the job and reuses it for the new coordinator. When the new `StreamWriteOperatorCoordinator` registers its own `FlinkHiveSyncMetrics` under the same names, Flink ignores them with a "Name collision" warning. The group keeps reporting the first coordinator's metrics, which nothing updates anymore, so `hiveSyncSuccessCount` and the other metrics freeze at their values from before the restart. ### Summary and Changelog Every coordinator of the same write operator in the same job now shares one `FlinkHiveSyncMetrics` instance, so syncs after a restart update the metrics the group already reports, and the counts carry on across the restart. - Add `FlinkHiveSyncMetrics.register(MetricGroup, boolean)`. It returns the instance kept for the group's `<job_id>` and `<operator_id>` and registers it on the group. A group that does not identify them, as in tests, gets an instance of its own. - `StreamWriteOperatorCoordinator` gets its metrics through `register`. - Add `SharedCoordinatorMetricGroup`, a test metric group that keeps the first metric registered under a name, as Flink does, and that several coordinators can share. Add tests for sharing and isolation in `TestFlinkHiveSyncMetrics`, and a test in `TestStreamWriteOperatorCoordinator` that runs two coordinators in turn on one group. Without the fix, the second test reports 1 failed sync instead of 2. ### Impact The Hive sync metrics keep reporting after the coordinator is recreated. No public API or config changes. ### Risk Level low. Only how the Hive sync metrics are created and registered changes. Each Hudi write operator in a job keeps one small instance on the JobManager for the life of the JVM. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
