rangareddy opened a new issue, #19507:
URL: https://github.com/apache/hudi/issues/19507
## Describe the problem
`HoodieMetadataMetrics` registers gauges whose names do not carry the table
prefix that every other Hudi metric carries, and in some cases carry no table
name at all. This is the producer-side cause of the CloudWatch failures in
#12182 / #13051.
`HoodieMetadataMetrics` receives a `HoodieMetricsConfig` and discards it:
```java
public HoodieMetadataMetrics(HoodieMetricsConfig metricsConfig,
HoodieStorage storage, boolean detailedMetricsEnabled) {
this.metrics = Metrics.getInstance(metricsConfig, storage); // config
not retained
```
so `setMetric`/`incrementMetric` call `metrics.registerGauge(action, value)`
directly, bypassing the `hoodie.metrics.reporter.metricsname.prefix` that
`Metrics#registerGauges` applies. Two consequences:
**1. Names with no dot at all** — nothing can derive a table from them:
- `lookup_meta_index_bloom_filters_file_count` —
`BaseTableMetadata.java:212`, on the normal bloom-index read path
- `<partition>_bootstrap_error` — `HoodieBackedTableMetadataWriter.java:482`
**2. Names with a dot but no table** — a consumer that derives a table from
the first segment gets the wrong answer:
- `<action>.count` / `<action>.totalDuration` —
`HoodieMetadataMetrics.java:156-157`, giving `initialize.count`,
`lookup_partitions.totalDuration`, ...
-
`<mdt_partition>.{baseFileCount,logFileCount,totalBaseFileSizeInBytes,totalLogFileSizeInBytes}`
— `HoodieMetadataMetrics.java:137-140`, giving `files.baseFileCount`,
`column_stats.logFileCount`, ...
These are registered in practice — `TestJavaHoodieBackedMetadata` asserts
the registry contains `initialize.count`, `files.baseFileCount` and friends.
## Why it matters beyond CloudWatch
`AWSGlueCatalogSyncClient`... (no) — `CloudWatchReporter` derives its
`Table` dimension by splitting the metric name on the first dot, so category 1
cannot be reported at all and category 2 is reported under a wrong `Table`.
CloudWatch bills per unique metric name plus dimension set, so the
wrong-dimension metrics are billed as distinct custom metrics.
Other reporters are unaffected in behaviour but still emit inconsistent
names.
## Suggested direction
Retain the `HoodieMetricsConfig` in `HoodieMetadataMetrics` and apply
`getMetricReporterMetricsNamePrefix()` in `setMetric` and `incrementMetric`.
That fixes both categories, for every reporter at once, in one place.
**This renames metrics**, so it breaks existing Graphite / Prometheus / JMX
/ Datadog dashboards and needs a release note — which is why #19476
deliberately did not do it. Prior fixes for the same symptom went producer-side
(`1a5a9f7f03ec` [HUDI-4439] #6164, and `100e9ac47590` [HUDI-9068] #12873), so
this is the direction with precedent.
## Context
Raised from review on #19476, which makes `CloudWatchReporter` stop losing
an entire batch when it meets one of these names. That is the "stop the
bleeding" half; this issue is the cause.
Credit to @voonhous for tracing the producer sites and the billing
implication.
## Related
- #19476
- #12182, #13051
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
