rahil-c opened a new issue, #19740: URL: https://github.com/apache/hudi/issues/19740
### Describe the problem you faced Record index lookup counters collected on executors are published to the configured metrics reporter at the commit boundary. This works on the Spark DataSource and DeltaStreamer write paths, but on the Spark SQL DML path (`UPDATE`, `MERGE INTO`) the counters never reach the reporter. The collection half is not at fault. The same counters were previously observable for these writes, so the executor-to-driver merge and the driver-side accumulation both work on this path. What does not happen is the final step, where the drain hands the values to `HoodieMetrics`. For contrast, the stock write metrics registered by `HoodieMetrics.updateCommitMetrics` (`commit.totalRecordsWritten`, `commit.duration`, and so on) are present in the registry for the same SQL write, so this is specific to the executor-metrics drain rather than to metrics being off for the path. ### To Reproduce 1. Create a table with the record index enabled and a metrics reporter configured. 2. Seed it, then run `UPDATE <table> SET <col> = <value>` with `hoodie.spark.sql.optimized.writes.enable=false`, or a `MERGE INTO` against it. 3. Read the reporter. The `<table>.rli.lookup.*` gauges are absent, while the `<table>.commit.*` gauges for the same commit are present. Two tests reproducing this are checked in and currently disabled, in `TestRliLookupMetricsOnSparkSql`: - `testUpdateWithoutOptimizedWritesPublishesCounters` - `testMergeIntoPublishesCounters` ### Expected behavior A Spark SQL write that tags records should publish the same record index lookup counters as the equivalent DataSource write. ### Environment Description * Hudi version: 1.3.0-SNAPSHOT (master) * Spark version: 3.5 * Running on Docker? : no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
