nsivabalan opened a new pull request, #20224: URL: https://github.com/apache/hudi/pull/20224
### Describe the issue this Pull Request addresses Hudi labels its Spark jobs. `HoodieEngineContext#setJobStatus` sets a job description on each significant step of a write, so someone reading the Spark UI can tell that a stage is index tagging rather than, say, marker reconciliation. There are roughly fifty of these labels across the write path, and they are genuinely useful. Five paths that do real work are missing one. Their stages appear in the UI with only Spark's own names — `mapToPair at HoodieJavaRDD.java:177`, `collect at SparkRDDWriteClient.java:441` — which says nothing about what Hudi was doing. And because Spark evaluates lazily, an untagged step's cost lands under whichever description happened to be set last, so it is not merely invisible: it is attributed to something else. I found these while analysing event logs from local Hudi writes across several configurations. ### Summary and Changelog Adds a job description to five paths. Eight lines, four files, all observational — no work is added, no configuration changes, and the writes are untouched. **`BaseWriteHelper`, before deduplication.** The class sets `Tagging:` before the index lookup but nothing before the dedup that precedes it, so dedup cost is reported as tagging. Now `Deduplicating records:`. **`SparkMetadataTableGlobalRecordLevelIndex#tagLocation`.** Record-index lookup inherits the description set before it, which is `Building workload profile:`. On a table using `hoodie.index.type=RECORD_INDEX` the lookup stages read as workload profiling — their stage names say `SparkMetadataTableGlobalRecordLevelIndex.java:133`, but the description does not. Now `Record index lookup:`. **`SparkStreamingMetadataWriteHandler`.** This is the significant one. Streaming writes to the metadata table are enabled by default on Spark, and that path sets no description at all. In a controlled comparison — the same workload, the same record-level index, toggling only `hoodie.metadata.streaming.write.enabled` — the share of stage wall clock belonging to stages with no recognisable Hudi description went from **under 2% to roughly 60%**. Now `Streaming write to metadata table:`. **`StreamSync`, around the transformer and around record creation.** Neither is tagged. Because Spark is lazy, the transform is forced inside the record-creation work, which itself runs under the description set before the source read. So on a HoodieStreamer pipeline with a transformer, the transform and record-creation cost both appear as source read. Now `Transforming source batch:` and `Creating Hudi records:`. ### Impact User-visible improvement to the Spark UI for anyone running Hudi on Spark, and to anything that reads a Spark event log. No public API change, no configuration, no storage format change, no change to what is written. The descriptions follow the existing convention — `HoodieSparkEngineContext` renders them as `<module>:<activity>`, where the module is the calling class's simple name. ### Risk Level none `setJobStatus` sets a thread-local string on the Spark context. The five call sites are on paths that already execute; none is in a loop or a hot inner path. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
