nsivabalan opened a new pull request, #20196: URL: https://github.com/apache/hudi/pull/20196
### Describe the issue this Pull Request addresses Part of #19262 (Hudi-aware agent skills and tools), under the Agentic Lakehouse umbrella #19256; plan in https://github.com/apache/hudi/issues/19262#issuecomment-5750570015. A user looking at a slow Hudi write sees Spark stages: "stage 14, 16000 tasks, 3 minutes". Spark cannot tell them that stage 14 is bloom-index probing, that its parallelism comes from `hoodie.bloom.index.parallelism`, or that the 10,000x task skew there means their key distribution defeats range pruning. Generic Spark profilers have the stage metrics but no idea which part of Hudi each stage belongs to, so their advice stays generic. Hudi already labels its own Spark jobs — `HoodieSparkEngineContext.setJobStatus` sets a job description on every significant step. This change reads those labels back out of a Spark event log and turns a stage listing into a Hudi-phase breakdown. ### Summary and Changelog Adds `HoodieSparkEventLogAnalyzer` plus a small `eventlog` package. It parses a Spark event log (the file written when `spark.eventLog.enabled=true`) and reports where a Hudi write spent its time, by Hudi phase and operation rather than by stage number. **No Spark session required** — it reads a file, so it runs as a plain `java -cp` process. Reads through `HoodieStorage`, so local, S3 and GCS paths all work, as do rolling-log directories and gzip. The log is parsed with Jackson rather than Spark's `JsonProtocol`, which is private and version-coupled. **Phases.** Nine buckets, deliberately coarse, because the useful answer is "60% of your time is index tagging", not a per-stage taxonomy: - Write operations: `SOURCE_READ_AND_TRANSFORM`, `DEDUP_AND_INDEX_TAGGING`, `DATA_TABLE_WRITE`, `MARKER_RECONCILIATION`, `METADATA_TABLE_WRITE` - Table services: `PLANNING`, `EXECUTION` - Plus `ARCHIVAL` and `UNKNOWN` Two bucket names state an attribution limit rather than hiding it. `SOURCE_READ_AND_TRANSFORM` is named for everything it contains because Hudi tags neither the transformer nor record creation, so their cost is forced inside the source-read stage and cannot be separated. `DEDUP_AND_INDEX_TAGGING` likewise covers one job: Hudi's `Building workload profile:` is the description under which dedup, index tagging and profiling all execute. A finer `subPhase` is kept alongside each stage — `recordIndexLookup`, `bloomIndexLookup`, `smallFileProbe`, `commitStats`, `compactionPlan` and others. That detail is what makes a coarse bucket actionable: a `DATA_TABLE_WRITE` row at 57% says little, but the same row tagged `[smallFileProbe]` with a 10,000x skew points straight at a pathological partition. **Operations.** Thirteen values (`WRITE_OPERATION`, `COMPACTION`, `CLUSTERING`, `CLEANING`, `ROLLBACK`, `RESTORE`, `SAVEPOINT`, …) resolved from the module half of the job description, falling back to phase-sequence inference, then `UNKNOWN`. The resolution source is reported so a reader can judge it. This matters because `PLANNING` and `EXECUTION` recur across every table service — only the operation tells a compaction plan from a clustering plan. Repeated rollback groups are promoted to `RESTORE`, which is what a restore looks like from the outside. **Two wire encodings.** Before HUDI-8596, `setJobStatus` used `setJobGroup`, so the module landed in `spark.jobGroup.id` and the description held the bare activity. After it, both are combined into the description. Both are handled, so the tool works on pre-1.0 logs too. **Per stage:** task count, duration percentiles, skew ratio, GC fraction, shuffle, spill, failure reasons. **Per application:** critical path, peak concurrent executors, and the share of wall clock per phase and per operation. TABLE output for humans, `--output JSON` for machines, with a documented schema and deterministic ordering. **Caveats are part of the output, not a footnote.** Hudi sets a job description before the Spark action and Spark evaluates lazily, so a bucket names where work was *forced*, not where it logically belongs. The JSON carries a `caveats[]` array and the table output has a "How far to trust the attribution" section; entries fire only for the buckets actually present in that log. **Known limitation, stated plainly in the docs.** With `hoodie.metadata.streaming.write.enabled` (the default on Spark), the streaming metadata-write path sets no job description at all, so a large share of the DAG cannot be attributed. Measured on a local three-way comparison — same workload and record-level index, toggling only that config — unattributed wall clock went from 3.9% to 60.5%. The tool reports this honestly rather than guessing, and the breakdown is accurate for non-streaming writes and for 0.x-style disjoint DAGs. Also documented: `outputBytes` and `outputRecords` are structurally zero for Hudi writes, because Hudi writes Parquet through its own file handles and Spark never observes the bytes. Commit metadata is the right source for written volume. **Tests** — `TestHoodieSparkEventLogAnalyzer`, 118 tests on synthetic event logs: both wire encodings, every phase, operation resolution by module and by sequence, rollback-to-restore promotion, sub-phase refinement, percentile and skew arithmetic, zero-task stages, stage retries, truncated and garbage logs, rolling directories, gzip, exit codes, and JSON determinism. Validated end to end against real event logs from local Hudi writes across several metadata-table configurations, and against a production Spark UI breakdown. ### Impact Purely additive. New class and new package in `hudi-utilities`; no existing file is modified. No public API change, no storage format change, no new `hoodie.*` configs. Read-only: it parses a log file and prints a report. ### Risk Level none ### Documentation Update `HoodieSparkEventLogAnalyzer.md` covers invocation, the phase and operation vocabularies, the JSON schema, and the caveats and limitations above. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
