hudi-agent commented on code in PR #13152: URL: https://github.com/apache/hudi/pull/13152#discussion_r3569338202
########## rfc/rfc-94/rfc-94.md: ########## @@ -0,0 +1,784 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-94: Hudi Timeline User Interface (UI) + +## Proposers + +- @voonhous + +## Approvers + +- @danny0405 +- @rahil-c +- @yihua + +## Status + +JIRA: [HUDI-9315](https://issues.apache.org/jira/browse/HUDI-9315) + +## Abstract + +Hudi Timeline metadata is stored as timestamped files representing state transitions of actions like `commit`, +`deltacommit` and `compaction`. These files are accessible via the CLI or a file explorer, but it's hard to visualize +concurrent actions, spot missing transitions, or tell how long each step took. Debugging timeline issues by reading +filenames is tedious. + +This RFC proposes a UI-based timeline visualization tool that parses these metadata files, groups related actions, and +renders them in a time-ordered, interactive view. Users can track the lifecycle of each operation, see concurrency +patterns, and spot anomalies or long-running tasks. The implementation extends `hudi-timeline-service` with new `/v2/` +REST APIs and a static HTML + JavaScript frontend powered by [vis-timeline](https://github.com/visjs/vis-timeline), +served via Javalin's built-in static file serving with zero new Java compile-time dependencies. + +## Background + +Today, we rely on the CLI or direct filesystem inspection to understand timeline state through metadata files. These +files represent different actions (e.g., `deltacommit`, `compaction`) and their lifecycle states (`requested`, +`inflight`, `completed`), encoded in file names like: + +```shell +20250409102118815.deltacommit.inflight +20250409102118815.deltacommit.requested +20250409102118815_20250409102124339.deltacommit +20250409102121593.compaction.inflight +20250409102121593.compaction.requested +20250409102121593_20250409102122232.commit +20250409102124581.deltacommit.inflight +20250409102124581.deltacommit.requested +20250409102124581_20250409102125667.deltacommit +20250409102124612.compaction.inflight +20250409102124612.compaction.requested +20250409102124612_20250409102124892.commit +20250409102127348.deltacommit.inflight +20250409102127348.deltacommit.requested +20250409102127348_20250409102128481.deltacommit +20250409102127500.compaction.inflight +20250409102127500.compaction.requested +20250409102127500_20250409102127721.commit +``` + +This works, but has a few problems: + +1. No visibility into concurrency + - Multiple actions (e.g., `deltacommit` and `compaction`) often run concurrently. + - The CLI doesn't help correlate or visualize overlapping operations. +2. Lack of temporal context + - Timestamps are embedded in filenames but are hard to compare visually - year, month and day can be quickly + determined, but minutes and seconds are harder to parse. + - No easy way to tell how long an action took or whether it's stalling unless you manually calculate the difference + between requested and completion time. +3. Hard to spot inconsistencies or missing states + - An `inflight` compaction without a corresponding `commit` can indicate a starved/stuck compaction, which usually + blocks archiving/cleaning. + - These gaps are easy to miss when scanning filenames. + +On top of that, all timeline files are now stored as Avro binaries. Inspecting their contents requires custom Avro +readers to convert the binaries to JSON. + +## Scope + +This RFC covers visualization of metadata available in Hudi tables. All features are **READ-ONLY** - there is no support +for starting or spawning jobs that mutate a Hudi table. + +Alongside the timeline, the UI surfaces two additional read-only metadata views: the table's configuration +(`hoodie.properties`) and its schema-change history. + +The following are **out of scope**: + +- **Archived timeline:** Only the active timeline is rendered. Loading instants from LSM-based archive files is left for + future work. +- **Metadata table overlay:** The metadata table's own timeline is not shown alongside the main table timeline. +- **Write/mutation operations:** The UI cannot trigger compactions, clustering, or any write action. +- **Authentication/authorization:** No access control is added. The timeline server is assumed to run in a trusted + network, same as today. + + **Threat model:** None of the four UI views is `/v1`-parity. The existing `/v1/` routes serve only + file-slice/base-file DTOs plus filename-level instant DTOs (action, requested/completion time, state); every `/v2` + view widens the read surface beyond that: + + - **Timeline** (`/v2/hoodie/view/timeline/instants/all`) is a superset of `/v1`'s `timeline/instants/all`. The + `/v1` route is served from the `FileSystemView`'s write timeline, which is restricted to completed instants plus + pending (log)compaction, and to write actions only. The `/v2` route reads the full active timeline, so it + additionally exposes `clean`, `rollback`, `savepoint`, `restore` and `indexing` instants, and every + requested/inflight state. + - **Instant detail** (`/v2/hoodie/view/timeline/instant`) has no `/v1` counterpart at all - no existing route + returns instant *content*. It returns deserialized instant metadata, e.g. a `HoodieCommitMetadata` carrying + per-partition write stats (file paths, record counts) and the table schema under the `schema` key of + `extraMetadata`. Its `instant` param is attacker-controlled and lands in a storage path, so the route resolves + the requested instant against the active timeline rather than reconstructing it from the params; see + [Handler Design](#handler-design). Without that, it is an arbitrary-path read, not a timeline read. + - **Table config** (`/v2/hoodie/view/table/config`) returns the full `hoodie.properties` via + `HoodieTableConfig.getProps()`. + - **Schema history** (`/v2/hoodie/view/table/schema/history`) exposes current and historical table schemas - the + same schema content the instant-detail view can surface via `extraMetadata`. + + All of it is read-only and already readable by anyone with filesystem access to `.hoodie/`; the UI adds no write or + mutation capability, and opens no new network interface (the server binds to all interfaces on the + driver/standalone host, as it does today). The most sensitive of the four is table config: properties can reference + KMS endpoints, lock-provider connection strings or external key/vault paths, though they rarely embed secrets + directly. The first cut serves table config unfiltered (sorted, as-is); a redacting/allowlisted config view is a + possible future refinement for less-trusted interfaces. The primary control is that all UI routes are gated behind + `--enable-ui` (off by default), with the server assumed to run on a trusted network. Operators on untrusted networks + should front the server with a reverse proxy or restrict it to a private interface / localhost via network policy. + +## Implementation + +Keeping the implementation lightweight is a priority - we should add as few dependencies as possible. Changes go into +the existing `hudi-timeline-service` module, which contains a Javalin web-application that caches filesystem metadata of +a Hudi table for job executors during tagging/writing. + +The first cut runs the UI on the Timeline Server in **STANDALONE** mode (see [Configuration](#configuration)) and is +self-contained within `hudi-timeline-service`. Enabling the UI on the **EMBEDDED** timeline server inside a Spark +driver, together with a Spark UI tab, requires cross-module wiring (`hudi-client-common`, `hudi-spark-client`); it is +designed below but deferred to a follow-up to keep the initial PR small and focused. The standalone UI lands first; the +embedded/Spark linking lands next. + +The Hudi Timeline UI has two parts: the frontend and backend. + +### Architecture + +The timeline server can run standalone or embedded inside a Spark driver. In embedded mode, a tab in the Spark UI links +directly to the Hudi Timeline UI. The embedded mode and Spark UI tab (right side of the diagram below) are a planned +follow-up; the first cut is standalone-only. + +```mermaid +graph LR + Browser["Browser"] + + subgraph Driver["Standalone / Spark Driver"] + subgraph TimelineServer["Javalin (Timeline Server)"] + Static["/ui entry + /ui/static/*\n(HTML, JS, CSS)"] + API["/v2/hoodie/view/* - TimelineHandler"] + Meta["HoodieTableMetaClient\n(active timeline, config, schema)"] + + API --> Meta + end + + subgraph SparkUI["Spark UI (:4040) - embedded mode (follow-up)"] + direction TB + SparkUIPad[ ] ~~~ Tabs["[Jobs] [Stages] ... [Hudi Timeline]"] + end + + style SparkUIPad fill:none,stroke:none,color:none + + Tabs -- "link" --> Static + end + + Browser -- "HTTP" --> Static + Browser -- "HTTP" --> API + Browser -. "HTTP\n(embedded mode)" .-> SparkUI +``` + +There are two categories of requests: + +1. **Static file requests** - Javalin serves JavaScript, CSS, and library assets from the classpath + (`src/main/resources/public/`) under the `/ui/static/` URL prefix; `UiHandler` serves `index.html` at `/ui`. No + server-side rendering or template engine is needed. +2. **REST API requests** (`/v2/hoodie/view/*`) - `TimelineHandler` processes these requests, reading from a short-lived + `HoodieTableMetaClient` built for the request's basepath - its `getActiveTimeline()` for the timeline routes, and + table config/schema for the config/schema routes - and returning JSON. + +### Frontend + +The frontend is static HTML pages with vanilla JavaScript, similar to the Spark Web UI. Javalin's built-in static file +serving handles files from the classpath - no template engine (e.g., Thymeleaf) is needed and no new Java compile-time +dependencies are added. + +No frontend build pipeline (npm, webpack, vite) is needed. Contributing to the UI requires only a text editor. Three +libraries are vendored as static assets: vis-timeline (timeline rendering), Bootstrap 5 (layout/styling), and renderjson +(collapsible JSON in the detail panel). + +#### File Structure + +``` +hudi-timeline-service/src/main/resources/public/ +├── index.html # Landing page with basepath input form +├── js/ +│ └── timeline.js # vis-timeline initialization and REST API calls +├── css/ +│ └── style.css # Basic styling +└── lib/ # Vendored third-party assets (see Dependency Impact) + ├── vis-timeline/ # Timeline rendering (Apache-2.0 OR MIT) + │ ├── vis-timeline-graph2d.min.js + │ └── vis-timeline-graph2d.min.css + ├── bootstrap/ # Layout/styling (MIT) + │ ├── bootstrap.bundle.min.js + │ └── bootstrap.min.css + └── renderjson/ # Collapsible JSON detail panel (ISC) + └── renderjson.js +``` + +#### JavaScript Delivery: Bundled, No External Calls + +All three libraries are served from bundled copies under `/ui/static/lib/` (`/ui/static/lib/vis-timeline/`, +`/ui/static/lib/bootstrap/`, `/ui/static/lib/renderjson/`). The UI makes no external network calls, so it works out of +the box in air-gapped and security-conscious deployments with no extra configuration. The bundled, minified assets add +~890KB to the JAR (vis-timeline ~575KB, Bootstrap 5 ~305KB, renderjson ~11KB). + +Pinning a vendored copy (rather than loading from a CDN) keeps the UI deterministic and avoids a runtime dependency on +an external host being reachable. If automatic patch updates are wanted later, a CDN source can be added as an opt-in +config flag without changing this default. + +#### vis-timeline Configuration + +The timeline is configured with groups and items that map to Hudi's timeline model: + +- **Groups:** One row per *comparable* action - `commit`, `deltacommit`, `replacecommit`, `clean`, `rollback`, + `savepoint`, `restore`, `indexing`. This is `HoodieTimeline.VALID_ACTIONS_IN_TIMELINE` with `compaction`, + `logcompaction` and `clustering` folded into the action they complete as, exactly as Hudi itself already does: + + | Pending action | Completes as | + |----------------|------------------| + | `compaction` | `commit` | + | `logcompaction`| `deltacommit` | + | `clustering` | `replacecommit` | + + This mapping is not a UI invention - it is `InstantComparatorV2.COMPARABLE_ACTIONS`, and the active timeline has + already applied it before the handler sees an instant. `HoodieTableMetaClient.getActiveTimeline()` builds an + `ActiveTimelineV2` with the layout filter enabled, which groups instants by `(requestedTime, comparableAction)` and + keeps only the highest state. So a completed compaction arrives as a *single* instant whose action is `commit` (its + completed file on disk is `<ts>_<ct>.commit`), and a raw `compaction` action only ever reaches the UI while the + instant is still pending. Giving `compaction`, `logcompaction` and `clustering` their own rows would therefore + produce rows that can never hold a completed instant. Grouping by comparable action instead keeps each row's pending + and completed items in the same lane, which is what makes a stalled instant legible (see **Items** below). +- **Items:** Completed instants are rendered as range bars spanning from `requestedTime` to `completionTime`. + Non-completed instants (requested or inflight) are rendered as point items at `requestedTime`. + + Items keep their *raw* action for labelling and colouring, so folding rows together does not erase the distinction: + a pending compaction is still labelled `compaction` and still sits in the `commit` row. This is precisely what the + stuck-compaction diagnosis in [Background](#background) needs - an inflight `compaction` point item with no + following range bar in that row is a compaction that never committed. A *completed* compaction does render as an + ordinary `commit` range bar, which is correct: on disk and to every other Hudi component, it is a commit. Users who + need to confirm the originating operation click the instant; the detail panel shows the `HoodieCommitMetadata`, + whose `operationType` reads `COMPACT` (or `CLUSTER`). +- **Color coding:** Items are colored by state: + - Green -> `COMPLETED` + - Yellow -> `INFLIGHT` + - Red -> `REQUESTED` +- **Tooltip:** On hover, shows the action type, requested time, completion time, and duration. +- **Click handler:** Clicking an instant fetches its detail via `/v2/hoodie/view/timeline/instant` and shows the + deserialized JSON in a detail panel below the timeline. + +### Backend + +A `hudi-timeline-service` instance already serves filesystem metadata for multiple table basePaths since the +`FileSystemView`s are cached in a map keyed by basepath. + +We extend this module with `/v2/` APIs that serve the UI's timeline, config and schema metadata, reading each table +through a short-lived `HoodieTableMetaClient` built per request (see [Handler Design](#handler-design)). + +#### API Specification + +| Method | Path | Parameters | Response | Description | +|--------|-----------------------------------------|-----------------------------------------------------------------------|-----------------|----------------------------------------------------------------------------------------------| +| GET | `/v2/hoodie/view/timeline/instants/all` | `basepath` (required) | `TimelineDTOV2` | All active instants (each with requested time, completion time, action, state), wrapped in a timeline DTO | +| GET | `/v2/hoodie/view/timeline/instant` | `basepath`, `instant`, `instantaction`, `instantstate` (all required) | JSON string | Deserialized content of a specific instant's metadata (Avro -> JSON). The triple must match an instant in the active timeline, else 404 - see [Handler Design](#handler-design) | +| GET | `/v2/hoodie/view/table/config` | `basepath` (required) | JSON object | The table's `hoodie.properties` (sorted) | +| GET | `/v2/hoodie/view/table/schema/history` | `basepath` (required), `limit` (optional, default 200, max 1000) | JSON object | Current schema, per-commit schema-change history from the last `limit` commits, and `.schema` internal-schema history when present | + +Static assets (JS, CSS, library files) are served from the classpath directory `src/main/resources/public/`, mounted +under the `/ui/static/` URL prefix via Javalin's static-files `hostedPath` (e.g., `/ui/static/js/timeline.js`, +`/ui/static/lib/...`). Namespacing everything UI under `/ui` keeps the UI surface from colliding with `/v1/`, `/v2/`, or +any future module-registered routes on the same Javalin instance, rather than reserving root prefixes like `/js`, +`/css`, and `/lib`. `UiHandler` additionally registers `GET /ui`, which returns `index.html` (with asset links pointing +at `/ui/static/...`) to give the UI a stable entry URL. + +**On response size and pagination:** `GET /v2/hoodie/view/timeline/instants/all` returns the full active timeline. The +active timeline is bounded by archiving (the unbounded archived timeline is out of scope), so instant counts are +typically modest. The first cut intentionally returns all active instants and relies on client-side zoom/scroll and +filtering for navigation. If active-timeline sizes become a concern, the endpoint can be extended additively with +optional `from`/`to` time-range query params (and/or a `limit`) without breaking the existing contract. + +#### DTO Design + +The UI's timeline endpoint returns a `TimelineDTOV2` built from two v2 DTOs in a `v2` package, leaving the existing +`/v1/` API contract untouched. + +The v1 `InstantDTO` already carries everything needed to render range bars - `fromInstant` populates both +`requestedTime` and `completionTime` from `HoodieInstant` (added under HUDI-9332) - so the UI could consume the v1 +timeline DTO directly. The v2 DTOs are not about exposing new fields; they are a deliberate, low-cost choice to give the +new `/v2/` API a cleaner JSON contract: + +- **`InstantDTOV2`** (`o.a.h.common.table.timeline.dto.v2`) - the same source fields as v1, with UI-oriented JSON keys: + - `action` - the instant's raw action (e.g., `commit`, `deltacommit`, `compaction`). Used to label and colour the + item, and to address the instant on the `/v2/hoodie/view/timeline/instant` route. + - `comparableAction` - the action the instant completes (or would complete) as, derived server-side from the + table's `InstantComparator` (`getComparableAction`). Equal to `action` for everything except pending + `compaction`/`logcompaction`/`clustering`. This is the vis-timeline *group* key, and computing it server-side + keeps the mapping table in one place rather than duplicating it in JavaScript. + - `requestedTime` (JSON `requestTs`) - requested timestamp (`HoodieInstant.requestedTime()`) + - `completionTime` (JSON `completionTs`) - completion timestamp (`HoodieInstant.getCompletionTime()`), null for + non-completed instants + - `state` - the instant state (`REQUESTED`, `INFLIGHT`, `COMPLETED`) + + Versus v1, this adds `comparableAction`, renames `requestedTime`/`completionTime` to `requestTs`/`completionTs`, and + drops v1's redundant legacy `ts` field (a duplicate of the requested time that the UI does not need). +- **`TimelineDTOV2`** - wraps a `List<InstantDTOV2>` (`instants`); this is what `/v2/hoodie/view/timeline/instants/all` + returns. + + Both v2 DTOs carry the `V2` suffix to mirror the repo's versioned-class convention (`versioning/v1`/`v2` with + `InstantGeneratorV1`/`V2`, `BaseTimelineV1`/`V2`) and to avoid a simple-name clash with the existing `dto.InstantDTO` + that `TimelineHandler` still imports for the v1 routes. + +#### Handler Design + +The v2 endpoints are served by the existing `TimelineHandler` (which already serves the v1 timeline routes); a separate +`UiHandler` serves only the UI entry page. + +`TimelineHandler` methods: + +1. `getTimelineV2(basePath)` - maps `getActiveTimeline()` from the request's `HoodieTableMetaClient` to a + `TimelineDTOV2`. The active timeline carries every `VALID_ACTIONS_IN_TIMELINE` action in all states + (requested/inflight/completed), which the vis-timeline groups and requested/inflight point items require. The + `FileSystemView` timeline (`getFileSystemView(basePath).getTimeline()`) cannot be used here: it is the write timeline + filtered to completed plus (log)compaction instants, so it drops `clean`/`rollback`/`savepoint`/`restore`/`indexing` + and every requested/inflight state. + + Note that the active timeline has already applied `TimelineLayout.filterHoodieInstantsByLatestState`, which collapses + each `(requestedTime, comparableAction)` group down to its highest state. The handler therefore sees at most one + instant per logical action, never a requested/inflight/completed triple, and a completed compaction reaches it as a + `commit`. This is the behaviour the UI's group model is built around, not something the handler should try to undo: + `getTimelineV2` populates `comparableAction` from the same mapping the filter used, and leaves `action` as-is. +2. `getInstantDetails(basePath, instant, action, state)` - reads the instant's Avro content via the active timeline's + `getContentStream(instant)` (the non-deprecated reader method; `getInstantDetails()` is `@Deprecated`) and + deserializes it to JSON. A malformed `state`/`action` returns 400, a read failure is logged and returns 500. + + **The requested instant must be resolved against the loaded active timeline, not reconstructed from the request + params.** Building a `HoodieInstant` straight from `instant`/`action`/`state` and handing it to the reader is a + path-traversal read: `getContentStream` resolves to + `new StoragePath(metaClient.getTimelinePath(), <instant>.<action>.<state>)`, and `StoragePath` runs + `URI.normalize()`, which collapses `..` segments. An `instant` of `../../../../tmp/x` therefore resolves cleanly + outside `.hoodie` and the handler reads whatever is there. The trailing `.<action>.<state>` suffix and the Avro + deserialization narrow what an attacker can actually retrieve, but this is an unvalidated attacker-controlled path + and should not ship as one. + + The handler already loads the active timeline for this request, so the fix costs nothing: look the `(instant, + action, state)` triple up among the timeline's instants and read *that* `HoodieInstant`, returning 404 when no + instant matches. This is strictly stronger than regex-validating the timestamp - it also refuses well-formed + timestamps that simply do not exist in the table, so the route can only ever read instants the timeline itself + lists. The instant generator is then only used for instants that came from the timeline. + + Unlike the deprecated `getInstantDetails()`, which read into a `byte[]` and closed the stream itself, + `getContentStream` hands back a raw open stream (`metaClient.getStorage().open(...)`) that the caller owns. The + handler must read it under try-with-resources so the handle is released on the deserialization-failure path too, not + just on success. +3. `getTableConfig(basePath)` - returns the table's `hoodie.properties` as a sorted JSON object. +4. `getSchemaHistory(basePath, limit)` - reconstructs schema evolution from two sources; see + [Schema-History Reconstruction](#schema-history-reconstruction) below. + +All four methods - `getTimelineV2`, `getInstantDetails`, `getTableConfig` and `getSchemaHistory` - build a short-lived +`HoodieTableMetaClient` for the request's basepath, read from it, and discard it; no metaClient is shared across +Javalin's request threads. At human-click frequency the construction cost is negligible: every route pays one +`hoodie.properties` read, and all but `getTableConfig` additionally pay one active-timeline listing (`getTimelineV2` to +map it, `getInstantDetails` to resolve the instant it reads through `getContentStream`, `getSchemaHistory` to walk the +completed commits). `getInstantDetails` then reads a single instant file's content on top of that. A fresh per-request +instance is always current and keeps each request's read self-consistent. A long-lived per-basepath cache is deliberately avoided: the data must be re-read on every request +anyway, so caching the metaClient would save little while forcing an in-place `reloadActiveTimeline()` / +`reloadTableConfig()` on a mutable object shared across concurrent same-basepath requests. `HoodieTableMetaClient` marks +its individual accessors `synchronized`, but a compound reload-then-read is not one critical section, so a concurrent +reload could swap the snapshot between one request's reload and its read - a fresh instance sidesteps that entirely. A +TTL is likewise avoided: at this request rate it would save no meaningful work and would only add a staleness window. The +UI also surfaces a **Refresh** control on the Table Config and Schema History tabs; because the server reads fresh on +every request, the button simply re-issues the fetch. The timeline view is fresh on every request too; instant details +are immutable once written and need no refresh. + +`UiHandler` registers `GET /ui`, returning `/public/index.html` from the classpath as the UI entry page. + +#### Schema-History Reconstruction + +`getSchemaHistory` combines both schema sources Hudi maintains, so it returns something useful whether or not the table +uses schema evolution: + +- **`currentSchema`** - the current table schema from `TableSchemaResolver.getTableSchema()`; `null` if it cannot be + resolved. +- **`history`** - walks the completed commits timeline (`commit`/`deltacommit`/`replacecommit`), the most recent `limit` + instants only (default 200, capped at 1000), reading each instant's `HoodieCommitMetadata` and taking the schema under Review Comment: 🤖 `getSchemaHistory` reads up to `limit` (capped at 1000) commit-metadata instants per request. If those per-instant reads go through the same raw `getContentStream` API you flagged for `getInstantDetails`, the same stream-close discipline is needed here — but at up to 1000 reads per request, and with the Refresh control re-issuing the fetch, a leaked handle would exhaust file descriptors far faster than the single-read path. Could you confirm each per-instant read is closed (e.g. via a helper like `readCommitMetadata`, or per-read try-with-resources)? <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
