cshuo commented on code in PR #20459: URL: https://github.com/apache/hudi/pull/20459#discussion_r4449906750
########## rfc/rfc-116/rfc-116.md: ########## @@ -0,0 +1,438 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-116: Support Branches and Tags + +## Proposers + +- @cshuo +- @danny0405 + +## Approvers + +- TBD + +## Status + +RFC reservation: [#20456](https://github.com/apache/hudi/pull/20456) + +## Abstract + +This RFC adds writable branches and read-only tags to Hudi. The data table and a single metadata table (MDT) share immutable files across branches, while each branch has its own timelines. MDT reads follow the selected branch’s timeline. A working branch can publish to main through strict fast-forward. Writing the Data-side `_SUCCESS` publishes the paired Data/MDT state to main. Tags protect paired historical Data/MDT states for time-travel reads and writable branch creation. + +## Background + +Branches let users write and validate changes before exposing them on main. Tags name historical states and retain the files needed to read them; ordinary time travel alone does not provide that protection. + +This RFC covers an initial subset of [apache/hudi#19066](https://github.com/apache/hudi/issues/19066): isolated batch writes, repeatable historical reads and publication to main. Strict fast-forward requires inherited logical write history; completed clustering, compaction and log compaction only change physical layout. If main and audit independently change data after a common ancestor, pausing main cannot make audit publishable; merging divergent changes is out of scope. + +| Area | Scope | +|---|---| +| Engine | Spark batch | +| Data | COW and MOR, table versions 8/9/10 | +| MDT | Enabled MOR table; one physical root with shared index files | +| Definitions | Fixed schema, keys, partition layout, merge configuration and index definitions | +| Branch creation | main's latest completed commit or an unexpired tag on main | +| Publication | Working branch → main, strict fast-forward | + +Out of scope: branch creation from working branches, versions below 8, Flink, streaming/incremental/CDC reads, bootstrap, merge/rebase, independent format upgrades per branch, external MDT service managers and non-main catalog sync. + +## Implementation + +### 1. Shared files and branch views + +Data and MDT branches with the same name form a pair. Each has its own configuration and active/archived timeline. Data files and MDT base/log files are shared; new writes produce immutable files. Branches can update the same inherited fileId independently. + +A new branch takes history and files from main's latest completed commit or a tag on main, and configuration from main. Fixed table definitions must agree across branches and cannot change after creation. Branch-local runtime tuning may differ. + +- **Data FSView:** Use the selected branch's MDT FILES, timeline and replace history. Directory listing alone does not establish membership. +- **MDT FSView:** MDT has no nested MDT. List physical files, then filter their requested instants against the selected MDT branch's active and archived timeline **before** constructing file groups/slices. An instant older than the active timeline start is not proof of membership. + +The design enforces three invariants: + +1. Reads and writes stay within the selected branch's paired Data/MDT state. +2. Publication preserves main's logical write history and tag/savepoint protections. +3. Shared files remain until all protections from branches, savepoints and pending instants allow deletion. + +### 2. Branches and metadata layout + +Main is the default branch. Initially, its metadata stays in the Data and MDT root `.hoodie/` directories. Working branches use `branches/<branch-name>/`; publications use `branches/publications/<publication-id>/` on both sides. Publication directories are created only when a working branch publishes to main. Branch discovery excludes the reserved `publications/` directory. +In the layouts below, `timeline/` includes active and archived history. + +```text +<table>/.hoodie/ + hoodie.properties + timeline/ # initial main Data timeline + tags/<tag-name>.json # initial main tags + branches/ + publications/<publication-id>/ # published main Data metadata + _SUCCESS # written last to publish the paired Data/MDT state + hoodie.properties + timeline/ + tags/ + <branch-name>/ # working-branch Data metadata + branch.json + hoodie.properties + timeline/ + tags/ + metadata/ + files/ + record_index/ + secondary_index_<index-name>/ + .hoodie/ + hoodie.properties + timeline/ # initial main MDT timeline + branches/ + publications/<publication-id>/ # paired published main MDT metadata + hoodie.properties + timeline/ + <branch-name>/ # paired working-branch MDT metadata + branch.json + hoodie.properties + timeline/ +``` + +#### Main resolution + +A `publicationId` has the form `p<N>`. Under the table lock, scan all Data and MDT publication directories, including unfinished preparations, and choose the largest numeric `N` plus one (`p1` if none exist). Reserve the number before releasing the lock by creating the Data-side `hoodie.properties` without overwrite; this also makes the prefix visible on object stores. Keep the current successful publication. A failed preparation's number may be reused only after its operation and storage requests have stopped and both directories have been fully removed under table coordination. + +Before the first successful publication, main uses the paired root `.hoodie/` metadata. Preparation directories without `_SUCCESS` do not change that selection. Once published, main uses the greatest **numeric** publication number with a Data-side `_SUCCESS`; that ID selects both metadata directories. Scan all listing pages and check Data-side `_SUCCESS` files under the read-capture lock. Publication failure leaves the current selection unchanged; missing or unreadable selected metadata is an error, not permission to revert to an older state. + +Ordinary commits, maintenance and tag changes update the selected root or publication metadata. They do not create publication directories or change `_SUCCESS` or the publication ID. A publication ID identifies metadata directories, not a fixed data snapshot. + +#### Create and drop + +Hold the table lock throughout branch creation: + +1. Select main's current completed Data/MDT state, or the historical state of `from_tag` on main. Record the inherited main Data instant as `baseInstant` (null if main has no completed commits). Reject missing/expired tags and requested/inflight instants that could change this state. +2. Copy the branch metadata listed below into the new paired directories, using main's configuration. Share Data files and MDT index files, including RLI/SI. Missing metadata or copy failures abort creation. Do not inherit pending table services as runnable work. +3. Confirm metadata copying has completed on both sides, then write the MDT `branch.json` first and the Data `branch.json` last. Release the lock after activation or failure. + +Both `branch.json` files carry the same never-reused `branchId`. The Data `branch.json` makes the branch discoverable only when the MDT identity matches. Inherited tags remain protected independently of their source. Replan services under new instants. + +For `from_tag`: + +- Select the historical Data/MDT state protected by the paired savepoints. Copy the existing archived timelines, including their manifests, and exclude writes, replacements and table services after the tag instant from the active timelines. +- Copy matching schema history and index definitions, the selected tag and both savepoints. Exclude unrelated tags/savepoints; reject definitions incompatible with main's fixed configuration. +- Share the protected files without rebuilding data or indexes; the activated branch no longer depends on the source tag's protection. + +The following metadata is maintained per branch. Copy applicable contents for each Data/MDT metadata root when creating a branch or preparing a publication; absent optional directories need not be created. + +| Metadata | Contents and handling | +|---|---| +| `hoodie.properties` | Complete configuration: use main's configuration for branch creation and the captured source configuration for publication. Adjust metadata paths for the target. | +| `timeline/` | Completed history, including savepoints and archived history with its manifests and referenced files. Preserve a consistent captured state; do not inherit pending instants as runnable work. | +| `.schema/` | Completed schema history matching the captured commits, including historical schemas and field IDs. Exclude pending files and later source state. | +| `.index_defs/index.json` or the configured index-definition file | Actual index definitions and versions, with configuration pointing to the target copy. | +| `tags/` | Valid tag definitions. Paired Data/MDT savepoints are included in their respective `timeline/` directories. Applies to Data metadata. | + +Schema and index-definition access must resolve through the selected branch/publication, with caches isolated accordingly. Do not copy `.aux` wholesale: locks, sample writes and legacy pending plans are excluded, as are `.temp`, `.heartbeat` and other lock state. Write branch `branch.json` files and publication `_SUCCESS` separately. Data and MDT base/log files remain shared. + +`drop_branch` requires a maintenance window for that branch: stop writers/services, drain readers and resolve pending instants. Under the table lock, remove the Data `branch.json`, then clean branch metadata. Shared files follow the cleaning rules below. Main cannot be dropped. Reusing a name requires completed metadata cleanup and a new `branchId`. + +### 3. Tags + +A tag belongs to one branch, defaults to main, and has no writable timeline. Names are unique within the owning branch; different branches may use the same tag name. Each Data instant can have at most one tag per branch. + +```text +<table>/ + .hoodie/ + hoodie.properties + timeline/ + tags/<tag-name>.json # initial main tag definition + branches/ + <branch-name>/ # working-branch Data metadata + branch.json + hoodie.properties + timeline/ + tags/<tag-name>.json # branch tag definition + metadata/ + files/ + record_index/ + secondary_index_<index-name>/ + .hoodie/ + hoodie.properties + timeline/ + branches/ +``` + +The layout shows tags on initial main and a working branch. Working branches and published main use the same relative `tags/` path within their Data metadata directories. A tag definition is: + +```json +{ + "name": "T2", + "instant": "20261026103000000", + "createdAt": "2026-10-26T11:00:00Z", + "expiresAt": "2026-11-02T11:00:00Z" +} +``` + +`name` matches the filename. `instant` identifies the completed Data commit and its corresponding MDT state. Create paired savepoints at this same requested instant in both branch timelines. Omit `expiresAt` for indefinite protection. Inheritance copies the tag and both savepoints. + +#### Creation and protection + +Under the table lock: + +1. Process conflicting expired tags and confirm earlier operations have stopped. Reject duplicate names, another tag at the same Data instant, and pending instants that could change the target snapshot. +2. Use native savepoint checks to select the Data file slices at the completed instant and the corresponding MDT file slices. Require the FILES and enabled index metadata partitions to be available at that instant. Reuse compatible completed savepoints or create them on both timelines at the same requested instant; the MDT savepoint records base/log files by metadata partition. Fail if the timeline metadata or file slices needed for savepoint creation cannot be resolved; do not substitute current MDT state or rebuild indexes. +3. Publish the tag JSON only after both savepoints are complete. Hold the lock through preparation so cleaning cannot remove their inputs. + +**TODO — MDT savepoint file protection:** Extend savepoint creation/deletion to the selected MDT branch timeline, recording historical base/log files by index partition. Reuse native MOR protection checks and adapt MDT incremental cleaning after savepoint deletion. Coordinate paired Data/MDT savepoints and enforce protection across branches; this integration is not provided by the current Data savepoint API. + +Both cleaners must honor completed savepoints. Set `hoodie.archive.beyond.savepoint=false` on both sides and preserve the timeline/archive/schema history needed to interpret protected files. Compaction and log compaction may proceed, but cannot release protected historical inputs. Reject writes, replacements, rollback or restore that invalidate a protected snapshot. These rules also apply to retained savepoints without a tag. + +#### Deletion and expiry + +Under the table lock, remove the tag definition, then use the paired `delete_savepoint` logic by default, including for reused savepoints. `retain_savepoint=true` retains both. Deletion releases protection for normal cleaning; it does not immediately delete files. Other branches' savepoints remain independent. + +Expired tags cannot serve new queries or appear in `show_tags`. `expire_tags` and cleaner use the default deletion behavior. An expired definition reserves its name and Data instant until removed. + +`delete_savepoint` accepts `branch` (default `main`) and deletes Data/MDT savepoints together at the specified instant. Reject deletion while a tag definition in that branch references the instant, including an expired definition awaiting cleanup. An absent-tag `drop_tag(..., if_exists=true)` is a no-op; it does not clean remaining savepoints. Partial failures are covered in section 7. + +#### Reads + +Resolve the tag to its branch and instant, then use the ordinary Data time-travel read path with the corresponding MDT state at the same instant. Savepoint metadata protects files from cleaning; it is not used as the query's file list. FILES, RLI and SI may serve historical reads only through the matching historical MDT view; never substitute their current state. + +Adapt time-travel validation for tags: retain pending-write checks, but accept a completed, valid tag savepoint even if the ordinary cleaner retention boundary has passed. Do not relax ordinary timestamp time travel. Historical MDT reads are guaranteed while the tag protection exists; ordinary timestamp reads do not gain additional retention. + +#### Historical MDT reads + +Savepoints retain historical files; the MDT read path reconstructs their state for tag queries and branch creation: + +1. **Pass the historical instant.** Extend MDT APIs to pass the tag's `instant` through FILES, RLI and SI lookups. +2. **Select historical files.** Reuse FSView historical slice selection and the MOR reader. Select the matching base files and, for v8+ writes, log files by write instant and branch timeline. Exclude later updates; use retained original logs when log-compaction output includes them. +3. **Match Data visibility.** Exclude pending Data updates and verify that MDT initialization, indexing and compaction results match the target Data state. Validate this when creating paired savepoints; later reads and branch creation resolve the same state from the retained timelines. +4. **Isolate caches.** Key cached slices, readers and index results by branch/publication identity and historical read state. + +Missing files or metadata fail the read; never fall back to the latest MDT state. + +### 4. Publication to main + +#### Conditions + +1. **Preserve main's writes.** The source must retain the logical write history inherited at `baseInstant`; rollback or restore that removes it makes the source ineligible for fast-forward, including before capture. Check main's later logical writes in both active and archived timelines. Reject writes absent from the source, including overwrite and partition deletion. Earlier history need not be rescanned. Repeated publication from the same source is allowed only while it retains previously published logical writes. + +2. **Allow completed table services.** Completed clustering, compaction and log compaction do not by themselves block publication; identify them from operation metadata. Reject requested/inflight instants that could change the state being published. Later source commits may proceed but are excluded from the captured state. + +3. **Preserve main's tags and savepoints.** The source must contain matching definitions for every unexpired main tag, its paired Data/MDT savepoints, and main's manual savepoints with their protected snapshots. This also applies after independent main table services. + +4. **Publish the source's paired state.** Use the captured source Data/MDT state without merging main's independent table-service results or timelines. The source remains independently writable. + +5. **Process expired tags before copying.** Process expiry on main and the source. If a tag expires during copying, abort before `_SUCCESS` and retry after expiry processing. Review Comment: Added an example of divergent histories: fast-forward is rejected when main has an independent logical write, leaving main unchanged. The Publication Modes section also adds specification about the limits and conflicts for each publication mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
