This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new e3a28fbdaf94 docs: Update documentation for new features in Hudi 1.2.0
(#18867)
e3a28fbdaf94 is described below
commit e3a28fbdaf946a6a5cc0f46229dac9d687f2aa99
Author: Y Ethan Guo <[email protected]>
AuthorDate: Thu May 28 09:27:50 2026 -0700
docs: Update documentation for new features in Hudi 1.2.0 (#18867)
---
website/docs/ai_overview.md | 4 +-
website/docs/azure_hoodie.md | 14 +-
website/docs/blob_unstructured_data.md | 45 ++++-
website/docs/cleaning.md | 47 +++++
website/docs/cli.md | 25 ++-
website/docs/clustering.md | 59 +++++-
website/docs/compaction.md | 13 +-
website/docs/concurrency_control.md | 42 ++++-
website/docs/deployment.md | 4 +-
website/docs/flink-quick-start-guide.md | 33 ++--
website/docs/flink_tuning.md | 47 +++++
website/docs/hoodie_streaming_ingestion.md | 106 ++++++++++-
website/docs/ingestion_flink.md | 244 +++++++++++++++++++++++--
website/docs/key_generation.md | 31 +++-
website/docs/lance_file_format.md | 129 +++++++++++--
website/docs/metadata.md | 31 ++++
website/docs/metadata_indexing.md | 33 +++-
website/docs/metrics.md | 67 ++++---
website/docs/overview.mdx | 4 +-
website/docs/precommit_validator.md | 67 +++++++
website/docs/procedures.md | 60 +++++-
website/docs/reading_tables_batch_reads.md | 26 +++
website/docs/reading_tables_streaming_reads.md | 44 +++++
website/docs/sql_ddl.md | 55 +++++-
website/docs/sql_queries.md | 33 +++-
website/docs/syncing_aws_glue_data_catalog.md | 4 +
website/docs/syncing_metastore.md | 41 +++++
website/docs/variant_type.md | 26 ++-
website/docs/vector_search.md | 42 ++++-
website/docs/writing_data.md | 29 ++-
30 files changed, 1293 insertions(+), 112 deletions(-)
diff --git a/website/docs/ai_overview.md b/website/docs/ai_overview.md
index 11c695d1bf9c..c28bcae6543a 100644
--- a/website/docs/ai_overview.md
+++ b/website/docs/ai_overview.md
@@ -18,7 +18,7 @@ Apache Hudi's AI-native capabilities bring this vision to
life with four foundat
### VECTOR Type and Similarity Search
-Store high-dimensional embedding vectors as first-class column types and run
approximate nearest neighbor (ANN)
+Store high-dimensional embedding vectors as first-class column types and run
vector similarity
search directly in Spark SQL.
```sql
@@ -99,7 +99,7 @@ query performance, while keeping the flexibility for
everything else.
Hudi's pluggable file format architecture supports **Lance**, a modern
columnar format purpose-built for
AI/ML workloads. Lance provides:
-- Efficient vector indexing and ANN search
+- Native vector column encoding (`FixedSizeList`) — no conversion overhead at
the file-format layer
- Fast random access for training data sampling
- Optimized storage for high-dimensional arrays and nested structures
diff --git a/website/docs/azure_hoodie.md b/website/docs/azure_hoodie.md
index a22d66598141..2730f6f9411a 100644
--- a/website/docs/azure_hoodie.md
+++ b/website/docs/azure_hoodie.md
@@ -2,7 +2,7 @@
title: Microsoft Azure
keywords: [ hudi, hive, azure, spark, presto]
summary: In this page, we go over how to configure Hudi with Azure filesystem.
-last_modified_at: 2020-05-25T19:00:57-04:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
In this page, we explain how to use Hudi on Microsoft Azure.
@@ -49,6 +49,18 @@ This combination works out of the box. No extra config
needed.
.load("/mountpoint/hudi-tables/customer")
```
+## Concurrency Control
+
+As of Hudi 1.2.0, the storage-based lock provider supports Azure ADLS Gen2
(`abfs://`, `abfss://`) and Azure Blob Storage (`wasb://`, `wasbs://`) base
paths for concurrency control. This allows multi-writer pipelines on Azure to
use storage-native conditional writes for locking — without requiring external
systems like ZooKeeper, or Hive Metastore.
+
+Add `hudi-azure-bundle` to your classpath and set:
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.StorageBasedLockProvider
+```
+
+The lock client supports multiple Azure authentication methods (connection
string, SAS token, managed identity, service principal, and
`DefaultAzureCredential`). See [Concurrency Control — Azure Storage-Based
Lock](concurrency_control.md#azure-storage-based-lock) for the full
configuration reference and authentication precedence.
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/docs/blob_unstructured_data.md
b/website/docs/blob_unstructured_data.md
index 74acc799a230..253656565084 100644
--- a/website/docs/blob_unstructured_data.md
+++ b/website/docs/blob_unstructured_data.md
@@ -3,7 +3,7 @@ title: "Unstructured Data"
keywords: [ hudi, blob, unstructured data, images, binary, pdf, audio, video,
inline, out-of-line, read_blob]
summary: "Store and query unstructured data (images, PDFs, audio, video) in
Hudi tables using the BLOB type with inline or out-of-line storage"
toc: true
-last_modified_at: 2026-04-25T00:00:00-00:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
import Tabs from '@theme/Tabs';
@@ -72,6 +72,7 @@ schema = pa.schema([
pa.field("external_path", pa.string()),
pa.field("offset", pa.int64()),
pa.field("length", pa.int64()),
+ pa.field("managed", pa.bool_()),
])),
]), metadata={b"hudi_type": b"BLOB"}),
])
@@ -84,7 +85,7 @@ The BLOB internal structure is a struct with three fields:
- `external_path` — file path for out-of-line data
- `offset` — byte offset in the file (null means read from start)
- `length` — byte length to read (null means read to end of file)
- - `managed` — boolean indicating whether Hudi manages the external file
+ - `managed` — boolean. Only meaningful for `OUT_OF_LINE` blobs. Marks
whether Hudi owns the lifecycle of the referenced external file. **Not consumed
by the cleaner yet** — set the value to record intent, and a future cleaner
implementation will use it: `true` → cleaner may delete the external file when
the blob row is no longer referenced; `false` → cleaner will leave the external
file in place.
</TabItem>
</Tabs>
@@ -112,7 +113,7 @@ INSERT INTO media_assets VALUES (
named_struct(
'type', 'INLINE',
'data', /* binary literal or column reference */,
- 'reference', CAST(NULL AS STRUCT<external_path: STRING, offset:
BIGINT, length: BIGINT>)
+ 'reference', CAST(NULL AS STRUCT<external_path: STRING, offset:
BIGINT, length: BIGINT, managed: BOOLEAN>)
)
);
```
@@ -158,7 +159,8 @@ INSERT INTO media_assets VALUES (
'reference', named_struct(
'external_path', 's3://my-bucket/media/container_001.bin',
'offset', 8388608, -- byte offset in the container
- 'length', 1073741824 -- number of bytes
+ 'length', 1073741824, -- number of bytes
+ 'managed', false -- intent flag; not consumed by
the cleaner yet
)
)
);
@@ -290,15 +292,44 @@ Out-of-line BLOBs keep the Hudi table footprint extremely
small:
| Property | Default | Description |
|:---------|:--------|:------------|
-| `hoodie.read.blob.inline.mode` | `CONTENT` | Controls how INLINE BLOBs are
read. `CONTENT` materializes raw bytes in the `data` column. `DESCRIPTOR`
surfaces `(position, size)` coordinates rewritten as OUT_OF_LINE references. |
+| `hoodie.read.blob.inline.mode` | `DESCRIPTOR` | Controls how INLINE BLOBs
are read. `DESCRIPTOR` (default) returns an out-of-line-shaped reference
pointing at the in-file coordinates of the bytes — no bytes are materialized.
`CONTENT` materializes the raw inline bytes directly in the `data` field on
every read. |
| `hoodie.blob.batching.max.gap.bytes` | `4096` | Maximum gap (in bytes)
between consecutive byte ranges before they are merged into a single read.
Larger values reduce I/O calls at the cost of reading some unused bytes. |
| `hoodie.blob.batching.lookahead.size` | `50` | Number of rows to buffer for
batch read detection. Larger values improve batching for sorted data but
increase memory usage. |
:::note
-DESCRIPTOR mode is only supported on Lance-backed tables. CONTENT mode is
always used for internal
-operations (compaction, merge, log replay) regardless of this setting.
+`DESCRIPTOR` mode is the default for all storage formats including Lance.
`CONTENT` mode is always
+used for internal operations (compaction, merge, log replay) regardless of
this setting.
:::
+:::caution Calling read_blob() on INLINE columns under DESCRIPTOR mode
+Under the default `DESCRIPTOR` mode, calling `read_blob()` on an INLINE BLOB
column **throws** —
+the raw bytes are not materialized in the scan, so there is nothing for
`read_blob()` to return.
+To read inline bytes with `read_blob()`, switch to `CONTENT` mode first:
+
+```sql
+SET hoodie.read.blob.inline.mode=CONTENT;
+SELECT asset_id, read_blob(content) AS raw_bytes
+FROM media_assets
+WHERE asset_id = 'asset_001';
+```
+
+This setting affects only INLINE columns — OUT_OF_LINE columns always fetch
from the external path
+regardless of mode.
+:::
+
+## Metastore Sync
+
+When syncing BLOB column schemas to Hive or BigQuery, Hudi maps the BLOB
struct to the target
+catalog's native struct type:
+
+| Catalog | BLOB representation |
+|:--------|:-------------------|
+| Hive | `STRUCT<type:STRING, data:BINARY,
reference:STRUCT<external_path:STRING, offset:BIGINT, length:BIGINT,
managed:BOOLEAN>>` |
+| BigQuery | Equivalent `STRUCT` fields |
+
+The raw binary payload is preserved in the struct representation, but
`read_blob()` is a Spark SQL
+function and is not available in Hive or BigQuery directly.
+
## Best Practices
1. **Choose the right mode** — Use inline for small, frequently-accessed
objects. Use out-of-line for
diff --git a/website/docs/cleaning.md b/website/docs/cleaning.md
index c3498c8aa914..829d0d57cd6d 100644
--- a/website/docs/cleaning.md
+++ b/website/docs/cleaning.md
@@ -3,6 +3,7 @@ title: Cleaning
toc: true
toc_min_heading_level: 2
toc_max_heading_level: 4
+last_modified_at: 2026-05-27T00:00:00-00:00
---
## Background
Cleaning is a table service employed by Hudi to reclaim space occupied by
older versions of data and keep storage costs
@@ -50,6 +51,41 @@ Hudi cleaner currently supports the below cleaning policies
to keep a certain nu
be retained are cleaned. Currently you can configure by parameter
[`hoodie.clean.hours.retained`](https://hudi.apache.org/docs/configurations/#hoodiecleanerhoursretained).
The corresponding Flink related config is
[`clean.retain_hours`](https://hudi.apache.org/docs/configurations/#cleanretain_hours).
+#### Empty Clean Commits for Append-Only Tables
+
+Append-only tables never accumulate updates, so the cleaner's
`earliest_commit_to_retain` pointer never advances —
+causing the cleaner to scan the full table history on every run. Hudi 1.2.0
introduced periodic _empty clean commits_
+to advance this pointer even when there is nothing to delete.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.write.empty.clean.interval.hours` | `-1` (disabled) | Interval in
hours at which an empty clean commit is created. `-1` disables the feature.
Must be `-1` or `>= 1`. When enabled, the cleaner advances
`earliest_commit_to_retain` so that subsequent clean plans only scan partitions
modified after the last empty clean's pointer. |
+
+#### Capping the Number of Commits Cleaned per Run
+
+Since 1.2.0, you can limit how many commits are cleaned in a single clean run,
which is useful for controlling job
+duration on tables that have fallen significantly behind on cleaning.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.clean.max.commits.to.clean` | `Long.MAX_VALUE` (unbounded) | Maximum
number of commits cleaned in a single clean commit. Applicable when the
cleaning policy is `KEEP_LATEST_COMMITS` or `KEEP_LATEST_BY_HOURS`. Must be `>=
1`. |
+
+#### Full-Clean Partition Filtering
+
+When incremental cleaning is disabled
(`hoodie.clean.incremental.enabled=false`), the cleaner scans every partition on
+every run. For very large tables this can cause OOM during planning. Hudi
1.2.0 added two configs to restrict which
+partitions are examined.
+
+:::note
+Both configs require `hoodie.clean.incremental.enabled=false`. If both are
set, `hoodie.clean.partition.filter.selected`
+takes precedence over the regex.
+:::
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.clean.partition.filter.regex` | (none) | Java regex pattern; only
partitions whose path matches are cleaned. |
+| `hoodie.clean.partition.filter.selected` | (none) | Comma-separated list of
partition paths to clean; takes precedence over the regex when both are set. |
+
### Configs
For details about all possible configurations and their default values see the
[configuration
docs](https://hudi.apache.org/docs/next/configurations/#Clean-Configs).
For Flink related configs refer
[here](https://hudi.apache.org/docs/next/configurations/#FLINK_SQL).
@@ -76,6 +112,17 @@ hoodie.clean.async=true
For Flink based writing, this is the default mode of cleaning. Please refer to
[`clean.async.enabled`](https://hudi.apache.org/docs/configurations/#cleanasyncenabled)
for details.
+#### Pre-Write Cleaner Policy
+
+By default the cleaner runs _after_ a write commits. Hudi 1.2.0 introduced
`hoodie.prewrite.cleaner.policy`, which
+lets you force a clean (or rollback of failed writes) _before_ each write
begins. This is useful in multi-writer
+deployments where you want a deterministic table state before every write —
see [concurrency control](concurrency_control.md)
+for related multi-writer configuration.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.prewrite.cleaner.policy` | `NONE` | Pre-write cleaning action.
`NONE`: no pre-write action (default). `CLEAN`: run a clean pass before each
write — this also rolls back failed writes as part of the clean.
`ROLLBACK_FAILED_WRITES`: only roll back any failed writes before each write,
without running a full clean. |
+
#### Run independently
Hoodie Cleaner can also be run as a separate process. Following is the command
for running the cleaner independently:
```
diff --git a/website/docs/cli.md b/website/docs/cli.md
index a29ffdfc2637..ddb8132d3cf3 100644
--- a/website/docs/cli.md
+++ b/website/docs/cli.md
@@ -1,7 +1,7 @@
---
title: CLI
keywords: [hudi, cli]
-last_modified_at: 2021-08-18T15:59:57-04:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
### Local set up
@@ -340,6 +340,13 @@ $ hdfs dfs -ls /app/uber/trips/.hoodie/*.inflight
-rw-r--r-- 3 vinoth supergroup 321984 2016-10-05 23:18
/app/uber/trips/.hoodie/20161005225920.inflight
```
+To list all inflight and requested instants that have been running longer than
a specified number of minutes, use `commits show_inflights`:
+
+```shell
+hudi:trips->commits show_inflights --lookbackInMins 30
+```
+
+This lists every inflight or requested instant whose requested timestamp is
older than 30 minutes, showing the commit time, action type, and current state.
This is useful for detecting hung or stuck writes. The `--lookbackInMins`
option defaults to `0` (returns all inflight/requested instants).
### Drilling Down to a specific Commit
@@ -675,6 +682,22 @@ corresponding to the library release version is used:
upgrade table
```
+### Record Index Lookup
+
+To look up a record's file location via the Record Level Index (RLI) stored in
the Metadata Table:
+
+```shell
+hudi:trips->metadata lookup-record-index --record_key <key>
+```
+
+For a partitioned (non-global) RLI, the partition path is required:
+
+```shell
+hudi:trips->metadata lookup-record-index --record_key <key> --partition_path
<partition>
+```
+
+The `--partition_path` argument is optional for a global RLI (where record
keys are unique across all partitions) and required for a partitioned RLI. If
`--partition_path` is omitted for a partitioned RLI, the command will return an
error. The output columns are `Record key`, `Partition path`, `File Id`, and
`Instant time`.
+
### Change Hudi Table Type
There are cases we want to change the hudi table type. For example, change COW
table to MOR for more efficient and
lower latency ingestion; change MOR to COW for better read performance and
compatibility with downstream engines.
diff --git a/website/docs/clustering.md b/website/docs/clustering.md
index 7442967b6505..7426bf9e2dcc 100644
--- a/website/docs/clustering.md
+++ b/website/docs/clustering.md
@@ -2,7 +2,7 @@
title: Clustering
summary: "In this page, we describe async compaction in Hudi."
toc: true
-last_modified_at: 2025-11-24T02:44:48
+last_modified_at: 2026-05-27T00:00:00-00:00
---
## Background
@@ -134,6 +134,47 @@ dynamically expanding the buckets for bucket index
datasets.
:::note The latter two strategies are applicable only for the Spark engine.
:::
+#### CommitBasedClusteringPlanStrategy
+
+Hudi 1.2.0 introduced
`org.apache.hudi.table.action.cluster.strategy.CommitBasedClusteringPlanStrategy`,
a plan
+strategy that schedules clustering based on commit patterns rather than just
file size. It groups file slices by the
+commits that produced them, making it easier to cluster data written in
specific time windows or under specific commit
+criteria.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.clustering.plan.strategy.class` |
`SparkSizeBasedClusteringPlanStrategy` | Set to
`org.apache.hudi.table.action.cluster.strategy.CommitBasedClusteringPlanStrategy`
to use commit-based planning. |
+| `hoodie.clustering.plan.strategy.earliest.commit.to.cluster` | (none) |
Earliest commit time (exclusive) to start clustering from. Only commits after
this instant are considered. Useful for incrementally clustering new data while
skipping already-clustered history. |
+
+#### SparkStreamCopyClusteringPlanStrategy
+
+Available since Hudi 1.2.0,
`org.apache.hudi.client.clustering.plan.strategy.SparkStreamCopyClusteringPlanStrategy`
+is a Spark-only plan strategy that performs binary file stitching (byte-level
copy) rather than re-reading and
+re-writing records. This can be significantly faster when the goal is simply
to coalesce small files and sort order is
+not required. It is paired with
+`org.apache.hudi.client.clustering.run.strategy.SparkStreamCopyClusteringExecutionStrategy`.
+
+#### Single-Group Clustering Control
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.clustering.plan.strategy.single.group.clustering.enabled` | `true` |
Whether to generate a clustering plan when only one file group is eligible. Set
to `false` to skip clustering when there is nothing meaningful to consolidate
(i.e., the partition already has a single file group). |
+
+#### File-Slice Sort Order in Clustering Plan Generation
+
+Since 1.2.0, the order in which file slices are packed into clustering groups
is configurable, giving more control over
+which files are colocated and how groups are filled.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.clustering.plan.strategy.file.slices.sort.by` | `SIZE` |
Comma-separated list of fields used to sort file slices when packing them into
clustering groups within a partition. `SIZE`: sort by file size descending
(largest first). `INSTANT_TIME`: sort by commit time ascending (oldest files
first). Example: `INSTANT_TIME,SIZE` sorts by commit time then by size. |
+
+#### Driver-Side Plan Generation
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.clustering.plan.generation.use.local.engine.context` | `false` |
When enabled, clustering group computation runs on the driver (local engine
context) instead of being distributed across executors. Enable when there are
only a few partitions with many files, where driver-local computation is more
resource-efficient than allocating executor slots. |
+
### Execution Strategy
After building the clustering groups in the planning phase, Hudi applies
execution strategy, for each group, primarily
@@ -251,6 +292,22 @@ In addition to the basic mode options, HoodieClusteringJob
supports the followin
These retry options are only effective when using `--mode scheduleAndExecute`.
The `--retry-last-failed-job` option requires `--job-max-processing-time-ms` to
be set to a positive value to detect stale inflight instants.
:::
+#### Automatic Expiration of Stale Clustering Instants
+
+When a clustering job is scheduled but never successfully executed (e.g., due
to a driver failure), the inflight
+`replacecommit` instant blocks future clustering runs. Hudi 1.2.0 adds
automatic expiration of such stale clustering
+instants, complementing the manual retry options above.
+
+:::note
+Expired clustering plan cleanup requires
`hoodie.clean.failed.writes.policy=LAZY`. With LAZY cleaning, the rollback of
+failed writes (triggered on the next write) also rolls back expired clustering
instants.
+:::
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.clustering.enable.expirations` | `false` | When enabled, rollback of
failed writes (under LAZY cleaning) also rolls back clustering `replacecommit`
instants whose heartbeat has expired. Clustering jobs record a heartbeat before
scheduling so other writers can detect stale attempts. |
+| `hoodie.clustering.expiration.threshold.mins` | `60` | A clustering instant
is not considered expired unless its creation time is at least this many
minutes old. Acts as a guardrail to avoid rolling back clustering attempts that
are still in progress. |
+
Note that to run this job while the original writer is still running, please
enable multi-writing:
```properties
diff --git a/website/docs/compaction.md b/website/docs/compaction.md
index 89b9214f0bd9..bf1dee7c12d6 100644
--- a/website/docs/compaction.md
+++ b/website/docs/compaction.md
@@ -4,7 +4,7 @@ summary: "In this page, we describe async compaction in Hudi."
toc: true
toc_min_heading_level: 2
toc_max_heading_level: 4
-last_modified_at: 2025-11-24T02:44:48
+last_modified_at: 2026-05-27T00:00:00-00:00
---
## Background
@@ -104,6 +104,17 @@ BoundedPartitionAwareCompactionStrategy</li></ul>
Please refer to [advanced
configs](https://hudi.apache.org/docs/next/configurations#Compaction-Configs)
for more details.
:::
+#### Metadata Table Compaction Trigger Strategy
+
+Available since Hudi 1.2.0, the metadata table (MDT) supports the same set of
compaction trigger strategies as the
+data table, plus a time-based option.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.metadata.compact.trigger.strategy` | `NUM_COMMITS` | Trigger
strategy for MDT compaction. Accepts the same values as
`hoodie.compact.inline.trigger.strategy`: `NUM_COMMITS`,
`NUM_COMMITS_AFTER_LAST_REQUEST`, `TIME_ELAPSED`, `NUM_AND_TIME`,
`NUM_OR_TIME`. |
+| `hoodie.metadata.compact.max.delta.commits` | `10` | Number of delta commits
after the last MDT compaction before a new one is scheduled (for
`NUM_COMMITS`-based strategies). |
+| `hoodie.metadata.compact.max.delta.seconds` | `7200` | Elapsed seconds after
the last MDT compaction before scheduling a new one. Takes effect only for
`TIME_ELAPSED`, `NUM_AND_TIME`, and `NUM_OR_TIME` strategies. |
+
## Ways to trigger Compaction
### Inline
diff --git a/website/docs/concurrency_control.md
b/website/docs/concurrency_control.md
index d6231aca79b7..2056acfca64c 100644
--- a/website/docs/concurrency_control.md
+++ b/website/docs/concurrency_control.md
@@ -4,7 +4,7 @@ summary: On this page, we discuss how to perform concurrent
writes to Hudi table
toc: true
toc_min_heading_level: 2
toc_max_heading_level: 4
-last_modified_at: 2025-11-23T14:20:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
Concurrency control defines how different writers, readers, and table services
coordinate access to a Hudi table. Hudi ensures atomic writes by publishing
commits atomically to the timeline, stamped with an instant time that denotes
when the action is deemed to have occurred. Unlike general-purpose file version
control, Hudi draws a clear distinction between writer processes that issue
[write operations](write_operations.md), table services that (re)write
data/metadata to optimize or per [...]
@@ -47,6 +47,7 @@ Add the corresponding cloud bundle to your classpath:
* For S3: `hudi-aws-bundle`
* For GCS: `hudi-gcp-bundle`
+* For Azure (`abfs://`, `abfss://`, `wasb://`, `wasbs://`): `hudi-azure-bundle`
Set this configuration:
@@ -54,7 +55,7 @@ Set this configuration:
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.StorageBasedLockProvider
```
-Supported for S3 and GCS (additional systems planned). This cloud-native
design works directly with storage features, simplifying large-scale cloud
operations.
+Supported for S3, GCS, and Azure ADLS Gen2 / Azure Blob Storage. This
cloud-native design works directly with storage features, simplifying
large-scale cloud operations.
Optional tuning configurations:
@@ -63,6 +64,27 @@ Optional tuning configurations:
| hoodie.write.lock.storage.validity.timeout.secs | 300 (Optional) | Validity
period (seconds) for each new lock. The provider renews its lock until the
lease extends or timeout occurs.<br /><br />`Config Param:
STORAGE_BASED_LOCK_VALIDITY_TIMEOUT_SECS`<br />`Since Version: 1.0.2` |
| hoodie.write.lock.storage.renew.interval.secs | 30 (Optional) | Interval
(seconds) between renewal attempts.<br /><br />`Config Param:
STORAGE_BASED_LOCK_RENEW_INTERVAL_SECS`<br />`Since Version: 1.0.2`
|
+#### Azure Storage-Based Lock
+
+Authentication is resolved in the following precedence order:
+
+| Priority | Config Key | Description |
+|----------|------------|-------------|
+| 1 (highest) | `hoodie.write.lock.azure.connection.string` | Azure Storage
connection string |
+| 2 | `hoodie.write.lock.azure.sas.token` | SAS token (not recommended for
production by Azure) |
+| 3 | `hoodie.write.lock.azure.managed.identity.client.id` | Client ID of a
user-assigned managed identity (`ManagedIdentityCredential`) |
+| 4 | `hoodie.write.lock.azure.client.tenant.id` + `.client.id` +
`.client.secret` | Service principal via `ClientSecretCredential` — all three
must be set |
+| 5 (lowest) | _(none)_ | `DefaultAzureCredential` chain (system-assigned
managed identity, environment variables, etc.) |
+
+Example configuration for service-principal authentication:
+
+```properties
+hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.StorageBasedLockProvider
+hoodie.write.lock.azure.client.tenant.id=<your-tenant-id>
+hoodie.write.lock.azure.client.id=<your-app-client-id>
+hoodie.write.lock.azure.client.secret=<your-client-secret>
+```
+
### Zookeeper-Based Lock Provider
```properties
@@ -359,6 +381,22 @@ hoodie.write.lock.client.num_retries
*Setting the right values for these depends on a case by case basis; some
defaults have been provided for general cases.*
+## Pre-Write Cleaner Policy
+
+When running multi-writer pipelines, failed writes can accumulate on storage
if a writer crashes before a clean cycle runs. Hudi 1.2.0 introduces
`hoodie.prewrite.cleaner.policy` to proactively handle this at write startup:
+
+| Config Key | Default | Description |
+|---|---|---|
+| `hoodie.prewrite.cleaner.policy` | `NONE` | Policy applied before starting a
new ingestion write commit. `NONE`: no pre-write action (default). `CLEAN`:
force a clean table service call (also rolls back failed writes).
`ROLLBACK_FAILED_WRITES`: only roll back failed writes without running a full
clean. |
+
+This is useful when a writer is perpetually crashing before completing a
`CLEAN`. See [Cleaning](cleaning.md) for the full list of cleaning
configurations.
+
+## Lock Audit Logging and Diagnostics
+
+The storage-based lock provider supports optional audit logging of lock
operations. When enabled, a `.hoodie/lock/audit_enabled.json` marker is written
to the table base path and lock acquisition/release events are recorded for
post-hoc debugging.
+
+For ZooKeeper-based locking, the ZK lock node now stores the Spark application
ID of the writer holding the lock, making it easier to correlate lock holders
with running Spark jobs in cluster UIs.
+
## Caveats
If you are using the `WriteClient` API, please note that multiple writes to
the table need to be initiated from 2 different instances of the write client.
diff --git a/website/docs/deployment.md b/website/docs/deployment.md
index 51f92e40d407..62497285a333 100644
--- a/website/docs/deployment.md
+++ b/website/docs/deployment.md
@@ -3,7 +3,7 @@ title: Deployment
keywords: [ hudi, administration, operation, devops, deployment]
summary: This section offers an overview of tools available to operate an
ecosystem of Hudi
toc: true
-last_modified_at: 2019-12-30T15:59:57-04:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
This section provides all the help you need to deploy and operate Hudi tables
at scale.
@@ -32,6 +32,8 @@ from varied sources such as DFS, Kafka and DB Changelogs and
ingest them to hudi
To use Hudi Streamer in Spark, the `hudi-utilities-slim-bundle` and Hudi Spark
bundle are required, by adding
`--packages
org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.0.1,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.0.1`
to the `spark-submit` command.
+Pick the Spark bundle that matches your Spark runtime — for example,
`hudi-spark3.3-bundle_2.12`, `hudi-spark3.4-bundle_2.12`,
`hudi-spark3.5-bundle_2.12` (Scala 2.12 or 2.13), `hudi-spark3.5-bundle_2.13`,
`hudi-spark4.0-bundle_2.13`, or `hudi-spark4.1-bundle_2.13`. Spark 4.0 and 4.1
require Java 17 or later at runtime; Spark 3.x runs on Java 8 or later.
+
- **Run Once Mode** : In this mode, Hudi Streamer performs one ingestion round
which includes incrementally pulling events from upstream sources and ingesting
them to hudi table. Background operations like cleaning old file versions and
archiving hoodie timeline are automatically executed as part of the run. For
Merge-On-Read tables, Compaction is also run inline as part of ingestion unless
disabled by passing the flag "--disable-compaction". By default, Compaction is
run inline for ever [...]
Here is an example invocation for reading from kafka topic in a single-run
mode and writing to Merge On Read table type in a yarn cluster.
diff --git a/website/docs/flink-quick-start-guide.md
b/website/docs/flink-quick-start-guide.md
index 2a04dc85abb2..23ef5efbe3cc 100644
--- a/website/docs/flink-quick-start-guide.md
+++ b/website/docs/flink-quick-start-guide.md
@@ -1,7 +1,7 @@
---
title: "Flink Quick Start"
toc: true
-last_modified_at: 2025-11-22T14:30:00+08:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -12,12 +12,13 @@ This page introduces Flink–Hudi integration and
demonstrates how Flink brings
### Flink Support Matrix
-| Hudi | Supported Flink versions
|
-| :----- |
:--------------------------------------------------------------------- |
-| 1.1.x | 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build), 2.0.x
|
-| 1.0.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x, 1.19.x, 1.20.x (default
build) |
-| 0.15.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x
|
-| 0.14.x | 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x
|
+| Hudi | Supported Flink versions
|
+| :----- |
:---------------------------------------------------------------------------- |
+| 1.2.x | 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build), 2.0.x, 2.1.x
|
+| 1.1.x | 1.17.x, 1.18.x, 1.19.x, 1.20.x (default build), 2.0.x
|
+| 1.0.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x, 1.19.x, 1.20.x (default
build) |
+| 0.15.x | 1.14.x, 1.15.x, 1.16.x, 1.17.x, 1.18.x
|
+| 0.14.x | 1.13.x, 1.14.x, 1.15.x, 1.16.x, 1.17.x
|
### Download Flink and Start Flink cluster
@@ -62,9 +63,9 @@ You can build the jar manually under path
`hudi-source-dir/packaging/hudi-flink-
Now start the SQL CLI:
```bash
-# For Flink versions: 1.17-1.20, 2.0
-export FLINK_VERSION=1.20
-export HUDI_VERSION=1.1.1
+# Supported Flink versions for Hudi 1.2.x: 1.17, 1.18, 1.19, 1.20 (default
build), 2.0, 2.1
+export FLINK_VERSION=1.20
+export HUDI_VERSION=1.2.0
wget
https://repo1.maven.org/maven2/org/apache/hudi/hudi-flink${FLINK_VERSION}-bundle/${HUDI_VERSION}/hudi-flink${FLINK_VERSION}-bundle-${HUDI_VERSION}.jar
-P /tmp/
./bin/sql-client.sh embedded -j
/tmp/hudi-flink${FLINK_VERSION}-bundle-${HUDI_VERSION}.jar shell
```
@@ -77,11 +78,11 @@ The SQL CLI only executes the SQL line by line.
Please add the desired dependency to your project:
```xml
-<!-- For Flink versions 1.17-1.20, 2.0-->
+<!-- Supported Flink versions for Hudi 1.2.x: 1.17, 1.18, 1.19, 1.20 (default
build), 2.0, 2.1 -->
<properties>
- <flink.version>1.20.0</flink.version>
+ <flink.version>1.20.1</flink.version>
<flink.binary.version>1.20</flink.binary.version>
- <hudi.version>1.1.1</hudi.version>
+ <hudi.version>1.2.0</hudi.version>
</properties>
<dependency>
<groupId>org.apache.hudi</groupId>
@@ -446,9 +447,9 @@ feature is that it lets you author streaming pipelines on
streaming or batch dat
- **Quick Start**: Read the Quick Start section above to get started quickly
with the Flink SQL client to write to (and read from) Hudi.
- **Configuration**: For [Global
Configuration](flink_tuning.md#global-configurations), set up through
`$FLINK_HOME/conf/flink-conf.yaml`. For per-job configuration, set up through
[Table Option](flink_tuning.md#table-options).
-- **Writing Data** : Flink supports different modes for writing, such as [CDC
Ingestion](ingestion_flink.md#cdc-ingestion), [Bulk
Insert](ingestion_flink.md#bulk-insert), [Index
Bootstrap](ingestion_flink.md#index-bootstrap), [Changelog
Mode](ingestion_flink.md#changelog-mode) and [Append
Mode](ingestion_flink.md#append-mode). Flink also supports multiple streaming
writers with [non-blocking concurrency
control](sql_dml.md#non-blocking-concurrency-control-experimental).
-- **Reading Data** : Flink supports different modes for reading, such as
[Streaming Query](sql_queries.md#streaming-query) and [Incremental
Query](sql_queries.md#incremental-query).
-- **Tuning**: For write/read tasks, this guide provides some tuning
suggestions, such as [Memory Optimization](flink_tuning.md#memory-optimization)
and [Write Rate Limit](flink_tuning.md#write-rate-limit).
+- **Writing Data** : Flink supports different modes for writing, such as [CDC
Ingestion](ingestion_flink.md#cdc-ingestion), [Bulk
Insert](ingestion_flink.md#bulk-insert), [Index
Bootstrap](ingestion_flink.md#index-bootstrap), [Changelog
Mode](ingestion_flink.md#changelog-mode) and [Append
Mode](ingestion_flink.md#append-mode). For high-throughput append pipelines,
choose an [append write buffer mode](ingestion_flink.md#append-write-buffer).
For upsert workloads at scale, use [Record-Leve [...]
+- **Reading Data** : Flink supports different modes for reading, such as
[Streaming Query](sql_queries.md#streaming-query) and [Incremental
Query](sql_queries.md#incremental-query). For improved push-down and resumable
reads, see [Flink Source V2](ingestion_flink.md#flink-source-v2). For
dimension-table joins, use [lookup join](ingestion_flink.md#lookup-join) with
an optional off-heap RocksDB cache.
+- **Tuning**: For write/read tasks, this guide provides some tuning
suggestions, such as [Memory
Optimization](flink_tuning.md#memory-optimization), the [Managed-Memory Write
Buffer](flink_tuning.md#managed-memory-write-buffer), and [Write Rate
Limit](flink_tuning.md#write-rate-limit).
- **Optimization**: Offline compaction is supported: [Offline
Compaction](compaction.md#flink-offline-compaction).
- **Query Engines**: Besides Flink, many other engines are integrated: [Hive
Query](syncing_metastore.md#flink-setup), [Presto Query](sql_queries.md#presto).
- **Catalog**: A Hudi‑specific catalog is supported: [Hudi
Catalog](sql_ddl/#create-catalog).
diff --git a/website/docs/flink_tuning.md b/website/docs/flink_tuning.md
index 28e70e48b1f2..bd08b4be0fba 100644
--- a/website/docs/flink_tuning.md
+++ b/website/docs/flink_tuning.md
@@ -115,3 +115,50 @@ the `write.rate.limit` option can be turned on to ensure
smooth writing.
| Option Name | Required | Default | Remarks |
| ----------- | ------- | ------- | ------- |
| `write.rate.limit` | `false` | `0` | Turn off by default |
+
+## Managed-Memory Write Buffer
+
+By default, the Flink write buffer uses JVM heap memory (`ON_HEAP`). In
containerized environments where heap memory is tightly budgeted, you can
switch to Flink's managed (off-heap) memory pool to reduce GC pressure and
avoid OOM errors.
+
+:::note
+When using `MANAGED` memory type, ensure `taskmanager.memory.managed.size` is
configured sufficiently in `flink-conf.yaml`.
+:::
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.buffer.memory.type` | Memory type for the write buffer: `ON_HEAP`
(default, uses JVM heap) or `MANAGED` (uses Flink managed off-heap memory) |
`ON_HEAP` | Switch to `MANAGED` to avoid OOM in memory-constrained deployments |
+| `write.memory.segment.page.size` | Page size in bytes for memory segments
used in the write buffer | `32768` (32 KB) | Tune for workload characteristics;
larger pages reduce overhead for large records |
+
+## Disruptor Buffer Tuning
+
+When `write.buffer.type=DISRUPTOR` is set in the table options (see [Append
Write Buffer](ingestion_flink.md#append-write-buffer)), the following tuning
options control the Disruptor ring buffer:
+
+| Option Name | Description | Default | Remarks |
+| ----------- | ------- | ------- | ------- |
+| `write.buffer.disruptor.ring.size` | Size of the Disruptor ring buffer (must
be a power of 2) | `16384` | Larger values absorb write bursts but consume more
heap memory |
+| `write.buffer.disruptor.wait.strategy` | Wait strategy for the Disruptor
consumer: `BLOCKING_WAIT` (default), `SLEEPING_WAIT`, `YIELDING_WAIT`,
`BUSY_SPIN_WAIT` | `BLOCKING_WAIT` | `BLOCKING_WAIT` is safest for
containerized environments; `BUSY_SPIN_WAIT` offers lowest latency at the cost
of a dedicated CPU core |
+
+## Timeline-Server-Based Markers
+
+As of Hudi 1.2.0, Flink writers support `TIMELINE_SERVER_BASED` marker type
(`hoodie.write.markers.type=TIMELINE_SERVER_BASED`). This is recommended over
`DIRECT` markers on object stores (S3, GCS, ADLS) where the high cost of
directory listings makes `DIRECT` markers slow.
+
+```sql
+CREATE TABLE my_table (...)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 's3a://my-bucket/my-table',
+ 'hoodie.write.markers.type' = 'TIMELINE_SERVER_BASED'
+ -- other options
+);
+```
+
+## Source V2 Read-Lag Metrics
+
+When [Source V2](ingestion_flink.md#flink-source-v2) is enabled
(`read.source-v2.enabled=true`), the following read-lag metrics are emitted to
help monitor streaming pipeline health:
+
+| Metric | Description |
+|--------|-------------|
+| `issuedInstantDelay` | Time elapsed (ms) between when a new instant was
written and when the source issued it for reading |
+| `sourceReaderIdleTime` | Time (ms) the source reader has been idle (no new
splits assigned) |
+
+These metrics are exposed through Flink's standard metrics system and can be
forwarded to Prometheus, JMX, or other reporters.
diff --git a/website/docs/hoodie_streaming_ingestion.md
b/website/docs/hoodie_streaming_ingestion.md
index 286d5765c751..1fb282a5d718 100644
--- a/website/docs/hoodie_streaming_ingestion.md
+++ b/website/docs/hoodie_streaming_ingestion.md
@@ -146,7 +146,9 @@ Usage: <main class> [options]
Default: 0
--op
Takes one of these values : UPSERT (default), INSERT, BULK_INSERT,
- INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE, DELETE_PARTITION
+ INSERT_OVERWRITE, INSERT_OVERWRITE_TABLE, DELETE_PARTITION, DELETE
+ (DELETE extracts HoodieKeys from source records and deletes the
+ corresponding records from the table.)
Default: UPSERT
Possible Values: [INSERT, INSERT_PREPPED, UPSERT, UPSERT_PREPPED,
BULK_INSERT, BULK_INSERT_PREPPED, DELETE, DELETE_PREPPED, BOOTSTRAP,
INSERT_OVERWRITE, CLUSTER, DELETE_PARTITION, INSERT_OVERWRITE_TABLE, COMPACT,
INDEX, ALTER_SCHEMA, LOG_COMPACT, UNKNOWN]
--payload-class
@@ -503,6 +505,77 @@ Check out [Kafka source
config](https://hudi.apache.org/docs/configurations#Kafk
Hudi Streamer also supports ingesting from Apache Pulsar via
`org.apache.hudi.utilities.sources.PulsarSource`.
Check out [Pulsar source
config](https://hudi.apache.org/docs/configurations#Pulsar-Source-Configs) for
more details.
+#### Amazon Kinesis
+
+Use the `JsonKinesisSource`
(`org.apache.hudi.utilities.sources.JsonKinesisSource`) to ingest JSON records
from an AWS Kinesis Data Stream into a Hudi table. It reads from every shard in
parallel, tracks per-shard progress in the Hudi Streamer checkpoint,
automatically handles shard splits and merges, and de-aggregates records
produced by the Kinesis Producer Library (KPL).
+
+##### Common configuration
+
+All keys use the prefix `hoodie.streamer.source.kinesis.`. The settings most
users need:
+
+| Config key | Default | Description |
+|---|---|---|
+| `hoodie.streamer.source.kinesis.stream.name` | (required) | Kinesis Data
Streams stream name. |
+| `hoodie.streamer.source.kinesis.region` | (required) | AWS region for the
stream (e.g., `us-east-1`). |
+| `hoodie.streamer.source.kinesis.starting.position` | `LATEST` | Where to
start when no checkpoint exists yet. `LATEST` starts at the tip of each shard;
`EARLIEST` replays from `TRIM_HORIZON`. |
+| `hoodie.streamer.source.kinesis.max.events` | `5000000` | Maximum number of
records read per batch across all shards. Tune to control batch size. |
+| `hoodie.streamer.source.kinesis.partitions` | `0` | Spark partitions to use
when reading. `0` means one Spark partition per Kinesis shard. Set a positive
value to repartition for downstream parallelism. |
+
+For credentials, the source uses the default AWS credential chain (instance
profile, environment variables, etc.). Authentication for custom endpoints
(e.g., LocalStack), API-level rate limiting, and retry tuning are also
available — see the [configurations reference](configurations.md) for the full
list of `hoodie.streamer.source.kinesis.*` keys.
+
+##### Checkpoint format
+
+Hudi Streamer persists Kinesis progress as a single checkpoint string on the
timeline. Each batch advances the checkpoint to the last record successfully
read from every shard, so a failed batch can be retried without skipping or
duplicating records.
+
+The checkpoint encodes per-shard state in plain text:
+
+```
+streamName,shardId:value,shardId:value,...
+```
+
+Each `value` is one of:
+
+- `lastSeq` — last sequence number consumed from an open shard.
+- `lastSeq@arrivalTime` — same, with the record's approximate arrival time
(epoch millis) for lag/observability.
+- `lastSeq|endSeq` — closed shard. `endSeq` is the shard's final sequence
number, used to detect data loss if the shard expires before being fully
consumed.
+- `lastSeq@arrivalTime|endSeq` — closed shard with arrival time.
+
+Example (sequence numbers abbreviated; Kinesis assigns each shard a 56-digit
decimal sequence number):
+
+```
+my-stream,shardId-000000000000:49590…88898,shardId-000000000001:49590…96306
+```
+
+You don't need to construct or parse this string yourself — it is read and
updated automatically by the source — but it's useful for debugging, manual
checkpoint resets, or comparing progress across shards.
+
+##### Minimal spark-submit example
+
+```properties
+# kinesis-source.properties
+hoodie.streamer.source.kinesis.stream.name=my-stream
+hoodie.streamer.source.kinesis.region=us-east-1
+hoodie.streamer.source.kinesis.starting.position=LATEST
+
+# Standard Hudi write / key-gen configs
+hoodie.datasource.write.recordkey.field=id
+hoodie.datasource.write.partitionpath.field=event_date
+hoodie.table.ordering.fields=ts
+```
+
+```bash
+spark-submit \
+ --packages
org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.2.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0
\
+ --class org.apache.hudi.utilities.streamer.HoodieStreamer \
+ hudi-utilities-slim-bundle-*.jar \
+ --props kinesis-source.properties \
+ --source-class org.apache.hudi.utilities.sources.JsonKinesisSource \
+ --table-type COPY_ON_WRITE \
+ --target-base-path s3://my-bucket/hudi/my-table \
+ --target-table my_db.my_table \
+ --op UPSERT \
+ --continuous
+```
+
#### Cloud storage event sources
AWS S3 storage provides an event notification service which will post
notifications when certain events happen in your S3 bucket:
https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html
@@ -636,3 +709,34 @@ to how you run Hudi Streamer.
```
For detailed information on how to configure and use
`HoodieMultiTableStreamer`, please refer [blog
section](/blog/2020/08/22/ingest-multiple-tables-using-hudi).
+
+## On-Demand Hive Sync (HudiHiveSyncJob)
+
+`org.apache.hudi.utilities.HudiHiveSyncJob` is a standalone Spark job that
syncs a Hudi table's metadata to Hive metastore independently of any ingestion
workflow. It is useful for backfills, manual data corrections, or reconciling
metastore metadata after direct writes.
+
+### Arguments
+
+| Argument | Required | Description |
+|---|---|---|
+| `--base-path` / `-sp` | Yes | Base path of the Hudi table. |
+| `--base-file-format` / `-bff` | No | Base file format. Default: `PARQUET`. |
+| `--props-file-path` | No | Path to a properties file with Hudi / Hive sync
configs. |
+| `--hoodie-conf` | No | Inline config override (repeatable). |
+| `--spark-master` | No | Spark master URL. Inherits from environment if
unset. |
+
+### Example
+
+```bash
+spark-submit \
+ --packages
org.apache.hudi:hudi-utilities-slim-bundle_2.12:1.2.0,org.apache.hudi:hudi-spark3.5-bundle_2.12:1.2.0
\
+ --class org.apache.hudi.utilities.HudiHiveSyncJob \
+ hudi-utilities-slim-bundle-*.jar \
+ --base-path s3://my-bucket/hudi/my-table \
+ --base-file-format PARQUET \
+ --hoodie-conf hoodie.datasource.hive_sync.mode=hms \
+ --hoodie-conf
hoodie.datasource.hive_sync.metastore.uris=thrift://hive-metastore:9083 \
+ --hoodie-conf hoodie.datasource.hive_sync.database=my_db \
+ --hoodie-conf hoodie.datasource.hive_sync.table=my_table
+```
+
+All `hoodie.datasource.hive_sync.*` options accepted by the DataSource writer
are also accepted here. See [Syncing to Hive Metastore](syncing_metastore.md)
for the full list.
diff --git a/website/docs/ingestion_flink.md b/website/docs/ingestion_flink.md
index e720a748c1c8..2e534e286d67 100644
--- a/website/docs/ingestion_flink.md
+++ b/website/docs/ingestion_flink.md
@@ -1,7 +1,7 @@
---
title: Using Flink
keywords: [hudi, flink, streamer, ingestion]
-last_modified_at: 2025-11-22T12:53:57+08:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
## CDC Ingestion
@@ -112,15 +112,24 @@ the compaction options `compaction.delta_commits` and
`compaction.delta_seconds`
For `INSERT` mode write operations, new Parquet files are written directly,
and the [auto‑file sizing](file_sizing.md) is not enabled.
-### In-Memory Buffer Sort
+### Append Write Buffer
-For append-only workloads, Hudi supports in-memory buffer sorting to improve
Parquet compression ratio. When enabled, data is sorted within the write buffer
before being flushed to disk. This improves columnar file compression
efficiency by grouping similar values together.
+For append-only workloads, Hudi supports several write-buffer strategies that
improve Parquet compression ratio and write throughput. Data is sorted or
batched within the write buffer before being flushed to disk, grouping similar
values together for better columnar compression.
-| Option Name | Required | Default | Remarks
|
-|-----------------------------|----------|---------|-------------------------------------------------------------------------------------------------------------------------------|
-| `write.buffer.sort.enabled` | `false` | `false` | Whether to enable buffer
sort within append write function. Improves Parquet compression ratio by
sorting data before writing |
-| `write.buffer.sort.keys` | `false` | `N/A` | Sort keys concatenated by
comma (e.g., `col1,col2`). Required when `write.buffer.sort.enabled` is `true`
|
-| `write.buffer.size` | `false` | `1000` | Buffer size in number of
records. When buffer reaches this size, data is sorted and flushed to disk
|
+The buffer strategy is selected with `write.buffer.type`. In Hudi 1.2.0 this
replaces the deprecated `write.buffer.sort.enabled` flag.
+
+| Option Name | Required | Default | Remarks
|
+|------------------------------------------|----------|------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `write.buffer.type` | `false` | `NONE` | Buffer
type for append write. Values: `NONE` (no buffering), `BOUNDED_IN_MEMORY`
(double buffer with async write), `DISRUPTOR` (ring-buffer with async write,
recommended for higher throughput), `CONTINUOUS_SORT` (TreeMap-based continuous
sort with incremental draining) |
+| `write.buffer.size` | `false` | `1000` | Record
count threshold at which the buffer is flushed. Applies to all non-`NONE`
buffer types
|
+| `write.buffer.sort.keys` | `false` | `N/A` |
Comma-separated sort key columns (e.g., `col1,col2`). Required for `DISRUPTOR`
and `CONTINUOUS_SORT` modes
|
+| `write.buffer.sort.continuous.drain.size`| `false` | `1` | Number of
records drained per flush cycle in `CONTINUOUS_SORT` mode. Default 1 provides
smooth incremental draining; increase for batching (e.g., 10–100)
|
+
+:::note
+`write.buffer.sort.enabled` is deprecated as of 1.2.0. Use
`write.buffer.type=DISRUPTOR` instead for equivalent behavior. The `DISRUPTOR`
and `CONTINUOUS_SORT` modes require `write.buffer.sort.keys` to be set.
+:::
+
+For Disruptor-specific tuning options, see
[flink_tuning.md](flink_tuning.md#disruptor-buffer-tuning).
### Disable Meta Fields
@@ -156,7 +165,7 @@ Only Copy‑on‑Write tables are supported.
### Clustering Plan Strategy
-Custom clustering strategy is supported.
+Custom clustering strategy is supported. Hudi 1.2.0 adds
`FlinkSkipSingleFileClusteringPlanStrategy`
(`org.apache.hudi.client.clustering.plan.strategy.FlinkSkipSingleFileClusteringPlanStrategy`),
which skips file groups that already consist of a single file, reducing
unnecessary rewrites.
| Option Name | Required | Default
| Remarks
|
|---------------------------------------------------------|----------|---------|--------------------------------------------------------------------------------------------------------------------------------------------------|
@@ -185,7 +194,7 @@ Hudi Flink writer supports two types of writer indexes:
| Cross‑Partition Changes | Cannot handle changes among partitions (unless
input is a CDC stream)
| No limit on
handling cross‑partition changes
|
:::note
-Bucket index supports only the `UPSERT` write operation and cannot be used
with the [append mode](#append-mode) in Flink.
+Bucket index supports `UPSERT` write operations on both COW and MOR tables. As
of Hudi 1.2.0, MOR + bucket index + upsert is fully supported. Bucket index
cannot be used with the [append mode](#append-mode) in Flink.
:::
### Bucket Index Examples
@@ -349,10 +358,215 @@ For Flink streaming reads, rate limiting helps avoid
backpressure when processin
The average read rate can be calculated as: **`read.splits.limit` /
`read.streaming.check-interval`** splits per second.
+Hudi 1.2.0 adds `read.commits.limit`, which complements `read.splits.limit` by
capping the number of commits (instants) consumed per check interval. This is
useful when tables have many small commits — limiting commits bounds the number
of splits regardless of their individual size.
+
+### Options
+
+| Option Name | Required | Default | Remarks
|
+|---------------------------------|----------|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `write.rate.limit` | `false` | `0` | Write
record rate limit per second to prevent traffic jitter and improve stability.
Default is 0 (no limit)
|
+| `read.splits.limit` | `false` | `Integer.MAX_VALUE` | Maximum
number of splits allowed to read in each instant check for streaming reads.
Average read rate = `read.splits.limit`/`read.streaming.check-interval`.
Default is no limit |
+| `read.commits.limit` | `false` | `(none)` | Maximum
number of commits (instants) allowed to read in each check interval.
Complements `read.splits.limit`. Average rate =
`read.commits.limit`/`read.streaming.check-interval`. Default is no limit |
+| `read.streaming.check-interval` | `false` | `60` | Check
interval in seconds for streaming reads. Default is 60 seconds (1 minute)
|
+
+## Flink Source V2
+
+Hudi 1.2.0 introduces a new Flink source implementation
([RFC-95](https://github.com/apache/hudi/blob/master/rfc/rfc-95/rfc-95.md))
based on
[FLIP-27](https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface),
available as an opt-in feature via the `read.source-v2.enabled` flag.
+
+### Why Source V2?
+
+The legacy Hudi Flink source was built on Flink's `SourceFunction` API. The
FLIP-27 rewrite brings:
+
+- **Resumable split assignment** — splits can be checkpointed independently,
enabling finer-grained recovery
+- **Checkpoint alignment** — the new API participates in Flink's coordinated
checkpoint protocol, improving end-to-end consistency
+- **Push-down support** — predicate push-down, partition pruning, and `LIMIT`
push-down are supported through the new source interface, reducing data scanned
at the source level
+
+### Enabling Source V2
+
+```sql
+CREATE TABLE t1 (
+ uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${path}',
+ 'table.type' = 'MERGE_ON_READ',
+ 'read.source-v2.enabled' = 'true' -- enable the FLIP-27 source
+);
+```
+
+### Options
+
+| Option Name | Required | Default | Remarks
|
+|---------------------------|----------|---------|---------------------------------------------------------------------------------------------------------|
+| `read.source-v2.enabled` | `false` | `false` | Whether to use the FLIP-27
new source (Source V2) to consume data files. Default is the legacy source |
+
+### Savepoint Incompatibility
+
+:::warning
+Savepoints taken with the **legacy source** (`read.source-v2.enabled=false`)
are **not compatible** with the Source V2 source, and vice versa. When
switching from the legacy source to Source V2, start a fresh job without
restoring from a legacy savepoint. If you need to preserve read progress,
record the last committed instant time and use `read.start-commit` to resume
from that point.
+:::
+
+## Record-Level Index (RLI) Bucket Indexing for Flink
+
+As of Hudi 1.2.0, the Flink writer supports the Record-Level Index (RLI)
backed by the metadata table, in addition to the existing `FLINK_STATE` and
`BUCKET` index types. RLI is stored in the metadata table and avoids the
state-backend overhead of `FLINK_STATE`, while supporting full global or
partition-scoped uniqueness guarantees.
+
+Two RLI variants are available via `index.type`:
+
+- `RECORD_LEVEL_INDEX` — partitioned RLI; enforces uniqueness per (partition
path, record key) pair
+- `GLOBAL_RECORD_LEVEL_INDEX` — global RLI; enforces uniqueness across all
partitions
+
+### Bootstrap
+
+When enabling RLI on an existing table, the bootstrap process loads existing
record locations into RocksDB before the first write. Bootstrap is triggered by
setting `index.bootstrap.enabled=true`.
+
+```sql
+CREATE TABLE my_hudi_table (
+ id BIGINT,
+ name STRING,
+ ts BIGINT,
+ dt STRING,
+ PRIMARY KEY (id) NOT ENFORCED
+)
+PARTITIONED BY (dt)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'hdfs:///warehouse/my_hudi_table',
+ 'table.type' = 'MERGE_ON_READ',
+ 'index.type' = 'RECORD_LEVEL_INDEX',
+ 'metadata.enabled' = 'true',
+ 'index.bootstrap.enabled' = 'true', -- enable bootstrap on first run
+ 'index.bootstrap.rocksdb.path' = '/tmp/hudi-rli-rocksdb'
+);
+```
+
+Once bootstrap completes (after the first successful checkpoint), you can
optionally restart the job with `index.bootstrap.enabled=false` to skip the
bootstrap operators. Leaving them enabled is harmless — they become no-ops on
subsequent runs and do not affect write performance.
+
+### In-Pipeline MDT Compaction
+
+For RLI workloads, the metadata table (MDT) accumulates log files that need
periodic compaction. The option `metadata.compaction.async.enabled` (default
`true`) runs MDT compaction inside the Flink pipeline after every
`metadata.compaction.delta_commits` (default `10`) delta commits.
+
+### Options
+
+| Option Name | Required | Default | Remarks
|
+|-------------------------------------|----------|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `index.type` | `false` | `FLINK_STATE` | Set to
`RECORD_LEVEL_INDEX` or `GLOBAL_RECORD_LEVEL_INDEX` to use the
metadata-table-backed RLI |
+| `index.bootstrap.enabled` | `false` | `false` | Bootstrap the
index from the existing table on first run. Blocks checkpoints during bootstrap
|
+| `index.bootstrap.rocksdb.path` | `false` | system temp dir | Local
path for RocksDB storage during RLI bootstrap. Each task manager creates a
unique subdirectory under this path |
+| `index.rli.cache.size` | `false` | `256` | Maximum memory
in MB for the RLI cache per bucket-assign task. Dynamically adjusted based on
historical usage |
+| `index.rli.lookup.minibatch.size` | `false` | `1000` | Maximum records
buffered per mini-batch during RLI lookup. Mini-batching reduces individual
index lookups. Minimum effective value is 1000 |
+| `metadata.compaction.async.enabled` | `false` | `true` | Whether to run
MDT compaction asynchronously within the Flink pipeline. Recommended to keep
enabled for RLI workloads |
+| `metadata.compaction.delta_commits` | `false` | `10` | Number of MDT
delta commits that trigger in-pipeline compaction
|
+
+:::note
+`GLOBAL_RECORD_LEVEL_INDEX` requires `metadata.enabled=true` and
`index.global.enabled=true`. The Flink table factory validates these
constraints automatically.
+:::
+
+## Lookup Join
+
+Hudi 1.2.0 adds a RocksDB-backed cache option for Flink lookup joins against
Hudi dimension tables. This avoids JVM heap pressure when the dimension table
is large.
+
### Options
-| Option Name | Required | Default | Remarks
|
-|---------------------------------|----------|---------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `write.rate.limit` | `false` | `0` | Write
record rate limit per second to prevent traffic jitter and improve stability.
Default is 0 (no limit)
|
-| `read.splits.limit` | `false` | `Integer.MAX_VALUE` | Maximum
number of splits allowed to read in each instant check for streaming reads.
Average read rate = `read.splits.limit`/`read.streaming.check-interval`.
Default is no limit |
-| `read.streaming.check-interval` | `false` | `60` | Check
interval in seconds for streaming reads. Default is 60 seconds (1 minute)
|
+| Option Name | Required | Default
| Remarks
|
+|--------------------------------|----------|---------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------|
+| `lookup.join.cache.type` | `false` | `heap`
| Storage backend for the lookup join cache. `heap` (default) stores rows in
JVM heap; `rocksdb` stores rows off-heap in an embedded RocksDB instance |
+| `lookup.join.rocksdb.path` | `false` |
`${java.io.tmpdir}/hudi-lookup-rocksdb` | Local directory for RocksDB data when
`lookup.join.cache.type=rocksdb`. Cleaned up when the lookup function closes
|
+| `lookup.async` | `false` | `false`
| Whether to enable async lookup join. Async join can improve throughput when
the lookup function has high latency |
+| `lookup.async-thread-number` | `false` | `16`
| Number of threads for async lookup join
|
+
+### Example
+
+```sql
+-- Streaming fact table with a processing-time attribute
+CREATE TABLE orders (
+ order_id BIGINT,
+ customer_id BIGINT,
+ amount DOUBLE,
+ proc_time AS PROCTIME(),
+ PRIMARY KEY (order_id) NOT ENFORCED
+) WITH (
+ 'connector' = 'hudi',
+ 'path' = 'hdfs:///warehouse/orders',
+ 'table.type' = 'MERGE_ON_READ',
+ 'read.streaming.enabled' = 'true'
+);
+
+-- Hudi dimension table with RocksDB-backed lookup cache
+CREATE TABLE customers (
+ customer_id BIGINT,
+ name STRING,
+ city STRING,
+ PRIMARY KEY (customer_id) NOT ENFORCED
+) WITH (
+ 'connector' = 'hudi',
+ 'path' = 'hdfs:///warehouse/customers',
+ 'lookup.join.cache.type' = 'rocksdb',
+ 'lookup.join.rocksdb.path' = '/tmp/hudi-lookup-rocksdb'
+);
+
+-- Lookup join keyed by the fact table's processing-time attribute
+SELECT o.order_id, c.name, o.amount
+FROM orders AS o
+JOIN customers FOR SYSTEM_TIME AS OF o.proc_time AS c
+ ON o.customer_id = c.customer_id;
+```
+
+## Virtual Metadata Columns
+
+Hudi metadata fields can be declared as `METADATA VIRTUAL` columns in the
Flink DDL. This allows accessing system metadata (e.g., commit time, record
key) without storing them as regular data columns.
+
+```sql
+CREATE TABLE events (
+ event_id BIGINT,
+ payload STRING,
+ -- virtual metadata columns (read-only, not persisted as data)
+ _hoodie_commit_time STRING METADATA VIRTUAL,
+ _hoodie_record_key STRING METADATA VIRTUAL,
+ _hoodie_partition_path STRING METADATA VIRTUAL,
+ PRIMARY KEY (event_id) NOT ENFORCED
+)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'hdfs:///warehouse/events'
+);
+
+-- Query metadata alongside data
+SELECT event_id, _hoodie_commit_time, payload FROM events;
+```
+
+:::note
+Only `VIRTUAL` metadata columns are supported. All valid virtual columns
correspond to Hudi's built-in meta fields (`_hoodie_commit_time`,
`_hoodie_commit_seqno`, `_hoodie_record_key`, `_hoodie_partition_path`,
`_hoodie_file_name`, `_hoodie_operation`).
+:::
+
+## Advanced Options
+
+### Hadoop Configuration Pass-through
+
+Hadoop filesystem configuration properties can be passed to the Flink writer
using the `properties.hadoop.*` prefix (or directly as `hadoop.*`):
+
+```sql
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 's3a://my-bucket/my-table',
+ 'properties.hadoop.fs.s3a.access.key' = 'AKID...',
+ 'properties.hadoop.fs.s3a.secret.key' = '...'
+)
+```
+
+### Kafka Offset Tracing
+
+For advanced Kafka offset tracing (internal/optional), the following
`kafka.offset.trace.*` options configure the checkpoint-service-based offset
lookup used in some deployment environments. These are advanced options with no
functional impact on standard Hudi writes:
+
+| Option Name | Default | Remarks
|
+|------------------------------------------|-------------------|------------------------------------------------------------|
+| `kafka.offset.trace.caller.service.name` | `ingestion-rt` | Caller
service name for checkpoint-service RPC headers |
+| `kafka.offset.trace.checkpoint.service` | `athena-job-manager` | Checkpoint
service name |
+| `kafka.offset.trace.dc` | `(none)` | Data center
for checkpoint offset lookup |
+| `kafka.offset.trace.env` | `(none)` | Environment
for checkpoint offset lookup |
+| `kafka.offset.trace.job.name` | `(none)` | Flink job
name for checkpoint offset lookup |
diff --git a/website/docs/key_generation.md b/website/docs/key_generation.md
index 3a7b109c3363..5f00956cefdb 100644
--- a/website/docs/key_generation.md
+++ b/website/docs/key_generation.md
@@ -2,7 +2,7 @@
title: Key Generation
summary: "In this page, we describe key generation in Hudi."
toc: true
-last_modified_at:
+last_modified_at: 2026-05-27T00:00:00-00:00
---
Hudi needs some way to point to records in the table, so that base/log files
can be merged efficiently for updates/deletes,
@@ -210,6 +210,35 @@ Partition path generated from key generator: "2020040118"
Input field value: "20200401" <br/>
Partition path generated from key generator: "04/01/2020"
+## Slash-Separated Date Partitioning
+
+By default, Hudi writes date-valued partition paths as a flat string (e.g.
`2024-03-15`).
+When `hoodie.datasource.write.slash.separated.date.partitioning` is set to
`true`, partition field
+values in `yyyy-MM-dd` format are stored as `yyyy/MM/dd` directory hierarchies
(e.g. `2024/03/15`).
+
+| Config Name |
Default | Description
|
+|----------------------------------------------------------------------|-----------|------------------------------------------------------------------------------------------------------------------------------------------|
+| `hoodie.datasource.write.slash.separated.date.partitioning` |
`false` | When `true`, transforms date partition values from `yyyy-MM-dd`
into `yyyy/MM/dd` directory paths. Cannot be used together with hive-style
partitioning (`hoodie.datasource.write.hive_style_partitioning=true`). |
+
+Example:
+
+```java
+df.write.format("hudi")
+ .option("hoodie.datasource.write.partitionpath.field", "event_date")
+ .option("hoodie.datasource.write.slash.separated.date.partitioning", "true")
+ .option("hoodie.table.name", tableName)
+ .mode("append")
+ .save(basePath)
+```
+
+A record with `event_date = "2024-03-15"` will be stored under
`basePath/2024/03/15/` instead of
+`basePath/2024-03-15/`.
+
+:::note
+`SHOW PARTITIONS` in Spark SQL correctly handles slash-separated date
partition paths: it displays
+the value in `yyyy-MM-dd` form (normalizing the `/` separators back to `-`)
for readability.
+:::
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/docs/lance_file_format.md
b/website/docs/lance_file_format.md
index 51269107f188..9754a4e31797 100644
--- a/website/docs/lance_file_format.md
+++ b/website/docs/lance_file_format.md
@@ -1,18 +1,28 @@
---
title: "Lance File Format"
keywords: [ hudi, lance, file format, vector, AI, ML, columnar, ANN, indexing]
-summary: "Use the Lance columnar file format with Hudi for vector-optimized
storage, ANN indexing, and efficient ML workloads"
+summary: "Use the Lance columnar file format with Hudi for vector-friendly
storage and efficient ML workloads"
toc: true
-last_modified_at: 2026-04-25T00:00:00-00:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
[Lance](https://lancedb.github.io/lance/) is a modern columnar data format
designed for AI and machine learning
workloads. Hudi's pluggable storage architecture lets you use Lance as the
base file format alongside Parquet
and ORC, unlocking vector indexing, fast random access, and optimized
high-dimensional array storage.
+:::caution Engine Support
+Lance file format support is **Spark-only**. Attempting to read a Lance-backed
table from Flink or Hive throws a
+`HoodieValidationException`:
+> Lance base file format is currently only supported with the Spark engine.
Please use Parquet, ORC, or HFile
+> for non-Spark engines (Flink, Hive, Presto, Trino).
+
+The Lance JAR is **not bundled** in the Hudi distribution — you must add it to
your Spark classpath
+(see [Required Dependencies](#required-dependencies)).
+:::
+
## Enabling Lance in Hudi
-### Table Creation
+### Table Creation (COW)
Set the base file format to `lance` in table properties:
@@ -26,7 +36,26 @@ TBLPROPERTIES (
primaryKey = 'id',
type = 'cow',
hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger',
- hoodie.datasource.write.base.file.format = 'lance'
+ hoodie.table.base.file.format = 'lance'
+);
+```
+
+### Table Creation (MOR)
+
+Lance base files work with MOR tables — Lance files act as base files while
Avro log files capture
+incremental changes. Log compaction merges the delta log back into Lance base
files.
+
+```sql
+CREATE TABLE my_ai_table_mor (
+ id STRING,
+ embedding VECTOR(768),
+ metadata STRING
+) USING hudi
+TBLPROPERTIES (
+ primaryKey = 'id',
+ type = 'mor',
+ hoodie.record.merger.impls = 'org.apache.hudi.DefaultSparkRecordMerger',
+ hoodie.table.base.file.format = 'lance'
);
```
@@ -39,18 +68,21 @@ TBLPROPERTIES (
.option("hoodie.datasource.write.recordkey.field", "id")
.option("hoodie.record.merger.impls",
"org.apache.hudi.DefaultSparkRecordMerger")
- .option("hoodie.datasource.write.base.file.format", "lance")
+ .option("hoodie.table.base.file.format", "lance")
.mode("overwrite")
.save("/path/to/my_ai_table"))
```
### Required Dependencies
-Add the Lance Spark bundle to your Spark classpath:
+The Lance JAR is not bundled in Hudi. Add the appropriate Lance Spark bundle
JAR to your Spark classpath:
-| Component | Maven Coordinates |
-|:----------|:-----------------|
-| Lance Spark Bundle (Spark 3.5) |
`org.lance:lance-spark-bundle-3.5_2.12:0.4.0` |
+| Spark Version | Bundle JAR (Maven Central) |
+|:--------------|:---------------------------|
+| Spark 3.4 | `org.lance:lance-spark-bundle-3.4_2.12:0.4.0` |
+| Spark 3.5 | `org.lance:lance-spark-bundle-3.5_2.12:0.4.0` |
+| Spark 4.0 | `org.lance:lance-spark-bundle-4.0_2.13:0.4.0` |
+| Spark 4.1 | `org.lance:lance-spark-bundle-4.1_2.13:0.4.0` |
```bash
export LANCE_BUNDLE_JAR=/path/to/lance-spark-bundle-3.5_2.12-0.4.0.jar
@@ -74,7 +106,6 @@ file-level storage:
│ (same Hudi concepts as Parquet) │
├───────────────────────────────────┤
│ Lance Data Files (.lance) │
-│ IVF-PQ vector index │
│ Columnar storage │
│ Fragment-based layout │
├───────────────────────────────────┤
@@ -87,11 +118,49 @@ All Hudi table services work with Lance-backed tables:
- **Compaction** — merges log files into Lance base files
- **Clustering** — reorganizes Lance files for better data locality
- **Cleaning** — removes old Lance file versions
-- **Metadata indexing** — column stats and bloom filters work across Lance
files
+- **Metadata indexing** — bloom filters work across Lance files; column stats
and partition stats are
+ **automatically disabled** for Lance tables
+
+## VECTOR Storage on Lance
+
+VECTOR columns are stored natively in Lance as `FixedSizeList<Float32/Float64,
dim>` — Lance's own
+vector column encoding, so embeddings are written without conversion overhead
at the file-format
+layer.
+
+Only **FLOAT** and **DOUBLE** element types are supported as VECTOR columns on
Lance. INT8 vectors
+are not yet supported and will fail fast at write time.
+
+See [Vector Search](vector_search.md) for the `hudi_vector_search` TVF that
queries VECTOR columns.
+
+## BLOB Columns on Lance
+
+INLINE BLOB columns on Lance default to `DESCRIPTOR` read mode — standard
queries return an
+out-of-line-shaped reference descriptor rather than materializing the raw
bytes. To read inline
+byte content via `read_blob()`, set `hoodie.read.blob.inline.mode=CONTENT`. See
+[Unstructured Data](blob_unstructured_data.md) for full documentation.
+
+## Schema Evolution
+
+Lance supports the following schema changes at the Hudi layer:
+
+| Operation | Supported? |
+|:----------|:-----------|
+| Add column | Yes |
+| Rename column | Yes (via Hudi schema evolution) |
+| Promote `FLOAT` → `DOUBLE` | **No** — not supported on Lance |
+| Promote `FLOAT` → `STRING` | **No** — not supported on Lance |
+| Drop column | Yes |
+
+:::caution
+`FLOAT → DOUBLE` and `FLOAT → STRING` type promotions are supported for
Parquet tables but **not**
+for Lance. Attempting these on a Lance table will fail. Use `DOUBLE` from the
start if you anticipate
+needing higher precision.
+:::
## Vector Search with Lance
-The `hudi_vector_search` TVF leverages Lance's built-in IVF-PQ index for
approximate nearest neighbor search:
+Use the `hudi_vector_search` TVF to run vector similarity queries against
VECTOR columns on a
+Lance-backed table:
```sql
SELECT id, metadata, _hudi_distance
@@ -107,10 +176,38 @@ See [Vector Search](vector_search.md) for full
documentation on the TVF and dist
## Configuration Reference
-| Property | Description | Default |
-|:---------|:------------|:--------|
-| `hoodie.datasource.write.base.file.format` | Set to `lance` to use Lance as
the base file format | `parquet` |
-| `hoodie.record.merger.impls` | Must be
`org.apache.hudi.DefaultSparkRecordMerger` for Lance | — |
+| Property | Default | Description |
+|:---------|:--------|:------------|
+| `hoodie.table.base.file.format` | `parquet` | Set to `lance` to use Lance as
the base file format. |
+| `hoodie.record.merger.impls` | — | Must be
`org.apache.hudi.DefaultSparkRecordMerger` for Lance. |
+| `hoodie.lance.max.file.size` | `125829120` (120 MiB) | Target file size in
bytes for Lance base files. |
+| `hoodie.lance.write.allocator.size.bytes` | `268435456` (256 MiB) | Maximum
size of the Arrow child allocator used for buffering in-flight batch data.
Increase for tables with very large BLOB columns. |
+| `hoodie.lance.write.flush.byte.watermark` | `100663296` (96 MiB) | Byte-size
threshold at which the current write batch is flushed. Must be less than
`hoodie.lance.write.allocator.size.bytes`. |
+
+### File Sizing and Memory
+
+The three sizing configs work together:
+
+- **`hoodie.lance.max.file.size`** controls when Hudi rolls over to a new
Lance file, similar to
+ `hoodie.parquet.max.file.size` for Parquet tables.
+- **`hoodie.lance.write.allocator.size.bytes`** caps the Arrow allocator's
in-flight memory. Arrow
+ uses power-of-2 buffer doubling; the default 256 MiB accommodates the 128
MiB doubling step with
+ headroom.
+- **`hoodie.lance.write.flush.byte.watermark`** triggers an early batch flush
when Arrow buffers
+ approach the cap. The default 96 MiB (≈ 3/8 of the allocator cap) leaves
room for offset and
+ validity buffers to double without exceeding the allocator limit.
+
+For tables with large BLOB columns, increase both
`hoodie.lance.write.allocator.size.bytes` and
+`hoodie.lance.write.flush.byte.watermark` proportionally (keep watermark at
roughly 3/8 of allocator
+size).
+
+## Additional Notes
+
+- **`populateMetaFields=false`** is supported. User-defined key generators
work normally with Lance
+ tables.
+- **Complex types** (struct, array, map) are supported as Lance columns.
+- **VARIANT columns** are **not supported** on Lance. Attempting to write a
table with VARIANT columns
+ to Lance throws a `HoodieNotSupportedException`. Use Parquet for tables with
VARIANT columns.
## Mixed-Format Tables
diff --git a/website/docs/metadata.md b/website/docs/metadata.md
index c5a01b42dfa9..9c108c815cdd 100644
--- a/website/docs/metadata.md
+++ b/website/docs/metadata.md
@@ -89,6 +89,15 @@ If you turn off the metadata table after enabling, be sure
to wait for a few com
cleaned up, before re-enabling the metadata table again.
:::
+### Auto-Delete of Disabled MDT Partitions
+
+When an index is disabled in the write config, Hudi automatically deletes the
corresponding metadata table partition.
+Available since Hudi 1.2.0, this behavior is configurable.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.metadata.auto.delete.partitions` | `true` | When enabled (default),
metadata table partitions (indexes) that are disabled in the write config are
automatically deleted. Set to `false` to prevent accidental deletion in
multi-writer environments where not all writers may have the same config —
users must then drop indexes explicitly via Hudi CLI or `DROP INDEX`. |
+
## Leveraging metadata during queries
### files index
@@ -129,6 +138,28 @@ can bring up the writers sequentially after stopping the
writers for enabling me
configurations to only a subset of writers or table services is unsafe and can
lead to loss of data. So, please ensure you enable
metadata table across all writers.
+## MDT Cleaner and Compaction
+
+Hudi 1.2.0 introduced a config that lets the metadata table's cleaner derive
its retention policy directly from the
+data table, rather than requiring a separate configuration.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.metadata.derive.from.datatable.clean.policy` | `true` | When
enabled, the metadata table's cleaner uses the same cleaning policy (retention
count, hours, etc.) as the data table. See also
[cleaning](cleaning.md#mdt-cleaner-inherits-data-table-policy). |
+
+The metadata table's compaction and log compaction can also be delegated to an
external table service platform. See
+[compaction](compaction.md#delegating-mdt-compaction-to-an-external-platform)
for the full config reference.
+
+## Timeline Archival Controls
+
+Hudi 1.2.0 added two configs in `HoodieArchivalConfig` to fine-tune how the
timeline manifest and archival interact
+with the most recent clean.
+
+| Config Name | Default | Description |
+|---|---|---|
+| `hoodie.timeline.manifest.retained.versions` | `3` | Number of timeline
manifest file versions to retain. Older manifest versions are pruned during
archival. |
+| `hoodie.archive.block.on.latest.clean.ectr` | `false` | When enabled,
archival stops at the Earliest Commit To Retain (ECTR) from the last completed
clean. This prevents archiving commits whose data files still exist on storage,
avoiding inconsistencies between the timeline and actual data. |
+
## Related Resources
<h3>Blogs</h3>
* [Table service deployment models in Apache
Hudi](https://medium.com/@simpsons/table-service-deployment-models-in-apache-hudi-9cfa5a44addf)
diff --git a/website/docs/metadata_indexing.md
b/website/docs/metadata_indexing.md
index 7056a1e02671..dbdd523df60d 100644
--- a/website/docs/metadata_indexing.md
+++ b/website/docs/metadata_indexing.md
@@ -2,7 +2,7 @@
title: Indexing
summary: "In this page, we describe how to run metadata indexing
asynchronously."
toc: true
-last_modified_at:
+last_modified_at: 2026-05-27T00:00:00-00:00
---
Hudi maintains a scalable [metadata](metadata.md) that has some auxiliary data
about the table.
@@ -36,7 +36,7 @@ For more information on these indexes please refer [metadata
section](metadata/#
:::note
Please note in order to create secondary index:
1. The table must have a primary key and merge mode should be
[COMMIT_TIME_ORDERING](record_merger.md#commit_time_ordering).
-2. Record index must be enabled. This can be done by setting
`hoodie.metadata.record.index.enable=true` and then creating `record_index`.
Please note the example below.
+2. Record index must be enabled. This can be done by setting
`hoodie.metadata.global.record.level.index.enable=true` and then creating
`record_index`. Please note the example below.
:::
**Examples**
@@ -73,8 +73,8 @@ hoodie.metadata.index.column.stats.enable=true
-- [Optional Configs] - list of columns to index on. By default all columns
are indexed
hoodie.metadata.index.column.stats.column.list=col1,col2,...
--- [Required Configs] Record Level Index
-hoodie.metadata.record.index.enable=true
+-- [Required Configs] Record Level Index (Global RLI — single record key
unique across all partitions)
+hoodie.metadata.global.record.level.index.enable=true
-- [Required Configs] Bloom filter Index
hoodie.metadata.index.bloom.filter.enable=true
@@ -116,7 +116,7 @@ inserts.write.format("hudi").
// Create record index and secondary index for the table
spark.sql(s"CREATE TABLE test_table_external USING hudi LOCATION '$basePath'")
-spark.sql(s"SET hoodie.metadata.record.index.enable=true")
+spark.sql(s"SET hoodie.metadata.global.record.level.index.enable=true")
spark.sql(s"CREATE INDEX record_index ON test_table_external (uuid)")
spark.sql(s"CREATE INDEX idx_rider ON test_table_external (rider)")
spark.sql(s"SHOW INDEXES FROM hudi_indexed_table").show(false)
@@ -191,6 +191,24 @@ Enabling the metadata table and configuring a lock
provider are the prerequisite
configuration below.
:::
+#### Record-Level Index Configuration Keys
+
+Hudi supports two flavors of the Record Level Index, each with its own enable
flag and sizing configs:
+
+- **Global RLI** — record key is unique across the entire table (across
partitions).
+- **Partitioned RLI** — `partition_path + record_key` is unique within each
partition.
+
+| Config Name | Default | Notes |
+|---|---|---|
+| `hoodie.metadata.global.record.level.index.enable` | `false` | Enables the
global RLI. |
+| `hoodie.metadata.global.record.level.index.min.filegroup.count` | `10` | Min
file groups for the global RLI. |
+| `hoodie.metadata.global.record.level.index.max.filegroup.count` | `10000` |
Max file groups for the global RLI. |
+| `hoodie.metadata.record.level.index.enable` | `false` | Enables the
partitioned RLI. Independent toggle from the global RLI above. |
+| `hoodie.metadata.record.level.index.min.filegroup.count` | `1` | Min file
groups for the partitioned RLI. |
+| `hoodie.metadata.record.level.index.max.filegroup.count` | `10` | Max file
groups for the partitioned RLI. |
+| `hoodie.metadata.record.level.index.defer.init` | `false` | When enabled,
defers RLI initialization to the second commit on a fresh table so Hudi can
size file groups based on actual record volume. Applies to both global and
partitioned RLI. |
+| `hoodie.metadata.record.index.max.filegroup.size` | `1073741824` (1 GB) |
Maximum size in bytes of a single RLI file group. Larger file groups take
longer to compact. |
+
```
# ensure that async indexing is enabled
hoodie.metadata.index.async=true
@@ -284,7 +302,10 @@ indexer logs, we would find that it indeed caught up with
instant `2022041419542
### Drop Index
-To drop an index, just run the index in `dropindex` mode.
+To drop an index, just run the index in `dropindex` mode. Note that as of Hudi
1.2.0, when an index is disabled in
+the write config, Hudi automatically drops its metadata table partition by
default; see
+[`hoodie.metadata.auto.delete.partitions`](metadata.md#auto-delete-of-disabled-mdt-partitions)
to control this
+behavior.
```
spark-submit \
diff --git a/website/docs/metrics.md b/website/docs/metrics.md
index ec94d21aa45a..151d27f5d319 100644
--- a/website/docs/metrics.md
+++ b/website/docs/metrics.md
@@ -3,7 +3,7 @@ title: Metrics
keywords: [ hudi, administration, operation, devops, metrics]
summary: This section offers an overview of metrics in Hudi
toc: true
-last_modified_at: 2020-06-20T15:59:57-04:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
In this section, we will introduce the `MetricsReporter` and `HoodieMetrics`
in Hudi. You can view the metrics-related configurations
[here](configurations.md#METRICS).
@@ -204,29 +204,46 @@ These `HoodieMetrics` can then be plotted on a standard
tool like grafana. Below
## List of metrics:
-The below metrics are available in all timeline operations that involves a
commit such as deltacommit, compaction, clustering and rollback.
+The metrics below are emitted across timeline operations (deltacommit,
compaction, clustering, rollback, clean, archival) and post-commit callbacks.
When `hoodie.metrics.reporter.metricsname.prefix` is set, every name is
prefixed with `<prefix>.<name>`.
-Name | Description
+Name | Description
--- | ---
-commitFreshnessInMs | Milliseconds from the commit end time and the maximum
event time of the incoming records
-commitLatencyInMs | Milliseconds from the commit end time and the minimum
event time of incoming records
-commitTime | Time of commit in epoch milliseconds
-duration | Total time taken for the commit/rollback in milliseconds
-numFilesDeleted | Number of files deleted during a clean/rollback
-numFilesFinalized | Number of files finalized in a write
-totalBytesWritten | Bytes written in a HoodieCommit
-totalCompactedRecordsUpdated | Number of records updated in a compaction
operation
-totalCreateTime | Time taken for file creation during a Hoodie Insert operation
-totalFilesInsert | Number of newly written files in a HoodieCommit
-totalFilesUpdate | Number of files updated in a HoodieCommit
-totalInsertRecordsWritten | Number of records inserted or converted to
updates(for small file handling) in a HoodieCommit
-totalLogFilesCompacted | Number of log files under a base file in a file
group compacted
-totalLogFilesSize | Total size in bytes of all log files under a base file in
a file group
-totalPartitionsWritten | Number of partitions that took writes in a
HoodieCommit
-totalRecordsWritten | Number of records written in a HoodieCommit. For
inserts, it is the total numbers of records inserted. And for updates, it the
total number of records in the file.
-totalScanTime | Time taken for reading and merging logblocks in a log file
-totalUpdateRecordsWritten | Number of records that got changed in a
HoodieCommit
-totalUpsertTime | Time taken for Hoodie Merge
-
-These metrics can be found at org.apache.hudi.metrics.HoodieMetrics and
referenced from
-org.apache.hudi.common.model.HoodieCommitMetadata and
org.apache.hudi.common.model.HoodieWriteStat
+commitFreshnessInMs | Milliseconds from the commit end time and the maximum
event time of the incoming records.
+commitLatencyInMs | Milliseconds from the commit end time and the minimum
event time of incoming records.
+commitTime | Time of commit in epoch milliseconds.
+duration | Total time taken for the commit/rollback in milliseconds.
+numFilesDeleted | Number of files deleted during a clean/rollback.
+numFilesFinalized | Number of files finalized in a write.
+totalBytesWritten | Bytes written in a HoodieCommit.
+totalCompactedRecordsUpdated | Number of records updated in a compaction
operation.
+totalCreateTime | Time taken for file creation during a Hoodie Insert
operation.
+totalFilesInsert | Number of newly written files in a HoodieCommit.
+totalFilesUpdate | Number of files updated in a HoodieCommit.
+totalInsertRecordsWritten | Number of records inserted or converted to updates
(for small file handling) in a HoodieCommit.
+totalLogFilesCompacted | Number of log files under a base file in a file group
compacted.
+totalLogFilesSize | Total size in bytes of all log files under a base file in
a file group.
+totalPartitionsWritten | Number of partitions that took writes in a
HoodieCommit.
+totalRecordsWritten | Number of records written in a HoodieCommit. For
inserts, the total records inserted; for updates, the total records in the file.
+totalScanTime | Time taken for reading and merging log blocks in a log file.
+totalUpdateRecordsWritten | Number of records that got changed in a
HoodieCommit.
+totalUpsertTime | Time taken for Hoodie Merge.
+clean.duration | Wall-clock time in milliseconds for a clean operation.
+archive.duration | Wall-clock time in milliseconds for an archive operation.
+rollback.failure.counter | Incremented each time a rollback operation fails.
+postCommit.success.counter | Incremented each time all post-commit callbacks
succeed.
+postCommit.failure.counter | Incremented each time a post-commit callback
fails (post-commit failures are non-fatal).
+postCommit.duration | Wall-clock time in milliseconds for post-commit callback
execution.
+archival.archivalNumAllCommits | Total number of instants archived in this
archival run.
+archival.archivalNumWriteCommits | Number of write instants (commit,
deltacommit, replacecommit) archived.
+archival.archivalNumCleanCommits | Number of clean instants archived.
+archival.archivalNumRollbackCommits | Number of rollback instants archived.
+archival.archivalStatus | `1` if archival succeeded, `-1` if it failed.
+archival.archivalFailure.\<ExceptionClassName\> | Incremented on archival
failure; the suffix is the simple class name of the exception thrown.
+archival.archivalOutOfMemory | Incremented when archival fails due to an
`OutOfMemoryError`.
+\<action\>.totalCorruptedLogBlocks | Number of corrupted log blocks
encountered during compaction. Reported only when
`hoodie.metricscompaction.log.blocks.on=true`. `<action>` is the commit action
type (e.g., `commit`).
+\<action\>.totalRollbackLogBlocks | Number of rollback log blocks encountered
during compaction. Reported only when
`hoodie.metricscompaction.log.blocks.on=true`.
+\<action\>.totalLogBlocksCompacted | Total number of log blocks compacted.
Reported only when `hoodie.metricscompaction.log.blocks.on=true`.
+
+These metrics live in `org.apache.hudi.metrics.HoodieMetrics` (with
archival-specific names sourced from
`org.apache.hudi.client.utils.ArchivalMetrics`) and are referenced from
`org.apache.hudi.common.model.HoodieCommitMetadata` and
`org.apache.hudi.common.model.HoodieWriteStat`.
+
+In multi-tenant deployments where a single Spark job writes to multiple Hudi
tables, each table gets its own isolated `MetricRegistry`, scoped as
`<tableName>.<registryName>` so metrics from different tables do not collide.
No configuration is required.
diff --git a/website/docs/overview.mdx b/website/docs/overview.mdx
index 46c2d5b4be1c..ed439cef38a1 100644
--- a/website/docs/overview.mdx
+++ b/website/docs/overview.mdx
@@ -59,10 +59,10 @@ If you want to experience Apache Hudi integrated into an
end to end demo with Ka
Hudi brings first-class support for AI and unstructured data workloads to the
data lakehouse:
-- **[VECTOR type & Similarity Search](vector_search.md)** — Store embeddings
and run approximate nearest neighbor search directly in Spark SQL
+- **[VECTOR type & Similarity Search](vector_search.md)** — Store embeddings
and run vector similarity search directly in Spark SQL
- **[BLOB type for Unstructured Data](blob_unstructured_data.md)** — Store
images, PDFs, audio, and other binary data with inline or out-of-line storage
- **[VARIANT type for Semi-Structured Data](variant_type.md)** — Store
flexible JSON-like data (LLM outputs, model metadata, feature maps) without
rigid schemas
-- **[Lance File Format](lance_file_format.md)** — Vector-optimized columnar
format with built-in ANN indexing
+- **[Lance File Format](lance_file_format.md)** — Vector-friendly columnar
format for AI/ML workloads
See the full [AI-Native Lakehouse Overview](ai_overview.md) for use cases and
architecture.
diff --git a/website/docs/precommit_validator.md
b/website/docs/precommit_validator.md
index fe0a0dd77605..d67d9ec3a054 100644
--- a/website/docs/precommit_validator.md
+++ b/website/docs/precommit_validator.md
@@ -109,6 +109,73 @@ Hudi offers a [commit notification
service](platform_services_post_commit_callba
The commit notification service can be combined with pre-commit validators to
send a notification when a commit fails a validation. This is possible by
passing details about the validation as a custom value to the HTTP endpoint.
+## Notes on Validator Behavior
+
+Hudi 1.2.0 introduced the following behavioral refinements:
+
+**Metadata fields in SQL queries**: Validator SQL can now reference Hudi
metadata fields (`_hoodie_record_key`, `_hoodie_partition_path`,
`_hoodie_file_name`, `_hoodie_commit_time`, `_hoodie_commit_seqno`) directly in
query expressions.
+
+**Empty writes**: Empty write commits no longer cause pre-commit validators to
error. Validators are skipped gracefully when no records are present in the
write.
+
+## Failure Policy
+
+Hudi 1.2.0 introduces a configurable failure policy for pre-commit validators:
+
+| Config Key | Default | Description |
+|---|---|---|
+| `hoodie.precommit.validators.failure.policy` | `FAIL` | How to handle
validator failures. `FAIL`: block the commit with an exception. `WARN_LOG`:
emit a warning log but allow the commit to proceed (useful for soft
monitoring). |
+
+## Flink and Streaming-Offset Validators
+
+Available since Hudi 1.2.0. Flink writers now honor
`hoodie.precommit.validators` using the same configuration key as Spark.
Validators intended for use with Flink must extend the engine-agnostic
`org.apache.hudi.client.validator.BasePreCommitValidator` (in `hudi-common`),
which provides access to commit metadata and timeline information independently
of Spark.
+
+Two built-in streaming-offset validators are now available for Kafka-sourced
pipelines:
+
+| Validator Class | Engine | Description |
+|---|---|---|
+| `org.apache.hudi.sink.validator.FlinkKafkaOffsetValidator` | Flink |
Validates that the number of records written matches the Kafka offset
difference for the batch |
+| `org.apache.hudi.utilities.streamer.validator.SparkKafkaOffsetValidator` |
Spark / HoodieStreamer | Same semantics for Spark-based Kafka ingestion
pipelines |
+
+Both validators use the following configuration:
+
+| Config Key | Default | Description |
+|---|---|---|
+| `hoodie.precommit.validators.streaming.offset.tolerance.percentage` | `0.0`
| Tolerance percentage for offset-based record-count validation. A value of
`0.0` requires an exact match between expected records (from Kafka offset
delta) and actual records written. For upsert workloads with deduplication, set
a higher tolerance (e.g., `10.0` for 10%). |
+| `hoodie.precommit.validators.failure.policy` | `FAIL` | See [Failure
Policy](#failure-policy) above. |
+
+Example (Flink):
+```properties
+hoodie.precommit.validators=org.apache.hudi.sink.validator.FlinkKafkaOffsetValidator
+hoodie.precommit.validators.streaming.offset.tolerance.percentage=5.0
+hoodie.precommit.validators.failure.policy=WARN_LOG
+```
+
+## Pre-Write Validators
+
+Introduced in Hudi 1.2.0, pre-write validators run **before** data is written
to storage, in contrast to pre-commit validators which run **after** data is
written but before the commit is published to the timeline. This enables
earlier rejection of invalid operations, avoiding unnecessary I/O.
+
+Configuration:
+
+| Config Key | Default | Description |
+|---|---|---|
+| `hoodie.prewrite.validators` | `""` | Comma-separated list of
fully-qualified class names implementing
`org.apache.hudi.client.validator.PreWriteValidator`. |
+
+To implement a custom pre-write validator, implement the
`org.apache.hudi.client.validator.PreWriteValidator` interface:
+
+```java
+public interface PreWriteValidator {
+ <T> void validate(
+ String instantTime,
+ WriteOperationType writeOperationType,
+ HoodieTableMetaClient metaClient,
+ HoodieWriteConfig writeConfig,
+ HoodieEngineContext engineContext,
+ Option<HoodieData<HoodieRecord<T>>> recordsOpt) throws
HoodieValidationException;
+}
+```
+
+No built-in pre-write validator implementations are provided yet; this
framework is designed for custom user extensions. Unlike pre-commit validators,
pre-write validators have access to the incoming records before any write I/O
occurs.
+
## Related Resources
<h3>Blogs</h3>
diff --git a/website/docs/procedures.md b/website/docs/procedures.md
index 71edf434ed10..94dd41991598 100644
--- a/website/docs/procedures.md
+++ b/website/docs/procedures.md
@@ -2,7 +2,7 @@
title: SQL Procedures
summary: "In this page, we introduce how to use SQL procedures with Hudi."
toc: true
-last_modified_at: 2025-11-24T00:00:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -298,6 +298,64 @@ call show_archived_commits(table => 'test_hudi_table');
| 20220216171027021 | 435346 | 1 | 0
| 1 | 1 | 0
| 0 |
| 20220216171019361 | 435349 | 1 | 0
| 1 | 1 | 0
| 0 |
+### show_timeline
+
+Show timeline entries for a Hudi table. Returns instant-level information for
all timeline operations (commits, compactions, clustering, clean, rollback,
etc.) from the active and optionally archived timeline. Results are sorted by
timestamp descending.
+
+**Input**
+
+| Parameter Name | Type | Required | Default Value | Description
|
+|----------------|---------|----------|---------------|----------------------------------------------------------------------------------|
+| table | String | N* | None | Hudi table name
(mutually exclusive with `path`) |
+| path | String | N* | None | Base path of the Hudi
table (mutually exclusive with `table`) |
+| limit | Int | N | 20 | Max number of timeline
entries to return (ignored when both `startTime` and `endTime` are set) |
+| showArchived | Boolean | N | false | Whether to include
archived timeline entries |
+| filter | String | N | "" | SQL expression to
filter results on any output column |
+| startTime | String | N | "" | Start timestamp for
filtering (format: `yyyyMMddHHmmss`, inclusive) |
+| endTime | String | N | "" | End timestamp for
filtering (format: `yyyyMMddHHmmss`, inclusive) |
+
+\* Either `table` or `path` must be provided.
+
+**Output**
+
+| Output Name | Type | Description
|
+|----------------|--------|---------------------------------------------------------------------------------------|
+| instant_time | String | Requested timestamp of the instant
|
+| action | String | Action type: `commit`, `deltacommit`,
`compaction`, `clustering`, `clean`, `rollback`, etc. |
+| state | String | State of the instant: `REQUESTED`, `INFLIGHT`, or
`COMPLETED` |
+| requested_time | String | Wall-clock time when the instant was requested
(format: `MM-dd HH:mm:ss`) |
+| inflight_time | String | Wall-clock time when the instant became inflight
(format: `MM-dd HH:mm:ss`) |
+| completed_time | String | Wall-clock time when the instant completed
(format: `MM-dd HH:mm:ss`), or `null` |
+| timeline_type | String | `ACTIVE` or `ARCHIVED`
|
+| rollback_info | String | For rollback instants: what was rolled back; for
rolled-back instants: which rollback instant rolled them back; otherwise `null`
|
+
+**Example**
+
+```sql
+-- Show the 20 most recent timeline entries
+call show_timeline(table => 'test_hudi_table');
+
+-- Show up to 50 entries including archived timeline
+call show_timeline(table => 'test_hudi_table', limit => 50, showArchived =>
true);
+
+-- Filter to completed commits in a time range
+call show_timeline(
+ table => 'test_hudi_table',
+ startTime => '20251201000000',
+ endTime => '20251231235959',
+ filter => "action = 'commit' AND state = 'COMPLETED'"
+);
+
+-- Look up by base path instead of table name
+call show_timeline(path => 'hdfs:///user/hive/warehouse/test_hudi_table');
+```
+
+| instant_time | action | state | requested_time | inflight_time
| completed_time | timeline_type | rollback_info |
+|-------------------|--------|-----------|---------------------|---------------------|---------------------|---------------|---------------|
+| 20251205143022001 | commit | COMPLETED | 12-05 14:30:20 | 12-05
14:30:21 | 12-05 14:30:22 | ACTIVE | null |
+| 20251205141510003 | clean | COMPLETED | 12-05 14:15:09 | 12-05
14:15:10 | 12-05 14:15:10 | ACTIVE | null |
+| 20251205140030002 | commit | COMPLETED | 12-05 14:00:28 | 12-05
14:00:29 | 12-05 14:00:30 | ACTIVE | null |
+
### show_commit_files
Show files of a commit.
diff --git a/website/docs/reading_tables_batch_reads.md
b/website/docs/reading_tables_batch_reads.md
index 0a62f34d55e3..3b1e845e2145 100644
--- a/website/docs/reading_tables_batch_reads.md
+++ b/website/docs/reading_tables_batch_reads.md
@@ -19,6 +19,32 @@ val tripsDF = spark.read.
tripsDF.where(tripsDF.fare > 20.0).show()
```
+## Flink Batch (Snapshot) Read
+
+Flink can read a Hudi table as a snapshot (batch) query by leaving
`read.streaming.enabled` at its default value of `false`.
+
+```sql
+CREATE TABLE hudi_table (
+ uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${path}',
+ 'table.type' = 'MERGE_ON_READ'
+ -- read.streaming.enabled defaults to false → batch/snapshot read
+);
+
+-- Snapshot query
+SELECT * FROM hudi_table WHERE age > 25;
+```
+
+For more Flink read options, see [Using Flink](ingestion_flink.md).
+
## Daft
[Daft](https://www.daft.ai/) supports reading Hudi tables using
`daft.read_hudi()` function.
diff --git a/website/docs/reading_tables_streaming_reads.md
b/website/docs/reading_tables_streaming_reads.md
index 5055f42c0449..7191cedf0115 100644
--- a/website/docs/reading_tables_streaming_reads.md
+++ b/website/docs/reading_tables_streaming_reads.md
@@ -97,3 +97,47 @@ spark.readStream \
Spark SQL can be used within ForeachBatch sink to do INSERT, UPDATE, DELETE
and MERGE INTO.
Target table must exist before write.
:::
+
+## Flink Streaming Read
+
+Flink can continuously consume new commits from a Hudi table as a streaming
source. Enable this by setting `read.streaming.enabled=true` and optionally a
`read.start-commit`.
+
+```sql
+CREATE TABLE hudi_table (
+ uuid VARCHAR(20) PRIMARY KEY NOT ENFORCED,
+ name VARCHAR(10),
+ age INT,
+ ts TIMESTAMP(3),
+ `partition` VARCHAR(20)
+)
+PARTITIONED BY (`partition`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${path}',
+ 'table.type' = 'MERGE_ON_READ',
+ 'read.streaming.enabled' = 'true', -- enable streaming read
+ 'read.start-commit' = '20210316134557', -- start from this instant
(omit for latest)
+ 'read.streaming.check-interval' = '60' -- poll interval in seconds
+);
+
+SELECT * FROM hudi_table;
+```
+
+### Source V2 for Streaming
+
+As of Hudi 1.2.0, the [FLIP-27-based Source
V2](ingestion_flink.md#flink-source-v2) is available as an opt-in for streaming
reads. Source V2 participates in Flink's checkpoint protocol for finer-grained
recovery and supports partition pruning:
+
+```sql
+WITH (
+ 'connector' = 'hudi',
+ 'path' = '${path}',
+ 'read.streaming.enabled' = 'true',
+ 'read.source-v2.enabled' = 'true' -- enable FLIP-27 source (Hudi 1.2.0+)
+)
+```
+
+:::warning
+Savepoints taken with the legacy source are not compatible with Source V2.
Start a fresh job when switching. See [Flink Source
V2](ingestion_flink.md#flink-source-v2) for migration details.
+:::
+
+For a full list of Flink streaming read options (rate limiting, commits limit,
CDC mode, etc.), see [Using Flink](ingestion_flink.md).
diff --git a/website/docs/sql_ddl.md b/website/docs/sql_ddl.md
index d1c5ba865bdb..2fc0d421dab2 100644
--- a/website/docs/sql_ddl.md
+++ b/website/docs/sql_ddl.md
@@ -2,7 +2,7 @@
title: SQL DDL
summary: "In this page, we discuss using SQL DDL commands with Hudi"
toc: true
-last_modified_at:
+last_modified_at: 2026-05-27T00:00:00-00:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -696,6 +696,14 @@ SHOW PARTITIONS hudi_table;
ALTER TABLE hudi_table DROP PARTITION (dt='2021-12-09', hh='10');
```
+:::note Slash-separated date partitioning and SHOW PARTITIONS
+When a table is written with
`hoodie.datasource.write.slash.separated.date.partitioning=true`, the
+physical directory layout uses `yyyy/MM/dd` paths. `SHOW PARTITIONS` correctly
handles this: it
+returns partition values in the standard `col=yyyy-MM-dd` display format,
normalizing the `/`
+separators back to `-` for readability. See [Key
Generation](key_generation.md#slash-separated-date-partitioning)
+for details on configuring slash-separated partitioning.
+:::
+
### Show and drop index
**Syntax**
@@ -837,6 +845,39 @@ WITH (
);
```
+### Create Append-Only Table Without Primary Key
+
+Hudi 1.2.0 supports creating a Flink table **without a `PRIMARY KEY`** for
pure append workloads.
+In this mode, set `write.operation` to `insert`; Hudi will not enforce
record-level uniqueness and
+the record-key and ordering fields are optional.
+
+```sql
+-- Append-only table: no PRIMARY KEY required
+CREATE TABLE hudi_append_table (
+ id BIGINT,
+ name STRING,
+ ts BIGINT,
+ city STRING
+)
+PARTITIONED BY (`city`)
+WITH (
+ 'connector' = 'hudi',
+ 'path' = 'file:///tmp/hudi_append_table',
+ 'table.type' = 'COPY_ON_WRITE',
+ 'write.operation' = 'insert'
+);
+
+INSERT INTO hudi_append_table VALUES (1, 'Alice', 1695159649, 'sf'), (2,
'Bob', 1695091554, 'ny');
+```
+
+:::note
+Without a primary key, Hudi uses auto-generated record keys and does **not**
perform deduplication
+or upsert merging. This is equivalent to `bulk_insert` semantics and is well
suited for log/event
+ingestion pipelines where every incoming row should be appended as-is.
+If `write.operation` is any value other than `insert` and no `PRIMARY KEY` is
defined, Hudi will
+throw `"Primary key definition is missing"` at table creation time.
+:::
+
### Create Table in Non-Blocking Concurrency Control Mode
The following is an example of creating a Flink table in [Non-Blocking
Concurrency Control
mode](concurrency_control.md#non-blocking-concurrency-control).
@@ -967,3 +1008,15 @@ WITH (
| numeric | | not supported |
| null | | not supported |
| object | | not supported |
+
+### AI and Unstructured Data Types
+
+Hudi 1.2.0 introduces two additional column types for AI and unstructured data
workloads:
+
+- **`VECTOR(dim[, elementType])`** — stores fixed-dimension embedding vectors
(e.g. `VECTOR(768)`,
+ `VECTOR(768, FLOAT)`, `VECTOR(768, DOUBLE)`). Enables approximate
nearest-neighbor search via
+ the `hudi_vector_search` TVF. See [Vector Search](vector_search.md) for full
details.
+
+- **`BLOB`** — stores arbitrary binary objects (images, audio, documents)
either inline within the
+ base file or as external references. See [BLOB / Unstructured
Data](blob_unstructured_data.md)
+ for the storage modes, DDL syntax, and read APIs.
diff --git a/website/docs/sql_queries.md b/website/docs/sql_queries.md
index 9310c72fbc62..4b9d6c2dd695 100644
--- a/website/docs/sql_queries.md
+++ b/website/docs/sql_queries.md
@@ -2,7 +2,7 @@
title: SQL Queries
summary: "In this page, we go over querying Hudi tables using SQL"
toc: true
-last_modified_at:
+last_modified_at: 2026-05-27T00:00:00-00:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -14,6 +14,24 @@ This page will show how to issue different queries and
discuss any specific inst
## Spark SQL
The Spark [quickstart](quick-start-guide.md) provides a good overview of how
to use Spark SQL to query Hudi tables. This section will go into more advanced
configurations and functionalities.
+:::tip Setting Hudi read options at the session level
+Hudi 1.2.0 supports setting read options at the **Spark session level** using
the `spark.hoodie.*` prefix.
+Any `spark.hoodie.X` config set via `spark.conf.set` or `--conf` is treated
equivalently to `hoodie.X`.
+
+Config precedence (low → high):
+1. Global DFS properties
+2. `spark.hoodie.*` session-level configs (normalized to `hoodie.*`)
+3. Explicit `hoodie.*` data source options or per-table `SET` commands
+
+```sql
+-- Apply a Hudi read option for the entire session
+SET spark.hoodie.metadata.column.stats.enable = true;
+SELECT * FROM hudi_table WHERE price BETWEEN 10.0 AND 50.0;
+```
+
+If both `spark.hoodie.X` and `hoodie.X` are set, the explicit `hoodie.X` value
takes precedence.
+:::
+
### Snapshot Query
Snapshot queries are the most common query type for Hudi tables. Spark SQL
supports snapshot queries on both COPY_ON_WRITE and MERGE_ON_READ tables.
Using session properties, you can specify options around indexing to optimize
query performance, as shown below.
@@ -332,6 +350,19 @@ also changed to use completion time. To support
compatiblity, Hudi does a checkp
time to completion time depending on the source table version.
:::
+### Vector Similarity Search
+
+Hudi 1.2.0 introduces a `hudi_vector_search` table-valued function (TVF) for
approximate
+nearest-neighbor (ANN) search over `VECTOR` columns. This is an extension of
the
+`hudi_table_changes` TVF pattern.
+
+```sql
+-- Find the 10 nearest neighbors to a query vector in the 'embedding' column
+SELECT * FROM hudi_vector_search('db.embeddings_table', 'embedding',
ARRAY(0.1, 0.2, ...), 10);
+```
+
+See [Vector Search](vector_search.md) for the full API, supported metrics, and
setup instructions.
+
### Query Indexes and Timeline
Hudi also allows users to directly query the metadata partitions and check the
metadata corresponding to the table
diff --git a/website/docs/syncing_aws_glue_data_catalog.md
b/website/docs/syncing_aws_glue_data_catalog.md
index 35f43a8af472..4b1e295dc831 100644
--- a/website/docs/syncing_aws_glue_data_catalog.md
+++ b/website/docs/syncing_aws_glue_data_catalog.md
@@ -54,6 +54,10 @@
hoodie.datasource.meta.sync.glue.partition_index_fields.enable
hoodie.datasource.meta.sync.glue.partition_index_fields
```
+## Writer Version Table Property
+
+Hudi 1.2.0 Glue sync writes the table property `hudi_writer_version` (set to
the Hudi version that last synced the table) to the Glue Data Catalog entry on
every sync, consistent with HMS sync behavior.
+
## Other references
### Running AWS Glue Catalog Sync for Spark DataSource
diff --git a/website/docs/syncing_metastore.md
b/website/docs/syncing_metastore.md
index f260a585c1b6..059f172de003 100644
--- a/website/docs/syncing_metastore.md
+++ b/website/docs/syncing_metastore.md
@@ -297,3 +297,44 @@ While using hive beeline query, you need to enter settings:
```bash
set hive.input.format =
org.apache.hudi.hadoop.hive.HoodieCombineHiveInputFormat;
```
+
+## Spark Catalog Metastore Client
+
+When running Hudi inside a Spark environment that already has Hive support
enabled (e.g., SparkSQL with `spark.sql.catalogImplementation=hive`), the
standard `IMetaStoreClient` initialization can conflict with Spark's own Hive
classloader. Setting
+
+```properties
+hoodie.datasource.hive_sync.use_spark_catalog=true
+```
+
+(default: `false`) makes Hudi use `SparkCatalogMetaStoreClient` — a
Spark-native `IMetaStoreClient` implementation — instead of creating its own.
This avoids classloader conflicts in Hive-on-Spark setups. Requires a
`SparkSession` with Hive support active.
+
+## HMS 4.x Support via JDBC Fallback
+
+HMS 4.x changed several Thrift API method signatures (e.g., `get_table` →
`get_table_req`), which makes the standard Thrift-based HMS client
incompatible. Hudi 1.2.0 adds automatic fallback: when a Thrift metadata call
surfaces a `TApplicationException` anywhere in its cause chain, Hudi flips an
internal `thriftIncompatible` flag and reroutes the rest of that sync run
through the JDBC path.
+
+**Requirement:** sync mode must be `jdbc` with a valid JDBC URL so the
fallback client is available:
+
+```properties
+hoodie.datasource.hive_sync.mode=jdbc
+hoodie.datasource.hive_sync.jdbcurl=jdbc:hive2://hiveserver:10000
+hoodie.datasource.hive_sync.username=<username>
+hoodie.datasource.hive_sync.password=<password>
+```
+
+If the table is synced with `mode=hms` or `mode=hiveql` against HMS 4.x, Hudi
logs `"Thrift API incompatible with HMS but no JDBC fallback available.
Consider using mode=jdbc with a valid jdbcUrl."` and surfaces the original
exception — no automatic recovery happens.
+
+**Detection scope.** The flag is per `HoodieHiveSyncClient` instance, not
global, and only transitions from `false` to `true` (it never resets). In
practice this means the first Thrift call of each sync run probes once, and the
rest of that run uses the JDBC fallback. The next sync run starts with a fresh
probe.
+
+**JDBC connection failures surface separately.** With `mode=jdbc`, Hudi opens
the JDBC connection eagerly when the sync client is constructed — before any
Thrift call is attempted. A bad JDBC URL, missing driver, or wrong credentials
therefore fails at startup with `HoodieHiveSyncException: Failed to create
HiveMetaStoreClient` and the underlying JDBC exception as the cause in the
stack trace. This is a configuration-error path, not an HMS API mismatch, and
is the same behavior as `mode= [...]
+
+## Writer Version Table Property
+
+Hudi 1.2.0 sync writes the table property `hudi_writer_version` (set to the
Hudi version that last synced the table) to the Hive metastore entry on every
sync. This allows tooling and metastore administrators to identify which Hudi
version wrote a given table.
+
+To emit `TOUCH` events to the metastore for partition-level change tracking
(e.g., for downstream catalog notifications), set:
+
+```properties
+hoodie.meta.sync.touch.partitions.enabled=true
+```
+
+Default is `false`. When enabled, a TOUCH event is issued for each partition
that was modified in the sync operation.
diff --git a/website/docs/variant_type.md b/website/docs/variant_type.md
index bf42390afe2f..fdcf6f54b54b 100644
--- a/website/docs/variant_type.md
+++ b/website/docs/variant_type.md
@@ -3,7 +3,7 @@ title: "Semi-Structured Data (VARIANT)"
keywords: [ hudi, variant, semi-structured, json, schemaless, shredding,
parse_json, flexible schema]
summary: "Store and query semi-structured JSON-like data in Hudi tables using
the VARIANT type, with optional shredding for query performance"
toc: true
-last_modified_at: 2026-04-25T00:00:00-00:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
import Tabs from '@theme/Tabs';
@@ -253,12 +253,25 @@ binary `value` field.
| Engine | VARIANT Support |
|:-------|:---------------|
-| **Spark 4.0+** | Native `VariantType` — full read/write/query |
+| **Spark 4.0** | Native `VariantType` — full read/write/query for COW and
MOR; native `df.write` with `VariantType` on the V1 datasource |
+| **Spark 4.1** | Native `VariantType` — full read/write/query for COW and MOR
|
| **Spark 3.x** | Reads as `STRUCT<value: BINARY, metadata: BINARY>` —
backward compatible |
-| **Flink** | Reads as `ROW<metadata BYTES, value BYTES>` — cross-engine
compatible |
+| **Flink** | Native `VARIANT` operations are not supported. Tables written by
Spark with VARIANT columns can be read in Flink only as the underlying
`ROW<metadata BYTES, value BYTES>` struct. |
-A VARIANT table written by Spark 4.0 can be read by Spark 3.x or Flink, and
vice versa. The
-binary encoding is engine-independent.
+A VARIANT table written by Spark 4.0/4.1 can be read by Spark 3.x using the
underlying binary struct, or by Flink as `ROW<metadata BYTES, value BYTES>`.
The binary encoding is engine-independent.
+
+## Metastore Sync
+
+When syncing VARIANT column schemas to external catalogs, Hudi maps the binary
encoding to the
+target catalog's native struct type:
+
+| Catalog | VARIANT representation |
+|:--------|:----------------------|
+| Hive | `STRUCT<metadata:BINARY, value:BINARY>` |
+| BigQuery | `STRUCT` with `metadata` and `value` fields (`BYTES` type) |
+
+Query engines that support VARIANT (Spark 4.0+, Flink 2.1+) read the table
directly using the
+Parquet VARIANT annotation and do not go through the Hive/BigQuery metastore
representation.
## Use Cases for AI Workloads
@@ -344,3 +357,6 @@ CREATE TABLE api_responses (
- Native `VARIANT` keyword in DDL requires Spark 4.0+. On Spark 3.x, use the
struct representation.
- VARIANT shredding configuration is determined at write time based on the
schema definition.
- Complex path expressions within VARIANT may require casting to STRING and
then using JSON functions.
+- Native VARIANT operations are not supported on Flink. VARIANT columns
surface as `ROW<metadata BYTES, value BYTES>` and can be read but not natively
decoded or queried as a variant.
+- VARIANT columns are **not supported** on Lance-backed tables. Use Parquet as
the base file format
+ for tables containing VARIANT columns.
diff --git a/website/docs/vector_search.md b/website/docs/vector_search.md
index 9c53443161a7..4e6687057a6c 100644
--- a/website/docs/vector_search.md
+++ b/website/docs/vector_search.md
@@ -1,9 +1,9 @@
---
title: "Vector Search"
-keywords: [ hudi, vector, search, embeddings, similarity, cosine, ANN, nearest
neighbor, VECTOR type]
-summary: "Store embedding vectors in Hudi tables and run approximate nearest
neighbor search using the VECTOR type and hudi_vector_search TVF"
+keywords: [ hudi, vector, search, embeddings, similarity, cosine, nearest
neighbor, VECTOR type]
+summary: "Store embedding vectors in Hudi tables and run vector similarity
search using the VECTOR type and hudi_vector_search TVF"
toc: true
-last_modified_at: 2026-04-25T00:00:00-00:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
import Tabs from '@theme/Tabs';
@@ -13,6 +13,20 @@ Hudi's `VECTOR` type and `hudi_vector_search` table-valued
function (TVF) bring
to the data lakehouse. Store embeddings alongside your structured data and
query them with familiar Spark SQL —
no external vector database required.
+## Storage Format
+
+VECTOR columns are stored in Parquet as `FIXED_LEN_BYTE_ARRAY` — a
fixed-length binary encoding of the
+float array. Hudi stamps `hudi_type` metadata on the column so the Spark
reader knows to decode the
+bytes back into a typed array.
+
+On **Lance** tables, VECTOR columns are stored natively as Lance
`FixedSizeList<Float32/Float64, dim>`,
+so embeddings are written without conversion overhead at the file-format
layer. See
+[Lance File Format](lance_file_format.md) for details.
+
+The `VECTOR(dim[, elementType])` DDL syntax works across Spark 3.4, 3.5, 4.0,
and 4.1. Hudi's SQL
+parser normalizes `VECTOR(128, FLOAT)` to `VECTOR(128)` (FLOAT is the default
element type).
+Nesting VECTOR inside STRUCT, ARRAY, or MAP is not supported.
+
## VECTOR Type
The `VECTOR(dim[, elementType])` type declares a column that stores
fixed-dimensional embedding vectors.
@@ -98,8 +112,8 @@ INSERT INTO products VALUES (
## hudi_vector_search TVF
-The `hudi_vector_search` table-valued function performs approximate nearest
neighbor (ANN) search
-over a VECTOR column.
+The `hudi_vector_search` table-valued function returns the `top_k` rows from a
Hudi table whose
+VECTOR column is closest to a given query vector under a chosen distance
metric.
### Syntax
@@ -240,3 +254,21 @@ FROM hudi_vector_search(
- VECTOR columns must be **top-level fields** — nesting inside STRUCT, ARRAY,
or MAP is not supported.
- The query vector's element type must **exactly match** the corpus
embedding's element type (no implicit casting).
- VECTOR dimension and element type **cannot be changed** after table creation
via schema evolution.
+- **Flink cannot read VECTOR columns.** VECTOR data is stored as Parquet
`FIXED_LEN_BYTE_ARRAY`, which
+ Flink's Parquet reader does not decode back into a typed array. Flink can
still read all **other**
+ columns in a table that contains a VECTOR column — only the VECTOR column
itself is inaccessible.
+ Use Spark to query VECTOR columns.
+
+## Metastore Sync
+
+When syncing VECTOR column schemas to external catalogs, Hudi maps the binary
encoding to the
+target catalog's native binary type, preserving the original VECTOR metadata
in table properties:
+
+| Catalog | VECTOR representation |
+|:--------|:---------------------|
+| Hive | `BINARY` |
+| BigQuery | `BYTES` |
+
+The `VECTOR(dim, elementType)` dimension and element-type metadata is
preserved in
+`TBLPROPERTIES`/table descriptions so the table can be correctly reconstructed
by Spark even after
+a metastore round-trip.
diff --git a/website/docs/writing_data.md b/website/docs/writing_data.md
index 5e4570182692..dcdc5a33bd81 100644
--- a/website/docs/writing_data.md
+++ b/website/docs/writing_data.md
@@ -1,7 +1,7 @@
---
title: Batch Writes
keywords: [hudi, incremental, batch, processing]
-last_modified_at: 2024-03-13T15:59:57-04:00
+last_modified_at: 2026-05-27T00:00:00-00:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
@@ -441,5 +441,32 @@ inputDF.write.format("hudi")
.save(basePath)
```
+### Rolling Extra Metadata
+
+Rolling extra metadata allows you to automatically carry forward selected
commit metadata keys to every subsequent commit and clean instant without
having to walk the full timeline. This is particularly useful for persisting
checkpoint information such as Kafka offsets or Flink checkpoints across
commits.
+
+| Config | Default | Description |
+|---|---|---|
+| `hoodie.write.rolling.metadata.keys` | `""` (disabled) | Comma-separated
list of extra metadata keys to carry forward to each new commit and clean
instant. Values are read from recent completed instants and written into the
new commit metadata, so they remain accessible without walking the timeline.
New values override old ones. Only applies to data table commits and clean
instants. |
+| `hoodie.write.rolling.metadata.timeline.lookback.commits` | `10` | Maximum
number of completed instants to walk back when searching for the configured
rolling metadata keys. Higher values improve resilience at a small performance
cost. |
+
+**Example:**
+
+```java
+inputDF.write.format("hudi")
+ .option("hoodie.write.rolling.metadata.keys",
"kafka.offset.partition.0,kafka.offset.partition.1")
+ .option("hoodie.write.rolling.metadata.timeline.lookback.commits", "10")
+ // ... other options
+ .save(basePath)
+```
+
+### Advanced Storage Options
+
+The following advanced storage configuration options were added in Hudi 1.2.0:
+
+| Config | Default | Description |
+|---|---|---|
+| `hoodie.parquet.write.config.injector.class` | (none) | Fully-qualified
class name of a custom `HoodieParquetConfigInjector` implementation. Use this
to inject custom Parquet writer properties (e.g., disable dictionary encoding,
set bloom filter sizes) without modifying the Hudi source. The implementing
class must implement `org.apache.hudi.io.HoodieParquetConfigInjector`. |
+
## Java Client
We can use plain java to write to hudi tables. To use Java client we can
refere
[here](https://github.com/apache/hudi/blob/master/hudi-examples/hudi-examples-java/src/main/java/org/apache/hudi/examples/java/HoodieJavaWriteClientExample.java)
