This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 3f5766d72dc4 docs: cover native python and rust writes in the hudi-rs
quick start (#20470)
3f5766d72dc4 is described below
commit 3f5766d72dc40089e10a6cd5dc41c4c5601aa605
Author: Y Ethan Guo <[email protected]>
AuthorDate: Tue Nov 3 12:05:25 2026 -0800
docs: cover native python and rust writes in the hudi-rs quick start
(#20470)
---
website/docs/python-rust-quick-start-guide.md | 449 +++++++++++++++------
website/docs/use_cases.md | 4 +-
.../docs/python-rust-quick-start/write-bytes.png | Bin 0 -> 178228 bytes
.../docs/python-rust-quick-start/write-latency.png | Bin 0 -> 192218 bytes
website/static/llms.txt | 16 +-
.../version-1.2.0/python-rust-quick-start-guide.md | 449 +++++++++++++++------
website/versioned_docs/version-1.2.0/use_cases.md | 4 +-
.../version-1.2.1/python-rust-quick-start-guide.md | 449 +++++++++++++++------
website/versioned_docs/version-1.2.1/use_cases.md | 4 +-
9 files changed, 978 insertions(+), 397 deletions(-)
diff --git a/website/docs/python-rust-quick-start-guide.md
b/website/docs/python-rust-quick-start-guide.md
index bddded7d8f0a..140efe2fa8e5 100644
--- a/website/docs/python-rust-quick-start-guide.md
+++ b/website/docs/python-rust-quick-start-guide.md
@@ -1,184 +1,347 @@
---
title: "Python/Rust Quick Start"
toc: true
-last_modified_at: 2024-11-28T12:53:57+08:00
+last_modified_at: 2026-11-03T00:00:00-08:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-This guide will help you get started with
[Hudi-rs](https://github.com/apache/hudi-rs), the native Rust implementation
for Apache Hudi with Python bindings. Learn how to install, set up, and perform
basic operations using both Python and Rust interfaces.
+[hudi-rs](https://github.com/apache/hudi-rs) is the native Rust implementation
of Apache Hudi with Python bindings.
+It creates Hudi tables and runs upserts, updates, and deletes on them from a
single process, with no JVM and no
+Spark. Every write maintains the table's metadata table and record-level
index, so updating or deleting a handful
+of rows in a large table costs a few index lookups and the rewrite of the file
groups that hold those rows, not a
+scan of the table.
+
+This page gets you from `pip install` to a table you can upsert into, delete
from, and read incrementally, then
+shows how to query it from Python, Rust, DuckDB, Polars, and DataFusion.
## Installation
```bash
-# Python
+# Python (3.10+)
pip install hudi
# Rust
cargo add hudi
+cargo add arrow@59
+cargo add tokio@1 --features macros,rt-multi-thread
```
-## Usage Examples
+## Write a table in Python
-> [!NOTE]
-> These examples expect a Hudi table exists at `/tmp/trips_table`, created
using
-> the [quick start guide](quick-start-guide.md).
+A retrieval or training pipeline end to end: a table of document chunks with
their embeddings, keyed by
+`chunk_id`. Ingest, re-sync the documents that changed, drop the ones that
were removed, and read back only what
+changed since the last run.
-### Snapshot Query
+```python
+import tempfile
-Snapshot query reads the latest version of the data from the table. The table
API also accepts partition filters.
+import numpy as np
+import pyarrow as pa
+from hudi import HudiTable, HudiReadOptions, HudiQueryType
+
+def chunks(doc_ids, version):
+ rows = [(f"{d}#{i}", d, i, f"chunk {i} of {d}, version {version}") for d
in doc_ids for i in range(4)]
+ embeddings = np.random.default_rng(version).random((len(rows), 384),
dtype=np.float32)
+ return pa.table({
+ "chunk_id": [r[0] for r in rows],
+ "doc_id": [r[1] for r in rows],
+ "pos": pa.array([r[2] for r in rows], pa.int32()),
+ "text": [r[3] for r in rows],
+ "embedding": pa.array(embeddings.tolist(), pa.list_(pa.float32())),
+ "updated_at": pa.array([version] * len(rows), pa.int64()),
+ })
+
+table = HudiTable.create(
+ f"{tempfile.mkdtemp()}/contract_chunks",
+ record_key="chunk_id", # the record-level index is built on this
key
+ ordering_fields="updated_at", # the later version wins when a key
arrives twice
+ record_index_enabled=True, # on by default; locates keys without
scanning files
+)
-#### Python
+base = table.append(chunks(["doc1", "doc2", "doc3"], version=1))
-```python
-from hudi import HudiTableBuilder
-import pyarrow as pa
+# doc2 changed: re-embed its chunks and upsert them. The index finds the file
+# groups holding these keys; nothing else in the table is read.
+result = table.upsert(chunks(["doc2"], version=2))
+print(result.num_updates, result.num_inserts) # 4 0
-hudi_table = HudiTableBuilder.from_base_uri("/tmp/trips_table").build()
-batches = hudi_table.read_snapshot(filters=[("city", "=", "san_francisco")])
+# doc3 was removed: delete its chunks.
+table.delete("doc_id = 'doc3'")
-# convert to PyArrow table
-arrow_table = pa.Table.from_batches(batches)
-result = arrow_table.select(["rider", "city", "ts", "fare"])
-print(result)
+# Train on what changed after the initial load, not on the whole table.
+changed = table.read(
+ HudiReadOptions()
+ .with_query_type(HudiQueryType.Incremental)
+ .with_start_timestamp(base.completion_time)
+)
+print(pa.Table.from_batches(changed).num_rows) # 4, the re-embedded chunks
of doc2
+
+# Query the current snapshot with DuckDB (or Polars, pandas, anything Arrow).
+import duckdb
+snapshot = pa.Table.from_batches(table.read())
+duckdb.sql("SELECT doc_id, count(*) AS chunks FROM snapshot GROUP BY doc_id
ORDER BY doc_id").show()
```
-#### Rust
+`append`, `upsert`, and `overwrite` accept a `pyarrow.Table`,
`pyarrow.RecordBatch`, `pyarrow.RecordBatchReader`,
+or any object that implements the Arrow PyCapsule interface, so a Polars
DataFrame or a pandas 2.2+ DataFrame can
+be passed directly. Embeddings may arrive as `list<float32>` or
`fixed_size_list<float32>`; both are stored as lists.
+
+`update` sets columns by predicate and `delete_keys` deletes by key:
+
+```python
+table.update("doc_id = 'doc1'", {"text": "redacted"})
+table.delete_keys(["doc1#0", "doc1#1"])
+```
+
+## Write a table in Rust
+
+The same steps with the `hudi` crate. Batches are Arrow `RecordBatch`es.
```rust
+use arrow::array::{ArrayRef, Float64Array, Int64Array, RecordBatch,
StringArray};
+use arrow::datatypes::{DataType, Field, Schema};
+use hudi::config::table::TableTypeValue;
use hudi::error::Result;
-use hudi::table::builder::TableBuilder as HudiTableBuilder;
-use arrow::compute::concat_batches;
+use hudi::table::{ReadOptions, Table};
+use std::sync::Arc;
+
+fn trips(rows: &[(&str, &str, i64, f64)]) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("id", DataType::Utf8, false),
+ Field::new("city", DataType::Utf8, false),
+ Field::new("ts", DataType::Int64, false),
+ Field::new("fare", DataType::Float64, false),
+ ]));
+ RecordBatch::try_new(schema, vec![
+ Arc::new(StringArray::from(rows.iter().map(|r|
r.0).collect::<Vec<_>>())) as ArrayRef,
+ Arc::new(StringArray::from(rows.iter().map(|r|
r.1).collect::<Vec<_>>())),
+ Arc::new(Int64Array::from(rows.iter().map(|r|
r.2).collect::<Vec<_>>())),
+ Arc::new(Float64Array::from(rows.iter().map(|r|
r.3).collect::<Vec<_>>())),
+ ]).unwrap()
+}
#[tokio::main]
async fn main() -> Result<()> {
- let hudi_table =
HudiTableBuilder::from_base_uri("/tmp/trips_table").build().await?;
- let batches = hudi_table.read_snapshot(&[("city", "=",
"san_francisco")]).await?;
- let batch = concat_batches(&batches[0].schema(), &batches)?;
- let columns = vec!["rider", "city", "ts", "fare"];
- for col_name in columns {
- let idx = batch.schema().index_of(col_name).unwrap();
- println!("{}: {}", col_name, batch.column(idx));
- }
+ let mut table = Table::create("/tmp/trips")
+ .with_table_name("trips")
+ .with_table_type(TableTypeValue::CopyOnWrite) // or MergeOnRead
+ .with_record_key_fields(["id"])
+ .with_partition_fields(["city"])
+ .with_ordering_fields(["ts"])
+ .create()
+ .await?;
+
+ table.append([trips(&[("t1", "sf", 1, 30.0), ("t2", "sf", 1, 45.0), ("t3",
"nyc", 1, 20.0)])]).await?;
+
+ let result = table.upsert([trips(&[("t2", "sf", 2, 50.0), ("t4", "nyc", 2,
15.0)])]).await?;
+ println!(
+ "requested_time={} updates={} inserts={}",
+ result.requested_time.as_deref().unwrap_or("none"),
+ result.num_updates,
+ result.num_inserts
+ );
+
+ table.delete("id = 't1'").await?;
+
+ let batches = table.read(&ReadOptions::new()).await?;
+ println!("{} rows", batches.iter().map(|b| b.num_rows()).sum::<usize>());
Ok(())
}
```
-To run read-optimized (RO) query on Merge-on-Read (MOR) tables, set
`hoodie.read.use.read_optimized.mode` when creating the table.
+Both writers produce the same on-disk layout as the Spark writer.
-#### Python
+## Compact a Merge-on-Read table
+
+On a Merge-on-Read table (`table_type="mor"` in Python,
`TableTypeValue::MergeOnRead` in Rust), upserts, updates,
+and deletes append log files. Compaction folds them into new base files under
one compaction commit;
+rows keep their commit time, so incremental reads see no new rows from it. Run
it on demand, or set
+`hoodie.compact.inline=true` (in the write call) or
`hoodie.datasource.compaction.async.enable=true` (on a
+background task) to compact once `hoodie.compact.inline.max.delta.commits`
(default 5) delta commits accumulate.
+The metadata table compacts automatically.
+
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-hudi_table = (
- HudiTableBuilder
- .from_base_uri("/tmp/trips_table")
- .with_option("hoodie.read.use.read_optimized.mode", "true")
- .build()
+mor = HudiTable.create(
+ f"{tempfile.mkdtemp()}/contract_chunks_mor",
+ record_key="chunk_id",
+ ordering_fields="updated_at",
+ table_type="mor",
+ # Optional: also compact in the write call every 5 delta commits
+ hudi_options={"hoodie.compact.inline": "true"},
)
+mor.append(chunks(["doc1", "doc2"], version=1))
+mor.upsert(chunks(["doc2"], version=2)) # the updates go to log files
+
+result = mor.compact()
+print(result.num_file_groups, result.num_log_files)
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
-let hudi_table =
- HudiTableBuilder::from_base_uri("/tmp/trips_table")
- .with_option("hoodie.read.use.read_optimized.mode", "true")
- .build().await?;
+let mut mor = Table::create("/tmp/trips_mor")
+ .with_table_name("trips_mor")
+ .with_table_type(TableTypeValue::MergeOnRead)
+ .with_record_key_fields(["id"])
+ .with_partition_fields(["city"])
+ .with_ordering_fields(["ts"])
+ // Optional: also compact in the write call every 5 delta commits
+ .with_option("hoodie.compact.inline", "true")
+ .create()
+ .await?;
+mor.append([trips(&[("t1", "sf", 1, 30.0), ("t2", "nyc", 1, 20.0)])]).await?;
+mor.upsert([trips(&[("t1", "sf", 2, 31.0)])]).await?; // the update goes to a
log file
+
+let result = mor.compact().await?;
+println!("{} {}", result.num_file_groups, result.num_log_files); // 1 1
```
-> [!NOTE]
-> Currently reading MOR tables is limited to tables with Parquet data blocks.
+</TabItem>
+</Tabs>
-### Time-Travel Query
+## Why upserts and deletes are fast: the record-level index
-Time-travel query reads the data at a specific timestamp from the table. The
table API also accepts partition filters.
+A client with no index answers "which file holds key `doc2#1`?" by scanning
every file, or at best every
+file whose min/max statistics admit the key. With keys like `<doc>#<pos>` or
UUIDs that is the whole table on every
+sync, however few documents changed.
-#### Python
+Hudi keeps a [record-level index](indexes.md#record-index) in the table's
metadata table: a persistent map from record
+key to file group, sharded by key hash. hudi-rs writes it on `create` and
updates it on every commit. A sync that
+touches 10,000 chunks in a table of millions costs an index lookup of 10,000
keys, which names the file groups to
+rewrite (or, on a Merge-on-Read table, to append a log block to). The rest of
the table is never read.
-```python
-batches = (
- hudi_table
- .read_snapshot_as_of("20241231123456789", filters=[("city", "=",
"san_francisco")])
-)
-```
+## Use cases and numbers
-#### Rust
+The
[`benchmark/upsert`](https://github.com/apache/hudi-rs/tree/543642771a1440f8470053831463052e190b3923/benchmark/upsert)
harness in the hudi-rs
+repository runs five single-node workloads against hudi-rs, each a bulk load
followed by five upsert commits and five
+delete commits; the numbers below also include delta-rs and PyIceberg run
through the same commits on the same
+machine. Every end state is checked against a plain Parquet copy of the
expected table, and the machine's load, memory,
+and disk are recorded next to every timing. Each scenario below links to its
definition so the numbers can be
+reproduced on your own machine.
-```rust
-let batches =
- hudi_table
- .read_snapshot_as_of("20241231123456789", &[("city", "=",
"san_francisco")]).await?;
-```
+| Scenario | Models | Table | Each commit |
+|---|---|---|---|
+|
[`rag-chunks`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#rag-chunks)
| Re-syncing a
[RAG](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) knowledge
base as its sources change | 1M synthetic chunks with 384-float embeddings from
100,000 documents, keyed by `chunk_id` | re-ingest 1,000 changed documents
(10,000 chunks), delete 100 removed documents |
+|
[`training-set`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#training-set)
| Curating a fine-tuning set between training runs | 1M synthetic examples
keyed by `example_id` | upsert 10,000 corrected examples, delete 1,000 rejected
ones, then an incremental read of the changes and a time-travel read of the
first version |
+|
[`uuid`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#uuid)
| Random updates to a dimension table, with changed keys spread across every
file | 10M synthetic rows with random UUID keys | upsert 10,000 existing keys
plus 1,000 new, delete 1,000 |
+|
[`nyc-taxi`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#nyc-taxi)
| Correcting a public dataset in place | [NYC TLC yellow taxi
trips](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page), January
2022, 2.5M rows | upsert 1% of trips with corrected fares, delete 0.1% |
+|
[`tpcds-web-returns`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#tpcds-web-returns)
| A standard warehouse fact table | [TPC-DS](https://www.tpc.org/tpcds/)
`web_returns` at scale factor 10, 0.7M rows, composite key | upsert 1% matched
plus 10% new rows, delete 5% |
+
+{/* BENCHMARK_TABLE_START */}
+**Why hudi-rs is faster on upserts and deletes.** Low write amplification and
the record-level index: a Merge-on-Read
+commit appends only the changed records to log files, and the record-level
index points each key at its file group, so
+nothing else in the table is read or rewritten. Neither delta-rs nor PyIceberg
writes deletion vectors in their latest
+releases ([delta-rs
1.6.6](https://github.com/delta-io/delta-rs/releases/tag/python-v1.6.6),
+[PyIceberg
0.12.0](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.12.0)),
and neither keeps a
+record-level index, so every update or delete rewrites each data file it
touches. The changed keys here are spread
+across the table, so each Copy-on-Write commit rewrites nearly all of it: per
upsert commit, hudi-rs wrote 19 MiB on
+`rag-chunks` where delta-rs rewrote 1,689 MiB and PyIceberg 1,432 MiB, and 2
MiB on `uuid` against 671 MiB and 388 MiB.
-### Incremental Query
+<div style={{ textAlign: 'center' }}>
+ <img src="/assets/images/docs/python-rust-quick-start/write-latency.png"
alt="Seconds per upsert and delete commit on five workloads: hudi-rs 0.6.0
Merge-on-Read with the record-level index against delta-rs and PyIceberg"
width="1000"/>
+</div>
-Incremental query reads the changed data from the table for a given time range.
+<div style={{ textAlign: 'center' }}>
+ <img src="/assets/images/docs/python-rust-quick-start/write-bytes.png"
alt="MiB written per upsert and delete commit on the same five workloads"
width="1000"/>
+</div>
-#### Python
+Loading the 10M-row table in Hudi takes 5.7 s, about 2.4 s of which builds the
metadata table and its record-level
+index, an investment that pays off in the latency of every later upsert and
delete.
+{/* BENCHMARK_TABLE_END */}
+
+**Reproduce it.** [This
guide](https://gist.github.com/yihua/66ad80224f8e50020025dc76d138dd0a) pins the
hudi-rs
+commit, every package version and the delta-rs and PyIceberg cases, and gives
the exact commands.
+
+## Read a table
+
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-# read the records between t1 (exclusive) and t2 (inclusive)
-batches = hudi_table.read_incremental_records(t1, t2)
+from hudi import HudiReadOptions, HudiTableBuilder
+import pyarrow as pa
+
+table = HudiTableBuilder.from_base_uri("/tmp/trips").build()
+
+# Snapshot, with partition and row filters pushed down
+batches = table.read(HudiReadOptions(filters=[("city", "=", "sf")]))
+print(pa.Table.from_batches(batches))
+
+# Time travel
+batches =
table.read(HudiReadOptions().with_as_of_timestamp("20261029120000000"))
+
+# Incremental: rows from commits that completed after a point in time, such as
+# the completion_time of an earlier write
+from hudi import HudiQueryType
+since = "20261029120000000"
+batches =
table.read(HudiReadOptions().with_query_type(HudiQueryType.Incremental).with_start_timestamp(since))
-# read the records after t1
-batches = hudi_table.read_incremental_records(t1)
+# Stream batches instead of materializing the result
+for batch in table.read_stream(HudiReadOptions(projection=["id",
"fare"]).with_batch_size(4096)):
+ print(batch.num_rows)
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
-// read the records between t1 (exclusive) and t2 (inclusive)
-let batches = hudi_table.read_incremental_records(t1, Some(t2)).await?;
+use hudi::table::builder::TableBuilder as HudiTableBuilder;
+use hudi::table::{QueryType, ReadOptions};
-// read the records after t1
-let batches = hudi_table.read_incremental_records(t1, None).await?;
-```
+let table = HudiTableBuilder::from_base_uri("/tmp/trips").build().await?;
-> [!NOTE]
-> Currently the only supported format for the timestamp arguments is Hudi
Timeline format: `yyyyMMddHHmmssSSS` or `yyyyMMddHHmmss`.
+// Snapshot, with partition and row filters pushed down
+let batches = table.read(&ReadOptions::new().with_filters([("city", "=",
"sf")])?).await?;
-## Query Engine Integration
+// Time travel
+let batches =
table.read(&ReadOptions::new().with_as_of_timestamp("20261029120000000")).await?;
-Hudi-rs provides APIs to support integration with query engines. The sections
below highlight some commonly used APIs.
+// Incremental: rows from commits that completed after a point in time, such as
+// the completion_time of an earlier write
+let since = "20261029120000000";
+let options =
ReadOptions::new().with_query_type(QueryType::Incremental).with_start_timestamp(since);
+let batches = table.read(&options).await?;
+```
-### Table API
+</TabItem>
+</Tabs>
-Create a Hudi table instance using its constructor or the `TableBuilder` API.
+Timestamps accept the Hudi timeline format (`yyyyMMddHHmmssSSS`), Unix epoch
in seconds through nanoseconds, and
+RFC 3339 with a timezone offset.
-| Stage | API | Description
|
-|-----------------|-------------------------------------------|--------------------------------------------------------------------------------|
-| Query planning | `get_file_slices()` | For snapshot
query, get a list of file slices. |
-| | `get_file_slices_splits()` | For snapshot
query, get a list of file slices in splits. |
-| | `get_file_slices_as_of()` | For
time-travel query, get a list of file slices at a given time. |
-| | `get_file_slices_splits_as_of()` | For
time-travel query, get a list of file slices in splits at a given time. |
-| | `get_file_slices_between()` | For
incremental query, get a list of changed file slices between a time range. |
-| Query execution | `create_file_group_reader_with_options()` | Create a file
group reader instance with the table instance's configs. |
+To run a read-optimized query on a Merge-on-Read table (base files only, no
log merge), set
+`hoodie.read.use.read_optimized.mode` to `true` in the read options.
-### File Group API
+## Query with DuckDB, Polars, and DataFusion
-Create a Hudi file group reader instance using its constructor or the Hudi
table API `create_file_group_reader_with_options()`.
+hudi-rs returns Arrow, so anything that consumes Arrow can query a Hudi table
in Python.
-| Stage | API | Description
|
-|-----------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Query execution | `read_file_slice()` | Read records from
a given file slice; based on the configs, read records from only base file, or
from base file and log files, and merge records based on the configured
strategy. |
+```python
+import duckdb, polars as pl, pyarrow as pa
+from hudi import HudiTableBuilder
+trips =
pa.Table.from_batches(HudiTableBuilder.from_base_uri("/tmp/trips").build().read())
-### Apache DataFusion
+duckdb.sql("SELECT city, avg(fare) FROM trips GROUP BY city").show()
+print(pl.from_arrow(trips).group_by("city").agg(pl.col("fare").mean()))
+```
-Enabling the `hudi` crate with `datafusion` feature will provide a
[DataFusion](https://datafusion.apache.org/)
-extension to query Hudi tables.
+In Rust, the `datafusion` feature of the `hudi` crate registers a Hudi table
as a DataFusion table provider.
<details>
-<summary>Add crate hudi with datafusion feature to your application to query a
Hudi table.</summary>
+<summary>Add the hudi crate with the datafusion feature to your
application.</summary>
```shell
cargo new my_project --bin && cd my_project
-cargo add tokio@1 datafusion@43
+cargo add tokio@1 datafusion@55 # the DataFusion major hudi 0.6 builds against
cargo add hudi --features datafusion
```
-Update `src/main.rs` with the code snippet below then `cargo run`.
-
</details>
```rust
@@ -191,57 +354,83 @@ use hudi::HudiDataSource;
#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
- let hudi = HudiDataSource::new_with_options(
- "/tmp/trips_table",
- [("hoodie.read.input.partitions", "5")]).await?;
- ctx.register_table("trips_table", Arc::new(hudi))?;
- let df: DataFrame = ctx.sql("SELECT * from trips_table where city =
'san_francisco'").await?;
+ let hudi = HudiDataSource::new_with_options("/tmp/trips",
[("hoodie.read.input.partitions", "5")]).await?;
+ ctx.register_table("trips", Arc::new(hudi))?;
+ let df: DataFrame = ctx.sql("SELECT * FROM trips WHERE city =
'sf'").await?;
df.show().await?;
Ok(())
}
```
-### Other Integrations
+The same provider is available in Python through `HudiDataFusionDataSource`
when the `datafusion` extra of the
+`hudi` package is installed. Hudi tables can also be read by
[Daft](https://docs.daft.ai/en/stable/io/hudi/) and
+[Ray](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_hudi.html#ray.data.read_hudi).
-Hudi is also integrated with
+## Work with cloud storage
-- [Daft](https://docs.daft.ai/en/stable/io/hudi/)
--
[Ray](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_hudi.html#ray.data.read_hudi)
+Set the usual cloud credentials as environment variables (`AWS_*`, `AZURE_*`,
`GOOGLE_*`) and use an `s3://`,
+`az://`, or `gs://` base URI. Storage options can also be passed explicitly.
-### Work with cloud storage
-
-Ensure cloud storage credentials are set properly as environment variables,
e.g., `AWS_*`, `AZURE_*`, or `GOOGLE_*`.
-Relevant storage environment variables will then be picked up. The target
table's base uri with schemes such
-as `s3://`, `az://`, or `gs://` will be processed accordingly.
-
-Alternatively, you can pass the storage configuration as options via Table
APIs.
-
-#### Python
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-from hudi import HudiTableBuilder
+from hudi import HudiTable, HudiTableBuilder
-hudi_table = (
- HudiTableBuilder
- .from_base_uri("s3://bucket/trips_table")
- .with_option("aws_region", "us-west-2")
- .build()
-)
+table = HudiTable.create("s3://bucket/trips", record_key="id",
storage_options={"aws_region": "us-west-2"})
+
+reader =
HudiTableBuilder.from_base_uri("s3://bucket/trips").with_option("aws_region",
"us-west-2").build()
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
+use hudi::table::Table;
use hudi::table::builder::TableBuilder as HudiTableBuilder;
-async fn main() -> Result<()> {
- let hudi_table =
- HudiTableBuilder::from_base_uri("s3://bucket/trips_table")
- .with_option("aws_region", "us-west-2")
- .build().await?;
-}
+let mut table = Table::create("s3://bucket/trips")
+ .with_table_name("trips")
+ .with_record_key_fields(["id"])
+ .with_storage_option("aws_region", "us-west-2")
+ .create()
+ .await?;
+
+let reader = HudiTableBuilder::from_base_uri("s3://bucket/trips")
+ .with_option("aws_region", "us-west-2")
+ .build()
+ .await?;
```
+</TabItem>
+</Tabs>
+
+## Query engine integration
+
+For engines that plan their own scans, hudi-rs exposes the planning and
execution primitives directly.
+
+| Stage | API |
Description
|
+|-----------------|----------------------------------------------------|-----------------------------------------------------------------------------------------------------|
+| Query planning | `get_file_slices(options)` | The
file slices a read targets, dispatched on the query type (snapshot, time
travel, incremental). |
+| | `compute_table_stats(options)` |
Estimated row count and byte size for scan planning, from the metadata table
(snapshot only). |
+| Query execution | `create_file_group_reader_with_options()` | A
file group reader with the table's configs; pass the same read options used for
`get_file_slices`. |
+| | `read_file_slice()` / `read_file_slice_stream()` | Read
one file slice, merging base and log files per the table's merge strategy.
|
+
+## Resources
+
+- Rust API reference: [docs.rs/hudi](https://docs.rs/hudi), the `hudi` crate,
which re-exports
+ [`hudi-core`](https://docs.rs/hudi-core) and, with the `datafusion` feature,
+ [`hudi-datafusion`](https://docs.rs/hudi-datafusion).
+- Python API reference: the docstrings in the type stubs shipped with the
package,
+
[`hudi/_internal.pyi`](https://github.com/apache/hudi-rs/blob/main/python/hudi/_internal.pyi),
which IDEs and type checkers read.
+- The [hudi-rs README](https://github.com/apache/hudi-rs#quick-start): a
shorter quick start and the full read and write reference.
+- Runnable Rust example:
[`crates/core/examples/table_writes.rs`](https://github.com/apache/hudi-rs/blob/main/crates/core/examples/table_writes.rs).
+- Reader semantics:
[`docs/reader-spec.md`](https://github.com/apache/hudi-rs/blob/main/docs/reader-spec.md).
Writer design:
+
[`docs/writer-design.md`](https://github.com/apache/hudi-rs/blob/main/docs/writer-design.md).
+- Packages: [PyPI](https://pypi.org/project/hudi/) and
[crates.io](https://crates.io/crates/hudi). Source:
+ [github.com/apache/hudi-rs](https://github.com/apache/hudi-rs).
+- Hudi concepts (timeline, table types, query types): [overview](overview.mdx).
+
## Contributing
Check out the [contributing
guide](https://github.com/apache/hudi-rs/blob/main/CONTRIBUTING.md) for all the
details about making contributions to the project.
diff --git a/website/docs/use_cases.md b/website/docs/use_cases.md
index d0b6cecdee11..d89a3ee79acc 100644
--- a/website/docs/use_cases.md
+++ b/website/docs/use_cases.md
@@ -3,7 +3,7 @@ title: "Use Cases"
keywords: [ hudi, data ingestion, etl, real time, use cases]
summary: "Following are some sample use-cases for Hudi, which illustrate the
benefits in terms of faster processing & increased efficiency"
toc: true
-last_modified_at: 2019-12-30T15:59:57-04:00
+last_modified_at: 2026-10-29T00:00:00-07:00
---
Apache Hudi is a powerful [data lakehouse
platform](https://hudi.apache.org/blog/2021/07/21/streaming-data-lake-platform)
that shines in a variety of use cases due to its high-performance design, rich
feature set, and
@@ -135,4 +135,4 @@ For a worked example covering image embeddings, raw bytes,
and similarity search
- Incremental embedding pipelines: Hudi's [incremental
query](sql_queries.md#incremental-query) lets feature pipelines re-embed only
the rows that changed since a given commit, instead of reprocessing the full
batch.
- Multi-modal transactional guarantees: Vector updates, binary changes, and
metadata modifications commit atomically. A similarity query joining the
`VECTOR` column with the `BLOB` column never observes a state in which the
embedding refers to a deleted or stale binary asset.
- Automated table services: [File sizing](file_sizing.md),
[cleaning](cleaning.md), [compaction](compaction.md), and
[clustering](clustering.md) apply to tables that contain `VECTOR`, `BLOB`, and
`VARIANT` columns.
-- Open, multi-engine access: Hudi tables are written and read by [Apache
Spark](quick-start-guide.md) and [Apache Flink](flink-quick-start-guide.md),
and by the upcoming native [Python / Rust client
(hudi-rs)](python-rust-quick-start-guide.md) reading the underlying Parquet
representation directly.
+- Open, multi-engine access: Hudi tables are written and read by [Apache
Spark](quick-start-guide.md) and [Apache Flink](flink-quick-start-guide.md),
and by the native [Python / Rust client
(hudi-rs)](python-rust-quick-start-guide.md), which creates, upserts, deletes
and reads tables from a single process with no JVM.
diff --git
a/website/static/assets/images/docs/python-rust-quick-start/write-bytes.png
b/website/static/assets/images/docs/python-rust-quick-start/write-bytes.png
new file mode 100644
index 000000000000..1a73737a69a3
Binary files /dev/null and
b/website/static/assets/images/docs/python-rust-quick-start/write-bytes.png
differ
diff --git
a/website/static/assets/images/docs/python-rust-quick-start/write-latency.png
b/website/static/assets/images/docs/python-rust-quick-start/write-latency.png
new file mode 100644
index 000000000000..c20f765e4a36
Binary files /dev/null and
b/website/static/assets/images/docs/python-rust-quick-start/write-latency.png
differ
diff --git a/website/static/llms.txt b/website/static/llms.txt
index f407d633bc50..a4373070534d 100644
--- a/website/static/llms.txt
+++ b/website/static/llms.txt
@@ -1,6 +1,6 @@
# Apache Hudi
-> Apache Hudi is an open source data lakehouse platform that brings
database-like capabilities — ACID transactions, record-level upserts and
deletes, indexing, change data capture and incremental processing — to data
stored on cloud object storage or HDFS. Hudi tables can be written and queried
from Apache Spark, Apache Flink, Presto, Trino, Hive and more, and can
interoperate with Apache Iceberg and Delta Lake metadata via Apache XTable.
+> Apache Hudi is an open source data lakehouse platform that brings
database-like capabilities — ACID transactions, record-level upserts and
deletes, indexing, change data capture and incremental processing — to data
stored on cloud object storage or HDFS. Hudi tables can be written and queried
from Python and Rust without a JVM (hudi-rs: `pip install hudi` / `cargo add
hudi`), from Apache Spark, Apache Flink, Presto, Trino, Hive and more, and can
interoperate with Apache Iceberg and Del [...]
Hudi (pronounced "hoodie") pioneered transactional capabilities on top of data
lakes. Hudi lakehouses were in production at Uber and other places in 2017,
years before the term was coined in the industry.
It manages tables as a timeline of actions over open file formats (Apache
Parquet, Apache ORC, Lance file format, Apache Avro), supports Copy-on-Write
and Merge-on-Read storage, ships a multi-modal indexing subsystem
@@ -9,6 +9,7 @@ It can also synchronize your data to half dozen data catalogs
to keep table cons
## Documentation
+- [Python/Rust Quick
Start](https://hudi.apache.org/docs/python-rust-quick-start-guide): `pip
install hudi` or `cargo add hudi`, then create a table, upsert, update and
delete rows by record key, and query the result with PyArrow, DuckDB, Polars or
DataFusion. Single process, no JVM, no Spark; the record-level index makes
upserts and deletes cost index lookups instead of table scans. Its single-node
benchmark found hudi-rs upserts 1.2 to 12 times faster than delta-rs on all
five workload [...]
- [Spark Quick Start](https://hudi.apache.org/docs/quick-start-guide): Create,
write and query your first Hudi table with Spark (Scala, Python, SQL).
- [Flink Quick Start](https://hudi.apache.org/docs/flink-quick-start-guide):
Stream data into Hudi tables with Apache Flink.
- [Timeline](https://hudi.apache.org/docs/timeline): How Hudi organizes a
table as an ordered log of commits — the core of its ACID design.
@@ -88,6 +89,19 @@ It can also synchronize your data to half dozen data
catalogs to keep table cons
- [Building an ExaByte-level Data Lake Using Apache Hudi at
ByteDance](https://hudi.apache.org/blog/2021/09/01/building-eb-level-data-lake-using-hudi-at-bytedance):
Exabyte-scale Hudi deployment at ByteDance.
- [Modernizing Data Infrastructure at Peloton with
Hudi](https://hudi.apache.org/blog/2025/07/15/modernizing-datainfra-peloton-hudi):
CDC and lakehouse modernization at Peloton.
+## hudi-rs (Python and Rust, no JVM)
+
+- [hudi-rs on GitHub](https://github.com/apache/hudi-rs): Native Rust
implementation of Hudi with Python and C++ bindings. Rust and Python read and
write Hudi tables (create, append, upsert, update, delete, overwrite, and
experimental merge-on-read compaction) with the metadata table and record-level
index maintained on every commit; storage-compatible with the Spark writer.
+- [hudi on PyPI](https://pypi.org/project/hudi/): The Python package.
`HudiTable.create(path, record_key=...)`, `table.upsert(arrow_table)`,
`table.delete("id = ...")`, `table.read()`; accepts PyArrow, Polars and pandas
input.
+- [hudi on crates.io](https://crates.io/crates/hudi): The Rust crate;
`Table::create(...)`, `upsert`, `delete`, `read`, plus a DataFusion table
provider behind the `datafusion` feature.
+- [Rust API reference](https://docs.rs/hudi): docs.rs for the `hudi` crate,
which re-exports `hudi-core` and, with the `datafusion` feature,
`hudi-datafusion`.
+- [Python API
reference](https://github.com/apache/hudi-rs/blob/main/python/hudi/_internal.pyi):
The type stubs shipped with the `hudi` package, whose docstrings document the
Python API.
+- [hudi-rs README quick start](https://github.com/apache/hudi-rs#quick-start):
A short Python and Rust quick start, plus the full read and write reference.
+- [Runnable Rust
example](https://github.com/apache/hudi-rs/blob/main/crates/core/examples/table_writes.rs):
A tour of the Rust write API (create, append, upsert, update, delete,
overwrite) and reads.
+- [hudi-rs reader
semantics](https://github.com/apache/hudi-rs/blob/main/docs/reader-spec.md):
Read options, query types, filters and the guarantees hudi-rs readers give.
+- [hudi-rs writer
design](https://github.com/apache/hudi-rs/blob/main/docs/writer-design.md): How
hudi-rs writes commits, the metadata table and the record-level index.
+- [Upsert
benchmark](https://github.com/apache/hudi-rs/tree/main/benchmark/upsert):
Reproducible single-node upsert and delete workloads for hudi-rs (retrieval
corpus re-sync, training-set curation, random keys, NYC taxi, TPC-DS), with
correctness checked against a Parquet oracle; the quick start page compares the
results with delta-rs and PyIceberg.
+
## Optional
- [Releases & Downloads](https://hudi.apache.org/releases/download): All
Apache Hudi releases.
diff --git
a/website/versioned_docs/version-1.2.0/python-rust-quick-start-guide.md
b/website/versioned_docs/version-1.2.0/python-rust-quick-start-guide.md
index bddded7d8f0a..140efe2fa8e5 100644
--- a/website/versioned_docs/version-1.2.0/python-rust-quick-start-guide.md
+++ b/website/versioned_docs/version-1.2.0/python-rust-quick-start-guide.md
@@ -1,184 +1,347 @@
---
title: "Python/Rust Quick Start"
toc: true
-last_modified_at: 2024-11-28T12:53:57+08:00
+last_modified_at: 2026-11-03T00:00:00-08:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-This guide will help you get started with
[Hudi-rs](https://github.com/apache/hudi-rs), the native Rust implementation
for Apache Hudi with Python bindings. Learn how to install, set up, and perform
basic operations using both Python and Rust interfaces.
+[hudi-rs](https://github.com/apache/hudi-rs) is the native Rust implementation
of Apache Hudi with Python bindings.
+It creates Hudi tables and runs upserts, updates, and deletes on them from a
single process, with no JVM and no
+Spark. Every write maintains the table's metadata table and record-level
index, so updating or deleting a handful
+of rows in a large table costs a few index lookups and the rewrite of the file
groups that hold those rows, not a
+scan of the table.
+
+This page gets you from `pip install` to a table you can upsert into, delete
from, and read incrementally, then
+shows how to query it from Python, Rust, DuckDB, Polars, and DataFusion.
## Installation
```bash
-# Python
+# Python (3.10+)
pip install hudi
# Rust
cargo add hudi
+cargo add arrow@59
+cargo add tokio@1 --features macros,rt-multi-thread
```
-## Usage Examples
+## Write a table in Python
-> [!NOTE]
-> These examples expect a Hudi table exists at `/tmp/trips_table`, created
using
-> the [quick start guide](quick-start-guide.md).
+A retrieval or training pipeline end to end: a table of document chunks with
their embeddings, keyed by
+`chunk_id`. Ingest, re-sync the documents that changed, drop the ones that
were removed, and read back only what
+changed since the last run.
-### Snapshot Query
+```python
+import tempfile
-Snapshot query reads the latest version of the data from the table. The table
API also accepts partition filters.
+import numpy as np
+import pyarrow as pa
+from hudi import HudiTable, HudiReadOptions, HudiQueryType
+
+def chunks(doc_ids, version):
+ rows = [(f"{d}#{i}", d, i, f"chunk {i} of {d}, version {version}") for d
in doc_ids for i in range(4)]
+ embeddings = np.random.default_rng(version).random((len(rows), 384),
dtype=np.float32)
+ return pa.table({
+ "chunk_id": [r[0] for r in rows],
+ "doc_id": [r[1] for r in rows],
+ "pos": pa.array([r[2] for r in rows], pa.int32()),
+ "text": [r[3] for r in rows],
+ "embedding": pa.array(embeddings.tolist(), pa.list_(pa.float32())),
+ "updated_at": pa.array([version] * len(rows), pa.int64()),
+ })
+
+table = HudiTable.create(
+ f"{tempfile.mkdtemp()}/contract_chunks",
+ record_key="chunk_id", # the record-level index is built on this
key
+ ordering_fields="updated_at", # the later version wins when a key
arrives twice
+ record_index_enabled=True, # on by default; locates keys without
scanning files
+)
-#### Python
+base = table.append(chunks(["doc1", "doc2", "doc3"], version=1))
-```python
-from hudi import HudiTableBuilder
-import pyarrow as pa
+# doc2 changed: re-embed its chunks and upsert them. The index finds the file
+# groups holding these keys; nothing else in the table is read.
+result = table.upsert(chunks(["doc2"], version=2))
+print(result.num_updates, result.num_inserts) # 4 0
-hudi_table = HudiTableBuilder.from_base_uri("/tmp/trips_table").build()
-batches = hudi_table.read_snapshot(filters=[("city", "=", "san_francisco")])
+# doc3 was removed: delete its chunks.
+table.delete("doc_id = 'doc3'")
-# convert to PyArrow table
-arrow_table = pa.Table.from_batches(batches)
-result = arrow_table.select(["rider", "city", "ts", "fare"])
-print(result)
+# Train on what changed after the initial load, not on the whole table.
+changed = table.read(
+ HudiReadOptions()
+ .with_query_type(HudiQueryType.Incremental)
+ .with_start_timestamp(base.completion_time)
+)
+print(pa.Table.from_batches(changed).num_rows) # 4, the re-embedded chunks
of doc2
+
+# Query the current snapshot with DuckDB (or Polars, pandas, anything Arrow).
+import duckdb
+snapshot = pa.Table.from_batches(table.read())
+duckdb.sql("SELECT doc_id, count(*) AS chunks FROM snapshot GROUP BY doc_id
ORDER BY doc_id").show()
```
-#### Rust
+`append`, `upsert`, and `overwrite` accept a `pyarrow.Table`,
`pyarrow.RecordBatch`, `pyarrow.RecordBatchReader`,
+or any object that implements the Arrow PyCapsule interface, so a Polars
DataFrame or a pandas 2.2+ DataFrame can
+be passed directly. Embeddings may arrive as `list<float32>` or
`fixed_size_list<float32>`; both are stored as lists.
+
+`update` sets columns by predicate and `delete_keys` deletes by key:
+
+```python
+table.update("doc_id = 'doc1'", {"text": "redacted"})
+table.delete_keys(["doc1#0", "doc1#1"])
+```
+
+## Write a table in Rust
+
+The same steps with the `hudi` crate. Batches are Arrow `RecordBatch`es.
```rust
+use arrow::array::{ArrayRef, Float64Array, Int64Array, RecordBatch,
StringArray};
+use arrow::datatypes::{DataType, Field, Schema};
+use hudi::config::table::TableTypeValue;
use hudi::error::Result;
-use hudi::table::builder::TableBuilder as HudiTableBuilder;
-use arrow::compute::concat_batches;
+use hudi::table::{ReadOptions, Table};
+use std::sync::Arc;
+
+fn trips(rows: &[(&str, &str, i64, f64)]) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("id", DataType::Utf8, false),
+ Field::new("city", DataType::Utf8, false),
+ Field::new("ts", DataType::Int64, false),
+ Field::new("fare", DataType::Float64, false),
+ ]));
+ RecordBatch::try_new(schema, vec![
+ Arc::new(StringArray::from(rows.iter().map(|r|
r.0).collect::<Vec<_>>())) as ArrayRef,
+ Arc::new(StringArray::from(rows.iter().map(|r|
r.1).collect::<Vec<_>>())),
+ Arc::new(Int64Array::from(rows.iter().map(|r|
r.2).collect::<Vec<_>>())),
+ Arc::new(Float64Array::from(rows.iter().map(|r|
r.3).collect::<Vec<_>>())),
+ ]).unwrap()
+}
#[tokio::main]
async fn main() -> Result<()> {
- let hudi_table =
HudiTableBuilder::from_base_uri("/tmp/trips_table").build().await?;
- let batches = hudi_table.read_snapshot(&[("city", "=",
"san_francisco")]).await?;
- let batch = concat_batches(&batches[0].schema(), &batches)?;
- let columns = vec!["rider", "city", "ts", "fare"];
- for col_name in columns {
- let idx = batch.schema().index_of(col_name).unwrap();
- println!("{}: {}", col_name, batch.column(idx));
- }
+ let mut table = Table::create("/tmp/trips")
+ .with_table_name("trips")
+ .with_table_type(TableTypeValue::CopyOnWrite) // or MergeOnRead
+ .with_record_key_fields(["id"])
+ .with_partition_fields(["city"])
+ .with_ordering_fields(["ts"])
+ .create()
+ .await?;
+
+ table.append([trips(&[("t1", "sf", 1, 30.0), ("t2", "sf", 1, 45.0), ("t3",
"nyc", 1, 20.0)])]).await?;
+
+ let result = table.upsert([trips(&[("t2", "sf", 2, 50.0), ("t4", "nyc", 2,
15.0)])]).await?;
+ println!(
+ "requested_time={} updates={} inserts={}",
+ result.requested_time.as_deref().unwrap_or("none"),
+ result.num_updates,
+ result.num_inserts
+ );
+
+ table.delete("id = 't1'").await?;
+
+ let batches = table.read(&ReadOptions::new()).await?;
+ println!("{} rows", batches.iter().map(|b| b.num_rows()).sum::<usize>());
Ok(())
}
```
-To run read-optimized (RO) query on Merge-on-Read (MOR) tables, set
`hoodie.read.use.read_optimized.mode` when creating the table.
+Both writers produce the same on-disk layout as the Spark writer.
-#### Python
+## Compact a Merge-on-Read table
+
+On a Merge-on-Read table (`table_type="mor"` in Python,
`TableTypeValue::MergeOnRead` in Rust), upserts, updates,
+and deletes append log files. Compaction folds them into new base files under
one compaction commit;
+rows keep their commit time, so incremental reads see no new rows from it. Run
it on demand, or set
+`hoodie.compact.inline=true` (in the write call) or
`hoodie.datasource.compaction.async.enable=true` (on a
+background task) to compact once `hoodie.compact.inline.max.delta.commits`
(default 5) delta commits accumulate.
+The metadata table compacts automatically.
+
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-hudi_table = (
- HudiTableBuilder
- .from_base_uri("/tmp/trips_table")
- .with_option("hoodie.read.use.read_optimized.mode", "true")
- .build()
+mor = HudiTable.create(
+ f"{tempfile.mkdtemp()}/contract_chunks_mor",
+ record_key="chunk_id",
+ ordering_fields="updated_at",
+ table_type="mor",
+ # Optional: also compact in the write call every 5 delta commits
+ hudi_options={"hoodie.compact.inline": "true"},
)
+mor.append(chunks(["doc1", "doc2"], version=1))
+mor.upsert(chunks(["doc2"], version=2)) # the updates go to log files
+
+result = mor.compact()
+print(result.num_file_groups, result.num_log_files)
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
-let hudi_table =
- HudiTableBuilder::from_base_uri("/tmp/trips_table")
- .with_option("hoodie.read.use.read_optimized.mode", "true")
- .build().await?;
+let mut mor = Table::create("/tmp/trips_mor")
+ .with_table_name("trips_mor")
+ .with_table_type(TableTypeValue::MergeOnRead)
+ .with_record_key_fields(["id"])
+ .with_partition_fields(["city"])
+ .with_ordering_fields(["ts"])
+ // Optional: also compact in the write call every 5 delta commits
+ .with_option("hoodie.compact.inline", "true")
+ .create()
+ .await?;
+mor.append([trips(&[("t1", "sf", 1, 30.0), ("t2", "nyc", 1, 20.0)])]).await?;
+mor.upsert([trips(&[("t1", "sf", 2, 31.0)])]).await?; // the update goes to a
log file
+
+let result = mor.compact().await?;
+println!("{} {}", result.num_file_groups, result.num_log_files); // 1 1
```
-> [!NOTE]
-> Currently reading MOR tables is limited to tables with Parquet data blocks.
+</TabItem>
+</Tabs>
-### Time-Travel Query
+## Why upserts and deletes are fast: the record-level index
-Time-travel query reads the data at a specific timestamp from the table. The
table API also accepts partition filters.
+A client with no index answers "which file holds key `doc2#1`?" by scanning
every file, or at best every
+file whose min/max statistics admit the key. With keys like `<doc>#<pos>` or
UUIDs that is the whole table on every
+sync, however few documents changed.
-#### Python
+Hudi keeps a [record-level index](indexes.md#record-index) in the table's
metadata table: a persistent map from record
+key to file group, sharded by key hash. hudi-rs writes it on `create` and
updates it on every commit. A sync that
+touches 10,000 chunks in a table of millions costs an index lookup of 10,000
keys, which names the file groups to
+rewrite (or, on a Merge-on-Read table, to append a log block to). The rest of
the table is never read.
-```python
-batches = (
- hudi_table
- .read_snapshot_as_of("20241231123456789", filters=[("city", "=",
"san_francisco")])
-)
-```
+## Use cases and numbers
-#### Rust
+The
[`benchmark/upsert`](https://github.com/apache/hudi-rs/tree/543642771a1440f8470053831463052e190b3923/benchmark/upsert)
harness in the hudi-rs
+repository runs five single-node workloads against hudi-rs, each a bulk load
followed by five upsert commits and five
+delete commits; the numbers below also include delta-rs and PyIceberg run
through the same commits on the same
+machine. Every end state is checked against a plain Parquet copy of the
expected table, and the machine's load, memory,
+and disk are recorded next to every timing. Each scenario below links to its
definition so the numbers can be
+reproduced on your own machine.
-```rust
-let batches =
- hudi_table
- .read_snapshot_as_of("20241231123456789", &[("city", "=",
"san_francisco")]).await?;
-```
+| Scenario | Models | Table | Each commit |
+|---|---|---|---|
+|
[`rag-chunks`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#rag-chunks)
| Re-syncing a
[RAG](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) knowledge
base as its sources change | 1M synthetic chunks with 384-float embeddings from
100,000 documents, keyed by `chunk_id` | re-ingest 1,000 changed documents
(10,000 chunks), delete 100 removed documents |
+|
[`training-set`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#training-set)
| Curating a fine-tuning set between training runs | 1M synthetic examples
keyed by `example_id` | upsert 10,000 corrected examples, delete 1,000 rejected
ones, then an incremental read of the changes and a time-travel read of the
first version |
+|
[`uuid`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#uuid)
| Random updates to a dimension table, with changed keys spread across every
file | 10M synthetic rows with random UUID keys | upsert 10,000 existing keys
plus 1,000 new, delete 1,000 |
+|
[`nyc-taxi`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#nyc-taxi)
| Correcting a public dataset in place | [NYC TLC yellow taxi
trips](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page), January
2022, 2.5M rows | upsert 1% of trips with corrected fares, delete 0.1% |
+|
[`tpcds-web-returns`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#tpcds-web-returns)
| A standard warehouse fact table | [TPC-DS](https://www.tpc.org/tpcds/)
`web_returns` at scale factor 10, 0.7M rows, composite key | upsert 1% matched
plus 10% new rows, delete 5% |
+
+{/* BENCHMARK_TABLE_START */}
+**Why hudi-rs is faster on upserts and deletes.** Low write amplification and
the record-level index: a Merge-on-Read
+commit appends only the changed records to log files, and the record-level
index points each key at its file group, so
+nothing else in the table is read or rewritten. Neither delta-rs nor PyIceberg
writes deletion vectors in their latest
+releases ([delta-rs
1.6.6](https://github.com/delta-io/delta-rs/releases/tag/python-v1.6.6),
+[PyIceberg
0.12.0](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.12.0)),
and neither keeps a
+record-level index, so every update or delete rewrites each data file it
touches. The changed keys here are spread
+across the table, so each Copy-on-Write commit rewrites nearly all of it: per
upsert commit, hudi-rs wrote 19 MiB on
+`rag-chunks` where delta-rs rewrote 1,689 MiB and PyIceberg 1,432 MiB, and 2
MiB on `uuid` against 671 MiB and 388 MiB.
-### Incremental Query
+<div style={{ textAlign: 'center' }}>
+ <img src="/assets/images/docs/python-rust-quick-start/write-latency.png"
alt="Seconds per upsert and delete commit on five workloads: hudi-rs 0.6.0
Merge-on-Read with the record-level index against delta-rs and PyIceberg"
width="1000"/>
+</div>
-Incremental query reads the changed data from the table for a given time range.
+<div style={{ textAlign: 'center' }}>
+ <img src="/assets/images/docs/python-rust-quick-start/write-bytes.png"
alt="MiB written per upsert and delete commit on the same five workloads"
width="1000"/>
+</div>
-#### Python
+Loading the 10M-row table in Hudi takes 5.7 s, about 2.4 s of which builds the
metadata table and its record-level
+index, an investment that pays off in the latency of every later upsert and
delete.
+{/* BENCHMARK_TABLE_END */}
+
+**Reproduce it.** [This
guide](https://gist.github.com/yihua/66ad80224f8e50020025dc76d138dd0a) pins the
hudi-rs
+commit, every package version and the delta-rs and PyIceberg cases, and gives
the exact commands.
+
+## Read a table
+
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-# read the records between t1 (exclusive) and t2 (inclusive)
-batches = hudi_table.read_incremental_records(t1, t2)
+from hudi import HudiReadOptions, HudiTableBuilder
+import pyarrow as pa
+
+table = HudiTableBuilder.from_base_uri("/tmp/trips").build()
+
+# Snapshot, with partition and row filters pushed down
+batches = table.read(HudiReadOptions(filters=[("city", "=", "sf")]))
+print(pa.Table.from_batches(batches))
+
+# Time travel
+batches =
table.read(HudiReadOptions().with_as_of_timestamp("20261029120000000"))
+
+# Incremental: rows from commits that completed after a point in time, such as
+# the completion_time of an earlier write
+from hudi import HudiQueryType
+since = "20261029120000000"
+batches =
table.read(HudiReadOptions().with_query_type(HudiQueryType.Incremental).with_start_timestamp(since))
-# read the records after t1
-batches = hudi_table.read_incremental_records(t1)
+# Stream batches instead of materializing the result
+for batch in table.read_stream(HudiReadOptions(projection=["id",
"fare"]).with_batch_size(4096)):
+ print(batch.num_rows)
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
-// read the records between t1 (exclusive) and t2 (inclusive)
-let batches = hudi_table.read_incremental_records(t1, Some(t2)).await?;
+use hudi::table::builder::TableBuilder as HudiTableBuilder;
+use hudi::table::{QueryType, ReadOptions};
-// read the records after t1
-let batches = hudi_table.read_incremental_records(t1, None).await?;
-```
+let table = HudiTableBuilder::from_base_uri("/tmp/trips").build().await?;
-> [!NOTE]
-> Currently the only supported format for the timestamp arguments is Hudi
Timeline format: `yyyyMMddHHmmssSSS` or `yyyyMMddHHmmss`.
+// Snapshot, with partition and row filters pushed down
+let batches = table.read(&ReadOptions::new().with_filters([("city", "=",
"sf")])?).await?;
-## Query Engine Integration
+// Time travel
+let batches =
table.read(&ReadOptions::new().with_as_of_timestamp("20261029120000000")).await?;
-Hudi-rs provides APIs to support integration with query engines. The sections
below highlight some commonly used APIs.
+// Incremental: rows from commits that completed after a point in time, such as
+// the completion_time of an earlier write
+let since = "20261029120000000";
+let options =
ReadOptions::new().with_query_type(QueryType::Incremental).with_start_timestamp(since);
+let batches = table.read(&options).await?;
+```
-### Table API
+</TabItem>
+</Tabs>
-Create a Hudi table instance using its constructor or the `TableBuilder` API.
+Timestamps accept the Hudi timeline format (`yyyyMMddHHmmssSSS`), Unix epoch
in seconds through nanoseconds, and
+RFC 3339 with a timezone offset.
-| Stage | API | Description
|
-|-----------------|-------------------------------------------|--------------------------------------------------------------------------------|
-| Query planning | `get_file_slices()` | For snapshot
query, get a list of file slices. |
-| | `get_file_slices_splits()` | For snapshot
query, get a list of file slices in splits. |
-| | `get_file_slices_as_of()` | For
time-travel query, get a list of file slices at a given time. |
-| | `get_file_slices_splits_as_of()` | For
time-travel query, get a list of file slices in splits at a given time. |
-| | `get_file_slices_between()` | For
incremental query, get a list of changed file slices between a time range. |
-| Query execution | `create_file_group_reader_with_options()` | Create a file
group reader instance with the table instance's configs. |
+To run a read-optimized query on a Merge-on-Read table (base files only, no
log merge), set
+`hoodie.read.use.read_optimized.mode` to `true` in the read options.
-### File Group API
+## Query with DuckDB, Polars, and DataFusion
-Create a Hudi file group reader instance using its constructor or the Hudi
table API `create_file_group_reader_with_options()`.
+hudi-rs returns Arrow, so anything that consumes Arrow can query a Hudi table
in Python.
-| Stage | API | Description
|
-|-----------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Query execution | `read_file_slice()` | Read records from
a given file slice; based on the configs, read records from only base file, or
from base file and log files, and merge records based on the configured
strategy. |
+```python
+import duckdb, polars as pl, pyarrow as pa
+from hudi import HudiTableBuilder
+trips =
pa.Table.from_batches(HudiTableBuilder.from_base_uri("/tmp/trips").build().read())
-### Apache DataFusion
+duckdb.sql("SELECT city, avg(fare) FROM trips GROUP BY city").show()
+print(pl.from_arrow(trips).group_by("city").agg(pl.col("fare").mean()))
+```
-Enabling the `hudi` crate with `datafusion` feature will provide a
[DataFusion](https://datafusion.apache.org/)
-extension to query Hudi tables.
+In Rust, the `datafusion` feature of the `hudi` crate registers a Hudi table
as a DataFusion table provider.
<details>
-<summary>Add crate hudi with datafusion feature to your application to query a
Hudi table.</summary>
+<summary>Add the hudi crate with the datafusion feature to your
application.</summary>
```shell
cargo new my_project --bin && cd my_project
-cargo add tokio@1 datafusion@43
+cargo add tokio@1 datafusion@55 # the DataFusion major hudi 0.6 builds against
cargo add hudi --features datafusion
```
-Update `src/main.rs` with the code snippet below then `cargo run`.
-
</details>
```rust
@@ -191,57 +354,83 @@ use hudi::HudiDataSource;
#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
- let hudi = HudiDataSource::new_with_options(
- "/tmp/trips_table",
- [("hoodie.read.input.partitions", "5")]).await?;
- ctx.register_table("trips_table", Arc::new(hudi))?;
- let df: DataFrame = ctx.sql("SELECT * from trips_table where city =
'san_francisco'").await?;
+ let hudi = HudiDataSource::new_with_options("/tmp/trips",
[("hoodie.read.input.partitions", "5")]).await?;
+ ctx.register_table("trips", Arc::new(hudi))?;
+ let df: DataFrame = ctx.sql("SELECT * FROM trips WHERE city =
'sf'").await?;
df.show().await?;
Ok(())
}
```
-### Other Integrations
+The same provider is available in Python through `HudiDataFusionDataSource`
when the `datafusion` extra of the
+`hudi` package is installed. Hudi tables can also be read by
[Daft](https://docs.daft.ai/en/stable/io/hudi/) and
+[Ray](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_hudi.html#ray.data.read_hudi).
-Hudi is also integrated with
+## Work with cloud storage
-- [Daft](https://docs.daft.ai/en/stable/io/hudi/)
--
[Ray](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_hudi.html#ray.data.read_hudi)
+Set the usual cloud credentials as environment variables (`AWS_*`, `AZURE_*`,
`GOOGLE_*`) and use an `s3://`,
+`az://`, or `gs://` base URI. Storage options can also be passed explicitly.
-### Work with cloud storage
-
-Ensure cloud storage credentials are set properly as environment variables,
e.g., `AWS_*`, `AZURE_*`, or `GOOGLE_*`.
-Relevant storage environment variables will then be picked up. The target
table's base uri with schemes such
-as `s3://`, `az://`, or `gs://` will be processed accordingly.
-
-Alternatively, you can pass the storage configuration as options via Table
APIs.
-
-#### Python
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-from hudi import HudiTableBuilder
+from hudi import HudiTable, HudiTableBuilder
-hudi_table = (
- HudiTableBuilder
- .from_base_uri("s3://bucket/trips_table")
- .with_option("aws_region", "us-west-2")
- .build()
-)
+table = HudiTable.create("s3://bucket/trips", record_key="id",
storage_options={"aws_region": "us-west-2"})
+
+reader =
HudiTableBuilder.from_base_uri("s3://bucket/trips").with_option("aws_region",
"us-west-2").build()
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
+use hudi::table::Table;
use hudi::table::builder::TableBuilder as HudiTableBuilder;
-async fn main() -> Result<()> {
- let hudi_table =
- HudiTableBuilder::from_base_uri("s3://bucket/trips_table")
- .with_option("aws_region", "us-west-2")
- .build().await?;
-}
+let mut table = Table::create("s3://bucket/trips")
+ .with_table_name("trips")
+ .with_record_key_fields(["id"])
+ .with_storage_option("aws_region", "us-west-2")
+ .create()
+ .await?;
+
+let reader = HudiTableBuilder::from_base_uri("s3://bucket/trips")
+ .with_option("aws_region", "us-west-2")
+ .build()
+ .await?;
```
+</TabItem>
+</Tabs>
+
+## Query engine integration
+
+For engines that plan their own scans, hudi-rs exposes the planning and
execution primitives directly.
+
+| Stage | API |
Description
|
+|-----------------|----------------------------------------------------|-----------------------------------------------------------------------------------------------------|
+| Query planning | `get_file_slices(options)` | The
file slices a read targets, dispatched on the query type (snapshot, time
travel, incremental). |
+| | `compute_table_stats(options)` |
Estimated row count and byte size for scan planning, from the metadata table
(snapshot only). |
+| Query execution | `create_file_group_reader_with_options()` | A
file group reader with the table's configs; pass the same read options used for
`get_file_slices`. |
+| | `read_file_slice()` / `read_file_slice_stream()` | Read
one file slice, merging base and log files per the table's merge strategy.
|
+
+## Resources
+
+- Rust API reference: [docs.rs/hudi](https://docs.rs/hudi), the `hudi` crate,
which re-exports
+ [`hudi-core`](https://docs.rs/hudi-core) and, with the `datafusion` feature,
+ [`hudi-datafusion`](https://docs.rs/hudi-datafusion).
+- Python API reference: the docstrings in the type stubs shipped with the
package,
+
[`hudi/_internal.pyi`](https://github.com/apache/hudi-rs/blob/main/python/hudi/_internal.pyi),
which IDEs and type checkers read.
+- The [hudi-rs README](https://github.com/apache/hudi-rs#quick-start): a
shorter quick start and the full read and write reference.
+- Runnable Rust example:
[`crates/core/examples/table_writes.rs`](https://github.com/apache/hudi-rs/blob/main/crates/core/examples/table_writes.rs).
+- Reader semantics:
[`docs/reader-spec.md`](https://github.com/apache/hudi-rs/blob/main/docs/reader-spec.md).
Writer design:
+
[`docs/writer-design.md`](https://github.com/apache/hudi-rs/blob/main/docs/writer-design.md).
+- Packages: [PyPI](https://pypi.org/project/hudi/) and
[crates.io](https://crates.io/crates/hudi). Source:
+ [github.com/apache/hudi-rs](https://github.com/apache/hudi-rs).
+- Hudi concepts (timeline, table types, query types): [overview](overview.mdx).
+
## Contributing
Check out the [contributing
guide](https://github.com/apache/hudi-rs/blob/main/CONTRIBUTING.md) for all the
details about making contributions to the project.
diff --git a/website/versioned_docs/version-1.2.0/use_cases.md
b/website/versioned_docs/version-1.2.0/use_cases.md
index d0b6cecdee11..d89a3ee79acc 100644
--- a/website/versioned_docs/version-1.2.0/use_cases.md
+++ b/website/versioned_docs/version-1.2.0/use_cases.md
@@ -3,7 +3,7 @@ title: "Use Cases"
keywords: [ hudi, data ingestion, etl, real time, use cases]
summary: "Following are some sample use-cases for Hudi, which illustrate the
benefits in terms of faster processing & increased efficiency"
toc: true
-last_modified_at: 2019-12-30T15:59:57-04:00
+last_modified_at: 2026-10-29T00:00:00-07:00
---
Apache Hudi is a powerful [data lakehouse
platform](https://hudi.apache.org/blog/2021/07/21/streaming-data-lake-platform)
that shines in a variety of use cases due to its high-performance design, rich
feature set, and
@@ -135,4 +135,4 @@ For a worked example covering image embeddings, raw bytes,
and similarity search
- Incremental embedding pipelines: Hudi's [incremental
query](sql_queries.md#incremental-query) lets feature pipelines re-embed only
the rows that changed since a given commit, instead of reprocessing the full
batch.
- Multi-modal transactional guarantees: Vector updates, binary changes, and
metadata modifications commit atomically. A similarity query joining the
`VECTOR` column with the `BLOB` column never observes a state in which the
embedding refers to a deleted or stale binary asset.
- Automated table services: [File sizing](file_sizing.md),
[cleaning](cleaning.md), [compaction](compaction.md), and
[clustering](clustering.md) apply to tables that contain `VECTOR`, `BLOB`, and
`VARIANT` columns.
-- Open, multi-engine access: Hudi tables are written and read by [Apache
Spark](quick-start-guide.md) and [Apache Flink](flink-quick-start-guide.md),
and by the upcoming native [Python / Rust client
(hudi-rs)](python-rust-quick-start-guide.md) reading the underlying Parquet
representation directly.
+- Open, multi-engine access: Hudi tables are written and read by [Apache
Spark](quick-start-guide.md) and [Apache Flink](flink-quick-start-guide.md),
and by the native [Python / Rust client
(hudi-rs)](python-rust-quick-start-guide.md), which creates, upserts, deletes
and reads tables from a single process with no JVM.
diff --git
a/website/versioned_docs/version-1.2.1/python-rust-quick-start-guide.md
b/website/versioned_docs/version-1.2.1/python-rust-quick-start-guide.md
index bddded7d8f0a..140efe2fa8e5 100644
--- a/website/versioned_docs/version-1.2.1/python-rust-quick-start-guide.md
+++ b/website/versioned_docs/version-1.2.1/python-rust-quick-start-guide.md
@@ -1,184 +1,347 @@
---
title: "Python/Rust Quick Start"
toc: true
-last_modified_at: 2024-11-28T12:53:57+08:00
+last_modified_at: 2026-11-03T00:00:00-08:00
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
-This guide will help you get started with
[Hudi-rs](https://github.com/apache/hudi-rs), the native Rust implementation
for Apache Hudi with Python bindings. Learn how to install, set up, and perform
basic operations using both Python and Rust interfaces.
+[hudi-rs](https://github.com/apache/hudi-rs) is the native Rust implementation
of Apache Hudi with Python bindings.
+It creates Hudi tables and runs upserts, updates, and deletes on them from a
single process, with no JVM and no
+Spark. Every write maintains the table's metadata table and record-level
index, so updating or deleting a handful
+of rows in a large table costs a few index lookups and the rewrite of the file
groups that hold those rows, not a
+scan of the table.
+
+This page gets you from `pip install` to a table you can upsert into, delete
from, and read incrementally, then
+shows how to query it from Python, Rust, DuckDB, Polars, and DataFusion.
## Installation
```bash
-# Python
+# Python (3.10+)
pip install hudi
# Rust
cargo add hudi
+cargo add arrow@59
+cargo add tokio@1 --features macros,rt-multi-thread
```
-## Usage Examples
+## Write a table in Python
-> [!NOTE]
-> These examples expect a Hudi table exists at `/tmp/trips_table`, created
using
-> the [quick start guide](quick-start-guide.md).
+A retrieval or training pipeline end to end: a table of document chunks with
their embeddings, keyed by
+`chunk_id`. Ingest, re-sync the documents that changed, drop the ones that
were removed, and read back only what
+changed since the last run.
-### Snapshot Query
+```python
+import tempfile
-Snapshot query reads the latest version of the data from the table. The table
API also accepts partition filters.
+import numpy as np
+import pyarrow as pa
+from hudi import HudiTable, HudiReadOptions, HudiQueryType
+
+def chunks(doc_ids, version):
+ rows = [(f"{d}#{i}", d, i, f"chunk {i} of {d}, version {version}") for d
in doc_ids for i in range(4)]
+ embeddings = np.random.default_rng(version).random((len(rows), 384),
dtype=np.float32)
+ return pa.table({
+ "chunk_id": [r[0] for r in rows],
+ "doc_id": [r[1] for r in rows],
+ "pos": pa.array([r[2] for r in rows], pa.int32()),
+ "text": [r[3] for r in rows],
+ "embedding": pa.array(embeddings.tolist(), pa.list_(pa.float32())),
+ "updated_at": pa.array([version] * len(rows), pa.int64()),
+ })
+
+table = HudiTable.create(
+ f"{tempfile.mkdtemp()}/contract_chunks",
+ record_key="chunk_id", # the record-level index is built on this
key
+ ordering_fields="updated_at", # the later version wins when a key
arrives twice
+ record_index_enabled=True, # on by default; locates keys without
scanning files
+)
-#### Python
+base = table.append(chunks(["doc1", "doc2", "doc3"], version=1))
-```python
-from hudi import HudiTableBuilder
-import pyarrow as pa
+# doc2 changed: re-embed its chunks and upsert them. The index finds the file
+# groups holding these keys; nothing else in the table is read.
+result = table.upsert(chunks(["doc2"], version=2))
+print(result.num_updates, result.num_inserts) # 4 0
-hudi_table = HudiTableBuilder.from_base_uri("/tmp/trips_table").build()
-batches = hudi_table.read_snapshot(filters=[("city", "=", "san_francisco")])
+# doc3 was removed: delete its chunks.
+table.delete("doc_id = 'doc3'")
-# convert to PyArrow table
-arrow_table = pa.Table.from_batches(batches)
-result = arrow_table.select(["rider", "city", "ts", "fare"])
-print(result)
+# Train on what changed after the initial load, not on the whole table.
+changed = table.read(
+ HudiReadOptions()
+ .with_query_type(HudiQueryType.Incremental)
+ .with_start_timestamp(base.completion_time)
+)
+print(pa.Table.from_batches(changed).num_rows) # 4, the re-embedded chunks
of doc2
+
+# Query the current snapshot with DuckDB (or Polars, pandas, anything Arrow).
+import duckdb
+snapshot = pa.Table.from_batches(table.read())
+duckdb.sql("SELECT doc_id, count(*) AS chunks FROM snapshot GROUP BY doc_id
ORDER BY doc_id").show()
```
-#### Rust
+`append`, `upsert`, and `overwrite` accept a `pyarrow.Table`,
`pyarrow.RecordBatch`, `pyarrow.RecordBatchReader`,
+or any object that implements the Arrow PyCapsule interface, so a Polars
DataFrame or a pandas 2.2+ DataFrame can
+be passed directly. Embeddings may arrive as `list<float32>` or
`fixed_size_list<float32>`; both are stored as lists.
+
+`update` sets columns by predicate and `delete_keys` deletes by key:
+
+```python
+table.update("doc_id = 'doc1'", {"text": "redacted"})
+table.delete_keys(["doc1#0", "doc1#1"])
+```
+
+## Write a table in Rust
+
+The same steps with the `hudi` crate. Batches are Arrow `RecordBatch`es.
```rust
+use arrow::array::{ArrayRef, Float64Array, Int64Array, RecordBatch,
StringArray};
+use arrow::datatypes::{DataType, Field, Schema};
+use hudi::config::table::TableTypeValue;
use hudi::error::Result;
-use hudi::table::builder::TableBuilder as HudiTableBuilder;
-use arrow::compute::concat_batches;
+use hudi::table::{ReadOptions, Table};
+use std::sync::Arc;
+
+fn trips(rows: &[(&str, &str, i64, f64)]) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("id", DataType::Utf8, false),
+ Field::new("city", DataType::Utf8, false),
+ Field::new("ts", DataType::Int64, false),
+ Field::new("fare", DataType::Float64, false),
+ ]));
+ RecordBatch::try_new(schema, vec![
+ Arc::new(StringArray::from(rows.iter().map(|r|
r.0).collect::<Vec<_>>())) as ArrayRef,
+ Arc::new(StringArray::from(rows.iter().map(|r|
r.1).collect::<Vec<_>>())),
+ Arc::new(Int64Array::from(rows.iter().map(|r|
r.2).collect::<Vec<_>>())),
+ Arc::new(Float64Array::from(rows.iter().map(|r|
r.3).collect::<Vec<_>>())),
+ ]).unwrap()
+}
#[tokio::main]
async fn main() -> Result<()> {
- let hudi_table =
HudiTableBuilder::from_base_uri("/tmp/trips_table").build().await?;
- let batches = hudi_table.read_snapshot(&[("city", "=",
"san_francisco")]).await?;
- let batch = concat_batches(&batches[0].schema(), &batches)?;
- let columns = vec!["rider", "city", "ts", "fare"];
- for col_name in columns {
- let idx = batch.schema().index_of(col_name).unwrap();
- println!("{}: {}", col_name, batch.column(idx));
- }
+ let mut table = Table::create("/tmp/trips")
+ .with_table_name("trips")
+ .with_table_type(TableTypeValue::CopyOnWrite) // or MergeOnRead
+ .with_record_key_fields(["id"])
+ .with_partition_fields(["city"])
+ .with_ordering_fields(["ts"])
+ .create()
+ .await?;
+
+ table.append([trips(&[("t1", "sf", 1, 30.0), ("t2", "sf", 1, 45.0), ("t3",
"nyc", 1, 20.0)])]).await?;
+
+ let result = table.upsert([trips(&[("t2", "sf", 2, 50.0), ("t4", "nyc", 2,
15.0)])]).await?;
+ println!(
+ "requested_time={} updates={} inserts={}",
+ result.requested_time.as_deref().unwrap_or("none"),
+ result.num_updates,
+ result.num_inserts
+ );
+
+ table.delete("id = 't1'").await?;
+
+ let batches = table.read(&ReadOptions::new()).await?;
+ println!("{} rows", batches.iter().map(|b| b.num_rows()).sum::<usize>());
Ok(())
}
```
-To run read-optimized (RO) query on Merge-on-Read (MOR) tables, set
`hoodie.read.use.read_optimized.mode` when creating the table.
+Both writers produce the same on-disk layout as the Spark writer.
-#### Python
+## Compact a Merge-on-Read table
+
+On a Merge-on-Read table (`table_type="mor"` in Python,
`TableTypeValue::MergeOnRead` in Rust), upserts, updates,
+and deletes append log files. Compaction folds them into new base files under
one compaction commit;
+rows keep their commit time, so incremental reads see no new rows from it. Run
it on demand, or set
+`hoodie.compact.inline=true` (in the write call) or
`hoodie.datasource.compaction.async.enable=true` (on a
+background task) to compact once `hoodie.compact.inline.max.delta.commits`
(default 5) delta commits accumulate.
+The metadata table compacts automatically.
+
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-hudi_table = (
- HudiTableBuilder
- .from_base_uri("/tmp/trips_table")
- .with_option("hoodie.read.use.read_optimized.mode", "true")
- .build()
+mor = HudiTable.create(
+ f"{tempfile.mkdtemp()}/contract_chunks_mor",
+ record_key="chunk_id",
+ ordering_fields="updated_at",
+ table_type="mor",
+ # Optional: also compact in the write call every 5 delta commits
+ hudi_options={"hoodie.compact.inline": "true"},
)
+mor.append(chunks(["doc1", "doc2"], version=1))
+mor.upsert(chunks(["doc2"], version=2)) # the updates go to log files
+
+result = mor.compact()
+print(result.num_file_groups, result.num_log_files)
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
-let hudi_table =
- HudiTableBuilder::from_base_uri("/tmp/trips_table")
- .with_option("hoodie.read.use.read_optimized.mode", "true")
- .build().await?;
+let mut mor = Table::create("/tmp/trips_mor")
+ .with_table_name("trips_mor")
+ .with_table_type(TableTypeValue::MergeOnRead)
+ .with_record_key_fields(["id"])
+ .with_partition_fields(["city"])
+ .with_ordering_fields(["ts"])
+ // Optional: also compact in the write call every 5 delta commits
+ .with_option("hoodie.compact.inline", "true")
+ .create()
+ .await?;
+mor.append([trips(&[("t1", "sf", 1, 30.0), ("t2", "nyc", 1, 20.0)])]).await?;
+mor.upsert([trips(&[("t1", "sf", 2, 31.0)])]).await?; // the update goes to a
log file
+
+let result = mor.compact().await?;
+println!("{} {}", result.num_file_groups, result.num_log_files); // 1 1
```
-> [!NOTE]
-> Currently reading MOR tables is limited to tables with Parquet data blocks.
+</TabItem>
+</Tabs>
-### Time-Travel Query
+## Why upserts and deletes are fast: the record-level index
-Time-travel query reads the data at a specific timestamp from the table. The
table API also accepts partition filters.
+A client with no index answers "which file holds key `doc2#1`?" by scanning
every file, or at best every
+file whose min/max statistics admit the key. With keys like `<doc>#<pos>` or
UUIDs that is the whole table on every
+sync, however few documents changed.
-#### Python
+Hudi keeps a [record-level index](indexes.md#record-index) in the table's
metadata table: a persistent map from record
+key to file group, sharded by key hash. hudi-rs writes it on `create` and
updates it on every commit. A sync that
+touches 10,000 chunks in a table of millions costs an index lookup of 10,000
keys, which names the file groups to
+rewrite (or, on a Merge-on-Read table, to append a log block to). The rest of
the table is never read.
-```python
-batches = (
- hudi_table
- .read_snapshot_as_of("20241231123456789", filters=[("city", "=",
"san_francisco")])
-)
-```
+## Use cases and numbers
-#### Rust
+The
[`benchmark/upsert`](https://github.com/apache/hudi-rs/tree/543642771a1440f8470053831463052e190b3923/benchmark/upsert)
harness in the hudi-rs
+repository runs five single-node workloads against hudi-rs, each a bulk load
followed by five upsert commits and five
+delete commits; the numbers below also include delta-rs and PyIceberg run
through the same commits on the same
+machine. Every end state is checked against a plain Parquet copy of the
expected table, and the machine's load, memory,
+and disk are recorded next to every timing. Each scenario below links to its
definition so the numbers can be
+reproduced on your own machine.
-```rust
-let batches =
- hudi_table
- .read_snapshot_as_of("20241231123456789", &[("city", "=",
"san_francisco")]).await?;
-```
+| Scenario | Models | Table | Each commit |
+|---|---|---|---|
+|
[`rag-chunks`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#rag-chunks)
| Re-syncing a
[RAG](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) knowledge
base as its sources change | 1M synthetic chunks with 384-float embeddings from
100,000 documents, keyed by `chunk_id` | re-ingest 1,000 changed documents
(10,000 chunks), delete 100 removed documents |
+|
[`training-set`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#training-set)
| Curating a fine-tuning set between training runs | 1M synthetic examples
keyed by `example_id` | upsert 10,000 corrected examples, delete 1,000 rejected
ones, then an incremental read of the changes and a time-travel read of the
first version |
+|
[`uuid`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#uuid)
| Random updates to a dimension table, with changed keys spread across every
file | 10M synthetic rows with random UUID keys | upsert 10,000 existing keys
plus 1,000 new, delete 1,000 |
+|
[`nyc-taxi`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#nyc-taxi)
| Correcting a public dataset in place | [NYC TLC yellow taxi
trips](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page), January
2022, 2.5M rows | upsert 1% of trips with corrected fares, delete 0.1% |
+|
[`tpcds-web-returns`](https://github.com/apache/hudi-rs/blob/543642771a1440f8470053831463052e190b3923/benchmark/upsert/README.md#tpcds-web-returns)
| A standard warehouse fact table | [TPC-DS](https://www.tpc.org/tpcds/)
`web_returns` at scale factor 10, 0.7M rows, composite key | upsert 1% matched
plus 10% new rows, delete 5% |
+
+{/* BENCHMARK_TABLE_START */}
+**Why hudi-rs is faster on upserts and deletes.** Low write amplification and
the record-level index: a Merge-on-Read
+commit appends only the changed records to log files, and the record-level
index points each key at its file group, so
+nothing else in the table is read or rewritten. Neither delta-rs nor PyIceberg
writes deletion vectors in their latest
+releases ([delta-rs
1.6.6](https://github.com/delta-io/delta-rs/releases/tag/python-v1.6.6),
+[PyIceberg
0.12.0](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.12.0)),
and neither keeps a
+record-level index, so every update or delete rewrites each data file it
touches. The changed keys here are spread
+across the table, so each Copy-on-Write commit rewrites nearly all of it: per
upsert commit, hudi-rs wrote 19 MiB on
+`rag-chunks` where delta-rs rewrote 1,689 MiB and PyIceberg 1,432 MiB, and 2
MiB on `uuid` against 671 MiB and 388 MiB.
-### Incremental Query
+<div style={{ textAlign: 'center' }}>
+ <img src="/assets/images/docs/python-rust-quick-start/write-latency.png"
alt="Seconds per upsert and delete commit on five workloads: hudi-rs 0.6.0
Merge-on-Read with the record-level index against delta-rs and PyIceberg"
width="1000"/>
+</div>
-Incremental query reads the changed data from the table for a given time range.
+<div style={{ textAlign: 'center' }}>
+ <img src="/assets/images/docs/python-rust-quick-start/write-bytes.png"
alt="MiB written per upsert and delete commit on the same five workloads"
width="1000"/>
+</div>
-#### Python
+Loading the 10M-row table in Hudi takes 5.7 s, about 2.4 s of which builds the
metadata table and its record-level
+index, an investment that pays off in the latency of every later upsert and
delete.
+{/* BENCHMARK_TABLE_END */}
+
+**Reproduce it.** [This
guide](https://gist.github.com/yihua/66ad80224f8e50020025dc76d138dd0a) pins the
hudi-rs
+commit, every package version and the delta-rs and PyIceberg cases, and gives
the exact commands.
+
+## Read a table
+
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-# read the records between t1 (exclusive) and t2 (inclusive)
-batches = hudi_table.read_incremental_records(t1, t2)
+from hudi import HudiReadOptions, HudiTableBuilder
+import pyarrow as pa
+
+table = HudiTableBuilder.from_base_uri("/tmp/trips").build()
+
+# Snapshot, with partition and row filters pushed down
+batches = table.read(HudiReadOptions(filters=[("city", "=", "sf")]))
+print(pa.Table.from_batches(batches))
+
+# Time travel
+batches =
table.read(HudiReadOptions().with_as_of_timestamp("20261029120000000"))
+
+# Incremental: rows from commits that completed after a point in time, such as
+# the completion_time of an earlier write
+from hudi import HudiQueryType
+since = "20261029120000000"
+batches =
table.read(HudiReadOptions().with_query_type(HudiQueryType.Incremental).with_start_timestamp(since))
-# read the records after t1
-batches = hudi_table.read_incremental_records(t1)
+# Stream batches instead of materializing the result
+for batch in table.read_stream(HudiReadOptions(projection=["id",
"fare"]).with_batch_size(4096)):
+ print(batch.num_rows)
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
-// read the records between t1 (exclusive) and t2 (inclusive)
-let batches = hudi_table.read_incremental_records(t1, Some(t2)).await?;
+use hudi::table::builder::TableBuilder as HudiTableBuilder;
+use hudi::table::{QueryType, ReadOptions};
-// read the records after t1
-let batches = hudi_table.read_incremental_records(t1, None).await?;
-```
+let table = HudiTableBuilder::from_base_uri("/tmp/trips").build().await?;
-> [!NOTE]
-> Currently the only supported format for the timestamp arguments is Hudi
Timeline format: `yyyyMMddHHmmssSSS` or `yyyyMMddHHmmss`.
+// Snapshot, with partition and row filters pushed down
+let batches = table.read(&ReadOptions::new().with_filters([("city", "=",
"sf")])?).await?;
-## Query Engine Integration
+// Time travel
+let batches =
table.read(&ReadOptions::new().with_as_of_timestamp("20261029120000000")).await?;
-Hudi-rs provides APIs to support integration with query engines. The sections
below highlight some commonly used APIs.
+// Incremental: rows from commits that completed after a point in time, such as
+// the completion_time of an earlier write
+let since = "20261029120000000";
+let options =
ReadOptions::new().with_query_type(QueryType::Incremental).with_start_timestamp(since);
+let batches = table.read(&options).await?;
+```
-### Table API
+</TabItem>
+</Tabs>
-Create a Hudi table instance using its constructor or the `TableBuilder` API.
+Timestamps accept the Hudi timeline format (`yyyyMMddHHmmssSSS`), Unix epoch
in seconds through nanoseconds, and
+RFC 3339 with a timezone offset.
-| Stage | API | Description
|
-|-----------------|-------------------------------------------|--------------------------------------------------------------------------------|
-| Query planning | `get_file_slices()` | For snapshot
query, get a list of file slices. |
-| | `get_file_slices_splits()` | For snapshot
query, get a list of file slices in splits. |
-| | `get_file_slices_as_of()` | For
time-travel query, get a list of file slices at a given time. |
-| | `get_file_slices_splits_as_of()` | For
time-travel query, get a list of file slices in splits at a given time. |
-| | `get_file_slices_between()` | For
incremental query, get a list of changed file slices between a time range. |
-| Query execution | `create_file_group_reader_with_options()` | Create a file
group reader instance with the table instance's configs. |
+To run a read-optimized query on a Merge-on-Read table (base files only, no
log merge), set
+`hoodie.read.use.read_optimized.mode` to `true` in the read options.
-### File Group API
+## Query with DuckDB, Polars, and DataFusion
-Create a Hudi file group reader instance using its constructor or the Hudi
table API `create_file_group_reader_with_options()`.
+hudi-rs returns Arrow, so anything that consumes Arrow can query a Hudi table
in Python.
-| Stage | API | Description
|
-|-----------------|---------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Query execution | `read_file_slice()` | Read records from
a given file slice; based on the configs, read records from only base file, or
from base file and log files, and merge records based on the configured
strategy. |
+```python
+import duckdb, polars as pl, pyarrow as pa
+from hudi import HudiTableBuilder
+trips =
pa.Table.from_batches(HudiTableBuilder.from_base_uri("/tmp/trips").build().read())
-### Apache DataFusion
+duckdb.sql("SELECT city, avg(fare) FROM trips GROUP BY city").show()
+print(pl.from_arrow(trips).group_by("city").agg(pl.col("fare").mean()))
+```
-Enabling the `hudi` crate with `datafusion` feature will provide a
[DataFusion](https://datafusion.apache.org/)
-extension to query Hudi tables.
+In Rust, the `datafusion` feature of the `hudi` crate registers a Hudi table
as a DataFusion table provider.
<details>
-<summary>Add crate hudi with datafusion feature to your application to query a
Hudi table.</summary>
+<summary>Add the hudi crate with the datafusion feature to your
application.</summary>
```shell
cargo new my_project --bin && cd my_project
-cargo add tokio@1 datafusion@43
+cargo add tokio@1 datafusion@55 # the DataFusion major hudi 0.6 builds against
cargo add hudi --features datafusion
```
-Update `src/main.rs` with the code snippet below then `cargo run`.
-
</details>
```rust
@@ -191,57 +354,83 @@ use hudi::HudiDataSource;
#[tokio::main]
async fn main() -> Result<()> {
let ctx = SessionContext::new();
- let hudi = HudiDataSource::new_with_options(
- "/tmp/trips_table",
- [("hoodie.read.input.partitions", "5")]).await?;
- ctx.register_table("trips_table", Arc::new(hudi))?;
- let df: DataFrame = ctx.sql("SELECT * from trips_table where city =
'san_francisco'").await?;
+ let hudi = HudiDataSource::new_with_options("/tmp/trips",
[("hoodie.read.input.partitions", "5")]).await?;
+ ctx.register_table("trips", Arc::new(hudi))?;
+ let df: DataFrame = ctx.sql("SELECT * FROM trips WHERE city =
'sf'").await?;
df.show().await?;
Ok(())
}
```
-### Other Integrations
+The same provider is available in Python through `HudiDataFusionDataSource`
when the `datafusion` extra of the
+`hudi` package is installed. Hudi tables can also be read by
[Daft](https://docs.daft.ai/en/stable/io/hudi/) and
+[Ray](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_hudi.html#ray.data.read_hudi).
-Hudi is also integrated with
+## Work with cloud storage
-- [Daft](https://docs.daft.ai/en/stable/io/hudi/)
--
[Ray](https://docs.ray.io/en/latest/data/api/doc/ray.data.read_hudi.html#ray.data.read_hudi)
+Set the usual cloud credentials as environment variables (`AWS_*`, `AZURE_*`,
`GOOGLE_*`) and use an `s3://`,
+`az://`, or `gs://` base URI. Storage options can also be passed explicitly.
-### Work with cloud storage
-
-Ensure cloud storage credentials are set properly as environment variables,
e.g., `AWS_*`, `AZURE_*`, or `GOOGLE_*`.
-Relevant storage environment variables will then be picked up. The target
table's base uri with schemes such
-as `s3://`, `az://`, or `gs://` will be processed accordingly.
-
-Alternatively, you can pass the storage configuration as options via Table
APIs.
-
-#### Python
+<Tabs>
+<TabItem value="python" label="Python" default>
```python
-from hudi import HudiTableBuilder
+from hudi import HudiTable, HudiTableBuilder
-hudi_table = (
- HudiTableBuilder
- .from_base_uri("s3://bucket/trips_table")
- .with_option("aws_region", "us-west-2")
- .build()
-)
+table = HudiTable.create("s3://bucket/trips", record_key="id",
storage_options={"aws_region": "us-west-2"})
+
+reader =
HudiTableBuilder.from_base_uri("s3://bucket/trips").with_option("aws_region",
"us-west-2").build()
```
-#### Rust
+</TabItem>
+<TabItem value="rust" label="Rust">
```rust
+use hudi::table::Table;
use hudi::table::builder::TableBuilder as HudiTableBuilder;
-async fn main() -> Result<()> {
- let hudi_table =
- HudiTableBuilder::from_base_uri("s3://bucket/trips_table")
- .with_option("aws_region", "us-west-2")
- .build().await?;
-}
+let mut table = Table::create("s3://bucket/trips")
+ .with_table_name("trips")
+ .with_record_key_fields(["id"])
+ .with_storage_option("aws_region", "us-west-2")
+ .create()
+ .await?;
+
+let reader = HudiTableBuilder::from_base_uri("s3://bucket/trips")
+ .with_option("aws_region", "us-west-2")
+ .build()
+ .await?;
```
+</TabItem>
+</Tabs>
+
+## Query engine integration
+
+For engines that plan their own scans, hudi-rs exposes the planning and
execution primitives directly.
+
+| Stage | API |
Description
|
+|-----------------|----------------------------------------------------|-----------------------------------------------------------------------------------------------------|
+| Query planning | `get_file_slices(options)` | The
file slices a read targets, dispatched on the query type (snapshot, time
travel, incremental). |
+| | `compute_table_stats(options)` |
Estimated row count and byte size for scan planning, from the metadata table
(snapshot only). |
+| Query execution | `create_file_group_reader_with_options()` | A
file group reader with the table's configs; pass the same read options used for
`get_file_slices`. |
+| | `read_file_slice()` / `read_file_slice_stream()` | Read
one file slice, merging base and log files per the table's merge strategy.
|
+
+## Resources
+
+- Rust API reference: [docs.rs/hudi](https://docs.rs/hudi), the `hudi` crate,
which re-exports
+ [`hudi-core`](https://docs.rs/hudi-core) and, with the `datafusion` feature,
+ [`hudi-datafusion`](https://docs.rs/hudi-datafusion).
+- Python API reference: the docstrings in the type stubs shipped with the
package,
+
[`hudi/_internal.pyi`](https://github.com/apache/hudi-rs/blob/main/python/hudi/_internal.pyi),
which IDEs and type checkers read.
+- The [hudi-rs README](https://github.com/apache/hudi-rs#quick-start): a
shorter quick start and the full read and write reference.
+- Runnable Rust example:
[`crates/core/examples/table_writes.rs`](https://github.com/apache/hudi-rs/blob/main/crates/core/examples/table_writes.rs).
+- Reader semantics:
[`docs/reader-spec.md`](https://github.com/apache/hudi-rs/blob/main/docs/reader-spec.md).
Writer design:
+
[`docs/writer-design.md`](https://github.com/apache/hudi-rs/blob/main/docs/writer-design.md).
+- Packages: [PyPI](https://pypi.org/project/hudi/) and
[crates.io](https://crates.io/crates/hudi). Source:
+ [github.com/apache/hudi-rs](https://github.com/apache/hudi-rs).
+- Hudi concepts (timeline, table types, query types): [overview](overview.mdx).
+
## Contributing
Check out the [contributing
guide](https://github.com/apache/hudi-rs/blob/main/CONTRIBUTING.md) for all the
details about making contributions to the project.
diff --git a/website/versioned_docs/version-1.2.1/use_cases.md
b/website/versioned_docs/version-1.2.1/use_cases.md
index d0b6cecdee11..d89a3ee79acc 100644
--- a/website/versioned_docs/version-1.2.1/use_cases.md
+++ b/website/versioned_docs/version-1.2.1/use_cases.md
@@ -3,7 +3,7 @@ title: "Use Cases"
keywords: [ hudi, data ingestion, etl, real time, use cases]
summary: "Following are some sample use-cases for Hudi, which illustrate the
benefits in terms of faster processing & increased efficiency"
toc: true
-last_modified_at: 2019-12-30T15:59:57-04:00
+last_modified_at: 2026-10-29T00:00:00-07:00
---
Apache Hudi is a powerful [data lakehouse
platform](https://hudi.apache.org/blog/2021/07/21/streaming-data-lake-platform)
that shines in a variety of use cases due to its high-performance design, rich
feature set, and
@@ -135,4 +135,4 @@ For a worked example covering image embeddings, raw bytes,
and similarity search
- Incremental embedding pipelines: Hudi's [incremental
query](sql_queries.md#incremental-query) lets feature pipelines re-embed only
the rows that changed since a given commit, instead of reprocessing the full
batch.
- Multi-modal transactional guarantees: Vector updates, binary changes, and
metadata modifications commit atomically. A similarity query joining the
`VECTOR` column with the `BLOB` column never observes a state in which the
embedding refers to a deleted or stale binary asset.
- Automated table services: [File sizing](file_sizing.md),
[cleaning](cleaning.md), [compaction](compaction.md), and
[clustering](clustering.md) apply to tables that contain `VECTOR`, `BLOB`, and
`VARIANT` columns.
-- Open, multi-engine access: Hudi tables are written and read by [Apache
Spark](quick-start-guide.md) and [Apache Flink](flink-quick-start-guide.md),
and by the upcoming native [Python / Rust client
(hudi-rs)](python-rust-quick-start-guide.md) reading the underlying Parquet
representation directly.
+- Open, multi-engine access: Hudi tables are written and read by [Apache
Spark](quick-start-guide.md) and [Apache Flink](flink-quick-start-guide.md),
and by the native [Python / Rust client
(hudi-rs)](python-rust-quick-start-guide.md), which creates, upserts, deletes
and reads tables from a single process with no JVM.