This is an automated email from the ASF dual-hosted git repository.
yihua pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hudi-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 441ff6b3 docs: add a feature overview, rust quick start, and doc links
to the readme (#940)
441ff6b3 is described below
commit 441ff6b3ce3b34e2495f3c63bdc908bf248c41e4
Author: Y Ethan Guo <[email protected]>
AuthorDate: Fri Oct 30 13:56:21 2026 -0700
docs: add a feature overview, rust quick start, and doc links to the readme
(#940)
---
CONTRIBUTING.md | 2 +-
README.md | 217 ++++++++++++++++++++++++++++----
benchmark/filegroup/README.md | 2 +-
benchmark/filegroup/src/main.rs | 2 +-
crates/core/src/avro_to_arrow/README.md | 2 +-
docs/reader-spec.md | 13 +-
docs/writer-design.md | 39 ++++--
7 files changed, 236 insertions(+), 41 deletions(-)
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index f8df7ed7..ccf9e9dc 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -30,7 +30,7 @@ in https://github.com/apache/hudi-rs/contribute.
## File an issue
Testing and reporting bugs are also valueable contributions. Please follow
-the [issue
template](https://github.com/apache/hudi-rs/issues/new?template=bug_report.yml)
to file bug reports.
+the [issue
template](https://github.com/apache/hudi-rs/issues/new?template=1-bug.yml) to
file bug reports.
## Issue tracking
diff --git a/README.md b/README.md
index c2414d14..851e1f2e 100644
--- a/README.md
+++ b/README.md
@@ -56,17 +56,27 @@ bindings. It creates, writes, and reads Hudi tables from
Rust and Python in a si
[crates]: https://crates.io/crates/hudi
[crates-badge]:
https://img.shields.io/crates/d/hudi?style=flat-square&color=163669
-The `hudi` crate carries two features: `datafusion` (off by default, see
-[Apache DataFusion](#apache-datafusion)) and `spill-rocksdb` (on by default),
the merge map's
-on-disk tier, which a merge-on-read merge spills to when a file group's log
records exceed
-`hoodie.memory.merge.max.size`. RocksDB is built from source with `bindgen`,
so it needs `libclang`
-and a C++ toolchain; `default-features = false` drops it, and a merge that
would have spilled then
-fails instead.
+## Features
+
+| Area | Supported
|
+|-------------------|-----------------------------------------------------------------------------------------------------------------|
+| Table types | Copy-on-Write (COW), Merge-on-Read (MOR)
|
+| Table versions | Read: 6, 8, 9 (Hudi 0.14 and later). Write: 8, 9 (Hudi
1.0 and later), plus a narrow append path on 6 |
+| Queries | Snapshot, time travel, incremental, read-optimized
(MOR), streaming snapshot reads, column filters with pruning |
+| Writes | Create, append, upsert, update, delete (by predicate or
key), overwrite, dynamic partition overwrite |
+| Table services | Merge-on-read compaction (experimental: on demand,
inline, or in the background), rollback of failed writes, timeline archival |
+| Metadata table | File listings, column and partition stats, record-level
index, kept up to date on every write and compacted as it grows (on by default)
|
+| Base file formats | Parquet (read and write), Lance (read), HFile (metadata
table) |
+| Storage | Local file system, Amazon S3, Azure Blob Storage, Google
Cloud Storage |
+| APIs | Rust and Python (read and write); C++ and a C ABI
(`hudi-ffi`) for file-slice reads |
+| Query engines | [Apache DataFusion](#apache-datafusion) (`datafusion`
feature in Rust, included in Python); see [Other
Integrations](#other-integrations) |
## Quick start
Create a table, upsert and delete rows, then read what changed after the first
commit.
+### Python
+
```python
import tempfile
@@ -100,8 +110,102 @@ snapshot = pa.Table.from_batches(table.read())
print(sorted(snapshot.column("doc_id").to_pylist())) # ['doc1', 'doc2',
'doc4']
```
+### Rust
+
+```shell
+cargo add hudi
+cargo add arrow@59
+cargo add tokio@1 --features macros,rt-multi-thread
+```
+
+```rust
+use std::sync::Arc;
+
+use arrow::array::{Int64Array, RecordBatch, StringArray};
+use arrow::datatypes::{DataType, Field, Schema};
+use hudi::error::Result;
+use hudi::table::{QueryType, ReadOptions, Table};
+
+fn docs(ids: &[&str], version: i64) -> RecordBatch {
+ let schema = Arc::new(Schema::new(vec![
+ Field::new("doc_id", DataType::Utf8, false),
+ Field::new("title", DataType::Utf8, false),
+ Field::new("version", DataType::Int64, false),
+ ]));
+ let titles: Vec<String> = ids.iter().map(|d| format!("{d}
v{version}")).collect();
+ RecordBatch::try_new(
+ schema,
+ vec![
+ Arc::new(StringArray::from(ids.to_vec())),
+ Arc::new(StringArray::from(titles)),
+ Arc::new(Int64Array::from(vec![version; ids.len()])),
+ ],
+ )
+ .unwrap()
+}
+
+#[tokio::main]
+async fn main() -> Result<()> {
+ let base_uri = std::env::temp_dir().join(format!("docs_{}",
std::process::id()));
+ let mut table = Table::create(base_uri.to_str().unwrap())
+ .with_table_name("docs")
+ .with_record_key_fields(["doc_id"])
+ .with_ordering_fields(["version"])
+ .create()
+ .await?;
+ let first = table.append([docs(&["doc1", "doc2", "doc3"], 1)]).await?;
+
+ let result = table.upsert([docs(&["doc2", "doc4"], 2)]).await?;
+ println!("{} {}", result.num_updates, result.num_inserts); // 1 1
+
+ table.delete("doc_id = 'doc3'").await?;
+
+ // Commits that completed after the first one.
+ let options = ReadOptions::new()
+ .with_query_type(QueryType::Incremental)
+ .with_start_timestamp(first.completion_time.unwrap());
+ let changed = table.read(&options).await?;
+ let snapshot = table.read(&ReadOptions::new()).await?;
+ let num_rows = |batches: &[RecordBatch]| batches.iter().map(|b|
b.num_rows()).sum::<usize>();
+ println!("{} {}", num_rows(&changed), num_rows(&snapshot)); // 2 3
+ Ok(())
+}
+```
+
See [Writing Tables](#writing-tables) for the full write API, and
-[`table_writes.rs`](crates/core/examples/table_writes.rs) for a Rust example.
+[`crates/core/examples/table_writes.rs`](crates/core/examples/table_writes.rs)
+(`cargo run -p hudi-core --example table_writes`) for a runnable Rust tour of
it. The
+[Python/Rust quick start
guide](https://hudi.apache.org/docs/python-rust-quick-start-guide) on the
+Hudi website goes further, with DuckDB, Polars, and DataFusion queries. API
references are listed
+under [Documentation](#documentation).
+
+## Requirements
+
+- Python 3.10 or later. Wheels are published to PyPI.
+- Rust 1.94.1 or later.
+- The `hudi` crate carries two features: `datafusion` (off by default, see
+ [Apache DataFusion](#apache-datafusion)) and `spill-rocksdb` (on by
default), the merge map's
+ on-disk tier, which a merge-on-read merge spills to when a file group's log
records exceed
+ `hoodie.memory.merge.max.size`. RocksDB is built from source with `bindgen`,
so it needs
+ `libclang` and a C++ toolchain; `default-features = false` drops it, and a
merge that would have
+ spilled then fails instead.
+
+## Documentation
+
+- Quick start: the [Python and Rust quick start
guide](https://hudi.apache.org/docs/python-rust-quick-start-guide)
+ on the Hudi website, and the shorter [quick start
above](https://github.com/apache/hudi-rs#quick-start).
+- Rust API reference: [docs.rs/hudi](https://docs.rs/hudi) (the `hudi` crate
re-exports
+ [`hudi-core`](https://docs.rs/hudi-core) and, with the `datafusion` feature,
+ [`hudi-datafusion`](https://docs.rs/hudi-datafusion)).
+- Python API reference: the package ships type stubs with docstrings
+
([`python/hudi/_internal.pyi`](https://github.com/apache/hudi-rs/blob/main/python/hudi/_internal.pyi)),
which IDEs and
+ type checkers read.
+- Runnable Rust example:
[`crates/core/examples/table_writes.rs`](https://github.com/apache/hudi-rs/blob/main/crates/core/examples/table_writes.rs).
+- Reader semantics (read options, filters, guarantees):
[`docs/reader-spec.md`](https://github.com/apache/hudi-rs/blob/main/docs/reader-spec.md).
+- Writer design:
[`docs/writer-design.md`](https://github.com/apache/hudi-rs/blob/main/docs/writer-design.md).
+- Packages: [PyPI](https://pypi.org/project/hudi/) and
[crates.io](https://crates.io/crates/hudi).
+ Source: [github.com/apache/hudi-rs](https://github.com/apache/hudi-rs).
Contributing:
[`CONTRIBUTING.md`](https://github.com/apache/hudi-rs/blob/main/CONTRIBUTING.md).
+- Hudi concepts (timeline, table types, query types):
[hudi.apache.org/docs/overview](https://hudi.apache.org/docs/overview).
## Reading Tables
@@ -214,7 +318,10 @@ Timestamp strings without a timezone offset (for example
`yyyy-MM-dd'T'HH:mm:ss`
### Incremental Query
-Incremental query reads the changed data from the table for a given time range.
+Incremental query reads the changed data from the table for a given time
range, half-open
+`(start, end]`. On table version 8 and later (timeline layout v2) the bounds
are commit
+completion times, so start from a write result's `completion_time`; on older
tables they are
+requested times. The end defaults to everything committed so far.
#### Python
@@ -229,7 +336,7 @@ batches = hudi_table.read(
.with_end_timestamp(t2)
)
-# read the records after t1 (end defaults to the latest commit)
+# read the records after t1 (end defaults to everything committed so far)
batches = hudi_table.read(
HudiReadOptions()
.with_query_type(HudiQueryType.Incremental)
@@ -257,7 +364,7 @@ let options = ReadOptions::new()
.with_end_timestamp(t2);
let batches = hudi_table.read(&options).await?;
-// read the records after t1 (end defaults to the latest commit)
+// read the records after t1 (end defaults to everything committed so far)
let options = ReadOptions::new()
.with_query_type(QueryType::Incremental)
.with_start_timestamp(t1);
@@ -365,8 +472,9 @@ ArrowArrayStream* stream_ptr =
reader->read_file_slice_from_paths("relative/path
## Writing Tables
hudi-rs includes a native, single-node write path that is storage-compatible
-with the Apache Hudi Spark writer: tables written here are readable — and
-serviceable (compaction, clustering, cleaning) — by Spark, and vice versa.
+with the Apache Hudi Spark writer: Spark reads tables written here and runs
+its table services on them (compaction, clustering, cleaning), and hudi-rs
reads
+and writes tables Spark wrote.
Writes target COPY_ON_WRITE and MERGE_ON_READ tables at table version 9
(Hudi 1.1.x; version 8 via `with_table_version(8)`), and maintain the
metadata table (file listings, column/partition stats, record-level index) on
@@ -434,19 +542,19 @@ table.append(batch)
#### Rust
```rust
-use arrow_array::RecordBatch;
+use arrow::array::RecordBatch;
-let batch: RecordBatch = /* columns: uuid, city, ts, fare, ... */
-# unimplemented!();
+let batch: RecordBatch = todo!("columns: uuid, city, ts, fare, ...");
table.append([batch]).await?;
```
### Upsert
-`upsert` tags incoming records against the record-level index: existing keys
+`upsert` tags incoming records against the table's index (the record-level
+index when enabled, otherwise a scan of the base files' keys): existing keys
update their file group in place (a rewritten base file on COW, a log file on
-MOR); new keys pack into existing small files first and then form new file
-groups.
+MOR). On COW, new keys fill existing small files first and then form new file
+groups; on MOR, they always form new file groups.
#### Python
@@ -529,6 +637,55 @@ table.overwrite([batch]).await?;
table.dynamic_partition_overwrite([batch]).await?;
```
+### Compaction (Experimental)
+
+On a merge-on-read table, upserts, updates, and deletes land in log files.
`compact` folds
+them into new base files: every file group whose latest file slice has log
files gets a new
+base file of its merged rows, under one compaction commit. Rows keep their
commit time, so
+incremental reads see no new rows from a compaction. When no file group has
log files,
+nothing is written. A copy-on-write table has nothing to compact and returns
an error.
+
+Compaction can also run after delta commits once the number since the latest
compaction
+reaches `hoodie.compact.inline.max.delta.commits` (default 5): in the write
call with
+`hoodie.compact.inline=true`, or on a background task with
+`hoodie.datasource.compaction.async.enable=true`, which `await_compaction`
waits for. Both
+are off by default. The metadata table compacts itself as commits accumulate.
+
+#### Python
+
+```python
+mor = HudiTable.create(
+ "/tmp/trips_mor",
+ record_key="uuid",
+ table_type="mor",
+ # Optional: also compact in the write call every 5 delta commits.
+ hudi_options={"hoodie.compact.inline": "true"},
+)
+mor.append(batch)
+mor.upsert(batch) # updates go to log files
+
+result = mor.compact()
+print(result.num_file_groups, result.num_log_files, result.num_writes)
+```
+
+#### Rust
+
+```rust
+let mut mor = Table::create("/tmp/trips_mor")
+ .with_table_name("trips")
+ .with_table_type(TableTypeValue::MergeOnRead)
+ .with_record_key_fields(["uuid"])
+ // Optional: also compact in the write call every 5 delta commits.
+ .with_option("hoodie.compact.inline", "true")
+ .create()
+ .await?;
+mor.append([batch.clone()]).await?;
+mor.upsert([batch]).await?; // updates go to log files
+
+let result = mor.compact().await?;
+println!("{} {} {}", result.num_file_groups, result.num_log_files,
result.num_writes);
+```
+
### Behavior Notes
- Every write runs as a full Hudi transaction: instants are requested and
@@ -552,11 +709,11 @@ Create a Hudi table instance using its constructor or the
`TableBuilder` API.
All read APIs accept a `ReadOptions` (Rust) / `HudiReadOptions` (Python)
value. It stores three fields — `filters`, `projection`, and `hudi_options` —
and exposes chainable `with_*` builders for the rest. The available knobs:
- `query_type` (`with_query_type`) — `Snapshot` (default) or `Incremental`.
Drives dispatch in `read`, `read_stream`, and `get_file_slices`.
-- `filters` — column filters as `(field, op, value)` tuples. The field can be
any column (partition or data). Used for partition pruning, file-level stats
pruning (snapshot only), and row-level filtering.
+- `filters` — column filters as `(field, op, value)` tuples. The field can be
any column (partition or data). Used for partition pruning, file-level stats
pruning (copy-on-write and read-optimized reads), file-group selection through
the record-level index for `=`/`IN` filters on the record key (latest snapshot,
not read-optimized), predicate pushdown into the parquet reader, and row-level
filtering.
- `projection` — columns to return. Streaming pushes the projection down to
the parquet reader; eager reads project after merging.
- `batch_size` (`with_batch_size`) — rows per batch (streaming only; eager
reads return one batch per file slice).
- `as_of_timestamp` (`with_as_of_timestamp`) — snapshot/time-travel timestamp
(defaults to latest commit).
-- `start_timestamp` / `end_timestamp` (`with_start_timestamp` /
`with_end_timestamp`) — incremental range (defaults to earliest…latest).
+- `start_timestamp` / `end_timestamp` (`with_start_timestamp` /
`with_end_timestamp`) — incremental range, bounding completion times on table
version 8 and later (defaults to the beginning of the timeline…everything
committed so far).
- `hudi_options` — Hudi configs for this read (e.g.
`hoodie.read.use.read_optimized.mode`). A config that selects *which* read to
perform — `hoodie.read.query.type` and the as-of/start/end timestamps — is
per-read only and is dropped when set on the table. The rest describe *how* to
read: set them on the table and override them here. See [Read
configs](#read-configs).
| Stage | API
| Description
|
@@ -577,7 +734,7 @@ silently redirect every later read.
|-----------------------------------------------------------------|---------------|---------------|--------------------------------------------------------------------------------------------------------------------------|
| `hoodie.read.query.type` | `snapshot`
| per read | `snapshot` or `incremental`.
|
| `hoodie.read.as.of.timestamp` | latest
commit | per read | Snapshot time-travel point.
|
-| `hoodie.read.start.timestamp` / `hoodie.read.end.timestamp` | earliest /
latest | per read | Incremental window, half-open `(start, end]`.
|
+| `hoodie.read.start.timestamp` / `hoodie.read.end.timestamp` | earliest /
latest | per read | Incremental window, half-open `(start, end]`; completion
times on table version 8 and later, requested times before. |
| `hoodie.read.file.group.reader.version` | `2`
| table or read | Which file group reader merges a slice. Version 2 is the
default; a read it cannot serve falls back to version 1. |
| `hoodie.read.use.read_optimized.mode` | `false`
| table or read | Read base files only, skipping the log files, on MOR
tables. |
| `hoodie.read.stream.batch_size` | `1024`
| table or read | Rows per batch for streaming reads.
|
@@ -619,7 +776,7 @@ bindings) are not tied to these versions. The
`hudi[datafusion]` extra first shi
| hudi-rs | DataFusion | arrow | Python `datafusion` package |
|---------|------------|-------|-----------------------------|
-| 0.6 (unreleased) | 55 | 59 | `>=55.0.0,<56` |
+| 0.6 | 55 | 59 | `>=55.0.0,<56` |
| 0.5 | 54 | 58 | `>=54.0.0,<55` |
| 0.4 | 46 | 54 | not available |
| 0.3 | 43 | 53 | not available |
@@ -642,7 +799,7 @@ slows compilation noticeably. It goes away once a Lance
release shares the works
```shell
cargo new my_project --bin && cd my_project
-cargo add tokio@1 datafusion@55
+cargo add tokio@1 datafusion@55 # the DataFusion major that pairs with the
hudi release, see the table above
cargo add hudi --features datafusion
```
@@ -735,6 +892,20 @@ async fn main() -> Result<()> {
}
```
+## Project layout
+
+| Path | Contents
|
+|---------------------|-----------------------------------------------------------------------------|
+| `crates/hudi` | The `hudi` crate: re-exports `hudi-core`, and
`hudi-datafusion` with the `datafusion` feature |
+| `crates/core` | `hudi-core`: tables, timeline, file groups, metadata
table, readers, writers |
+| `crates/datafusion` | `hudi-datafusion`: the DataFusion `TableProvider`
|
+| `crates/ffi` | `hudi-ffi`: a C ABI for engines, over the Arrow C Data
Interface |
+| `python` | The `hudi` Python package (PyO3)
|
+| `cpp` | The C++ binding (`cxx`)
|
+| `crates/test` | Shared test fixtures
|
+| `benchmark` | Benchmark harnesses
|
+| `docs` | Reader spec and writer design
|
+
## Contributing
Check out the [contributing guide](./CONTRIBUTING.md) for all the details
about making contributions to the project.
diff --git a/benchmark/filegroup/README.md b/benchmark/filegroup/README.md
index be6e7f91..382739f3 100644
--- a/benchmark/filegroup/README.md
+++ b/benchmark/filegroup/README.md
@@ -40,7 +40,7 @@ cargo build -p fg-bench --release
--columns id,ts,value --iterations 5 --output-json run.json
# Override the streaming chunk size (hoodie.read.stream.batch_size). Omit to
use
-# the reader default (4096 rows).
+# the reader default (1024 rows).
./target/release/fg-bench --table /abs/path/to/table --batch-size 1024
```
diff --git a/benchmark/filegroup/src/main.rs b/benchmark/filegroup/src/main.rs
index 4423413a..13acb328 100644
--- a/benchmark/filegroup/src/main.rs
+++ b/benchmark/filegroup/src/main.rs
@@ -69,7 +69,7 @@ struct Args {
columns: Option<Vec<String>>,
/// Streaming chunk size (`hoodie.read.stream.batch_size`). When
- /// omitted the reader uses its built-in default (DEFAULT_BATCH_SIZE =
4096).
+ /// omitted the reader uses its built-in default (1024 rows).
/// Set this to measure the effect of chunk granularity on wall time. (The
/// peak-RSS effect shows only on the streaming `open()` path; this bench
/// drives `read()`, which retains the full output, so `max_rss` is
diff --git a/crates/core/src/avro_to_arrow/README.md
b/crates/core/src/avro_to_arrow/README.md
index 06595b93..5793f83f 100644
--- a/crates/core/src/avro_to_arrow/README.md
+++ b/crates/core/src/avro_to_arrow/README.md
@@ -27,4 +27,4 @@
- Removed `reader.rs` (not needed for Hudi use case).
- Original tests are removed.
- DataFusion errors (`datafusion_common::error::Result`) replaced with Hudi
errors (`crate::error::Result`).
-- Adjusted `AvroArrowArrayReader` to work with `Iterator<Item =
AvroResult<Value>>` instead of a reader-based approach.
+- Only the schema conversion (`schema.rs`, `to_arrow_schema`) is kept; the
array reader is removed.
diff --git a/docs/reader-spec.md b/docs/reader-spec.md
index 1eae8581..4448b5e5 100644
--- a/docs/reader-spec.md
+++ b/docs/reader-spec.md
@@ -38,7 +38,7 @@ For tutorials and end-to-end examples, see the
[README](../README.md). Covers Ru
The two `QueryType` variants on `ReadOptions`:
- **Snapshot** — latest table state at one commit (the latest by default; an
explicit `as_of_timestamp` for time-travel).
-- **Incremental** — records changed in the half-open range (`start_timestamp`,
`end_timestamp`].
+- **Incremental** — records changed in the half-open range (`start_timestamp`,
`end_timestamp`]. On timeline layout v2 (table version 8 and later) the bounds
are commit completion times, so pass a write result's `completion_time` as the
start; on layout v1 they are requested times.
Snapshot reads have an **eager** form returning all batches and a
**streaming** form yielding batches as they're read. Incremental reads
currently expose only the eager form.
@@ -59,7 +59,7 @@ Per-slice reads — reading a single `FileSlice` the caller
already selected (ty
| `with_query_type(QueryType)` | `hoodie.read.query.type`
| `snapshot` |
| `with_as_of_timestamp(ts)` | `hoodie.read.as.of.timestamp`
| latest commit (Snapshot only) |
| `with_start_timestamp(ts)` | `hoodie.read.start.timestamp`
| `19700101000000000` (Incremental only) |
-| `with_end_timestamp(ts)` | `hoodie.read.end.timestamp`
| latest commit (Incremental only) |
+| `with_end_timestamp(ts)` | `hoodie.read.end.timestamp`
| everything committed so far (Incremental only) |
| `with_batch_size(n)` | `hoodie.read.stream.batch_size`
| `1024` (streaming, base-file-only slices) |
`with_batch_size` scope: it sizes the batches of a slice that has **no log
files**, which is read straight from the base file. A slice **with** log files
is merged, and the merge sizes its own chunks (`MERGE_CHUNK_ROWS`) so that one
chunk of merging is a bounded amount of synchronous work;
`hoodie.read.stream.batch_size` has no effect there. The two happen to agree at
1024 today.
@@ -98,11 +98,12 @@ For `IN` / `NOT IN`, the value string is split on unescaped
commas and trimmed:
`with_filters` parses and cardinality-validates upfront; an unrecognized
operator or empty `IN` / `NOT IN` value list errors at the builder rather than
at read time.
-The `field` may be any column. Filters drive three things:
+The `field` may be any column. Filters drive these:
- **Partition pruning** when the field is a partition column. Always applied.
-- **File-level stats pruning** when the field is a data column with min/max
stats in the metadata table. Snapshot/time-travel only — incremental file
planning does not stats-prune.
-- **Row-group pruning** inside a base file, from the parquet footer's own
statistics. Conservative: a row group that cannot match is never fetched, and
one that might match is kept.
+- **File-level stats pruning** when the field is a data column with min/max
stats, from the metadata table's column stats, falling back to the parquet
footer for a file without a stats record. Copy-on-write tables and
read-optimized reads only, since a merged slice's rows are not described by its
base file's stats. Snapshot/time-travel only — incremental file planning does
not stats-prune.
+- **Record-index file-group selection** when the record-level index is enabled
and an `=` / `IN` filter names the record key: a latest-snapshot read (not time
travel, not read-optimized) keeps only the file groups the index names.
+- **Parquet pushdown** inside a base file: row groups that cannot match are
skipped using the footer's statistics, and a row filter decodes the filter
columns first and the rest only for rows that pass. A slice without log files
pushes every filter; one that merges log files pushes only the record-key
filters. Conservative: a row group or row that might match is kept.
- **Row-level mask** applied to every returned batch. This is the
authoritative filter; pruning is best-effort.
Values are strings; they are cast to the target column's Arrow type at filter
time. Unparseable values (e.g. `"abc"` against `Int64`) error.
@@ -297,7 +298,7 @@ A single-use iterator returned by streaming APIs. `for
batch in stream:` or `nex
### Snapshot atomicity
-A snapshot read is pinned to one completed commit timestamp; mid-read writes
are not visible. The timeline is loaded once when `Table` / `HudiTable` is
constructed and reused for every subsequent read on that instance — commits
that land *after* construction require building a new instance to observe. When
`as_of_timestamp` is unset, "latest commit" means the latest commit in the
cached timeline, not in storage at call time.
+A snapshot read is pinned to one completed commit timestamp; mid-read writes
are not visible. The timeline is loaded once when `Table` / `HudiTable` is
constructed and reused for every subsequent read on that instance. Writes,
`compact`, and `await_compaction` through that instance refresh it; commits
made by other writers or other instances after construction require building a
new instance to observe. When `as_of_timestamp` is unset, "latest commit" means
the latest commit in the cache [...]
### Incremental semantics
diff --git a/docs/writer-design.md b/docs/writer-design.md
index af6523f4..c604b67f 100644
--- a/docs/writer-design.md
+++ b/docs/writer-design.md
@@ -37,7 +37,7 @@ design as implemented in `crates/core/src/write/`.
- Table versions 8 (Hudi 1.0.x) and 9 (Hudi 1.1.x, the default), timeline
layout v2 with completion-time instants. The one exception is
`append` on a table below version 8 (timeline layout v1), which takes
- a narrow path in `table/writer.rs`: unpartitioned append-only tables with
+ a narrow path in `write/append.rs`: unpartitioned append-only tables with
`hoodie.populate.meta.fields=false` and no metadata table, one base file
per call.
- Metadata table (MDT) with `files`, `column_stats`, `partition_stats`, and
@@ -49,7 +49,8 @@ design as implemented in `crates/core/src/write/`.
**Not yet implemented**
-- Write bindings for Python / C++ / DataFusion (the read path covers those).
+- Write bindings for C++ / DataFusion (the read path covers those; Python has
the
+ full write API).
- Clustering and cleaning as *drivers* (hudi-rs interoperates with Spark-run
clustering and cleaning, but does not schedule or execute them natively);
log compaction; compaction strategies other than "every file slice with log
@@ -176,7 +177,7 @@ deltacommit (§3.3).
| Verb | Semantics |
|------|-----------|
| `append` | Bulk-insert-like: new file groups per call, no index lookup,
size-split into `hoodie.parquet.max.file.size` buckets. Below version 8, one
base file per call on unpartitioned append-only tables (see §1). |
-| `upsert` / `upsert_with` | Index-tagged updates + inserts. Updates rewrite
their file group (COW) or append log blocks to their file slice (MOR). Inserts
pack into existing small files first. |
+| `upsert` / `upsert_with` | Index-tagged updates + inserts. Updates rewrite
their file group (COW) or append log blocks to their file slice (MOR). Inserts
pack into existing small files first on COW; on MOR they always form new file
groups (§4.3). |
| `update` | SQL-style column update on rows matching a filter. |
| `delete` / `delete_keys` | Filter- or key-based deletes. A file group left
empty still receives an empty base file version so the deletion is durable for
all readers. |
| `overwrite` | `INSERT_OVERWRITE_TABLE`: replaces all file groups via a
replacecommit. |
@@ -218,7 +219,18 @@ and assembled in plan order, so results are deterministic.
- Commit metadata records the **data schema** (no `_hoodie_*` meta fields).
- Incoming batches may differ from the table schema in nullability only
(Spark evolves table schemas to all-nullable); such batches are aligned
- rather than rejected. Name/type/order changes still error.
+ rather than rejected. Fixed-size lists are written as lists and list item
+ fields are named `element`, as Java's Avro conversion reads them back.
+ Name/type/order changes still error.
+- Record keys: one or more key fields of string, boolean, or integer type.
+ Several fields form a composite key encoded as Java's complex key
+ generator does (`field:value` pairs joined by commas). Date, decimal,
+ timestamp, and floating-point key fields are rejected, since Java writers
+ store them in more than one form (`keygen/record_key.rs`).
+- Base file footers carry the record-key bloom filter only when
+ `hoodie.parquet.bloom.filter.enabled=true` or `hoodie.index.type` names a
+ bloom index: the record index locates keys without it. A Java writer that
+ tags with a BLOOM or SIMPLE index needs it set on tables hudi-rs writes.
### 4.6 Compaction
@@ -328,7 +340,8 @@ the next one runs.
- **Incremental** reads follow Hudi 1.x completion-time semantics: instants
are selected by *completion* time in `(start, end]`; rows and MOR log blocks
are then filtered by membership in the exact selected instant set. The
- default `end` is the latest completion time. This matches the Spark reader,
+ default `end` is the greatest completion time, so it covers everything
+ committed so far. This matches the Spark reader,
including the boundary case where an instant's request time equals `start`
but its completion is later.
@@ -378,6 +391,13 @@ The acceptance gate for interop lives in
Spark-created and to an rs-created partitioned table; filtered Spark reads
with data skipping (column_stats + partition_stats pruning) must match the
same reads without it.
+- **I — Spark indexes on rs tables**: Spark upserts with SIMPLE, BLOOM, and
+ GLOBAL_BLOOM indexes on rs-written COW/MOR tables find every key rs wrote
+ and re-insert a key rs deleted without duplicates. These tables set
+ `hoodie.parquet.bloom.filter.enabled=true`, since rs leaves the footer
+ bloom filter out by default.
+- **J — Spark SQL DML on rs tables**: Spark SQL UPDATE/DELETE/INSERT on
+ rs-written COW/MOR tables in a non-default database, then rs writes on top.
- **K — rs compaction**: rs compacts an rs-written MOR table, and its handles
compact the metadata table inline every two delta commits; Spark reads the
result (snapshot and time travel, with and without the metadata table),
@@ -393,8 +413,10 @@ crates/core/src/write/
schema alignment, log/parquet file tasks
create.rs Table::create builder, hoodie.properties, MDT bootstrap
append.rs append verb, instant time generation, parquet writer props
- rewrite.rs upsert/update/delete/overwrite engines (COW rewrite groups,
- MOR delta logs), instant request/complete helpers
+ rewrite/ upsert/update/delete/overwrite engines: mod.rs (instant
+ request/complete helpers), cow.rs (COW rewrite groups),
+ mor.rs (MOR delta logs), rows.rs (row selection and merging),
+ validate.rs (write preconditions)
sizing.rs small-file packing: SizingConfig, insert bucket assignment
metadata.rs MDT updates: files, record_index, column/partition stats
metadata_compaction.rs
@@ -405,7 +427,8 @@ crates/core/src/write/
rollback.rs eager rollback of failed writes
archival.rs LSM timeline archival
lock.rs LockProvider + InProcessLockProvider
- keygen.rs record key generation
+ keygen.rs record keys and partition paths for write batches
+ (key encoding shared with readers: crates/core/src/keygen/)
crates/core/src/metadata/
payload_merger.rs HoodieMetadataPayload-equivalent record merger
