This is an automated email from the ASF dual-hosted git repository.
vinothchandar pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 0a1edf9e5548 docs(blog): streaming, write-heavy, read-heavy and
benchmark comparisons (#19273)
0a1edf9e5548 is described below
commit 0a1edf9e5548ae0191b162c22f5fbbb5aadb0673
Author: vinoth chandar <[email protected]>
AuthorDate: Tue Aug 11 15:45:07 2026 -0700
docs(blog): streaming, write-heavy, read-heavy and benchmark comparisons
(#19273)
---
...8-11-hudi-vs-iceberg-for-streaming-ingestion.md | 100 ++++++++++++++
...hudi-vs-delta-lake-for-write-heavy-workloads.md | 120 +++++++++++++++++
...08-13-hudi-vs-iceberg-performance-benchmarks.md | 109 +++++++++++++++
...d-heavy-workloads-point-lookups-with-indexes.md | 146 +++++++++++++++++++++
website/static/llms.txt | 4 +
5 files changed, 479 insertions(+)
diff --git a/website/blog/2026-08-11-hudi-vs-iceberg-for-streaming-ingestion.md
b/website/blog/2026-08-11-hudi-vs-iceberg-for-streaming-ingestion.md
new file mode 100644
index 000000000000..91ed12118dec
--- /dev/null
+++ b/website/blog/2026-08-11-hudi-vs-iceberg-for-streaming-ingestion.md
@@ -0,0 +1,100 @@
+---
+title: "Apache Hudi vs Apache Iceberg for Streaming Ingestion"
+excerpt: "Both formats accept streaming writes, but their architectures
diverge sharply under continuous ingestion. A mechanism-level comparison of
write paths, commit cadence, compaction, and concurrency."
+description: "Hudi vs Iceberg for streaming ingestion: write paths, commit
cadence, small files, compaction, and multi-writer concurrency compared at the
mechanism level."
+authors: [sivabalan]
+category: deep-dive
+image:
/assets/images/blog/2025-07-02-Lakehouse-Architecture-apache-hudi-and-apache-iceberg.png
+tags:
+- comparison
+- apache iceberg
+- streaming
+- data lakehouse
+---
+
+Both Apache Hudi and Apache Iceberg can ingest streaming data, but they are
built around different assumptions about how often a table is written to.
Hudi's architecture is streaming-native: Merge-on-Read tables absorb updates as
lightweight log appends, commits land every few minutes with automatic file
sizing, compaction and clustering run asynchronously without ever blocking
ingestion, and non-blocking concurrency control lets multiple writers land data
into the same file groups. Iceb [...]
+
+<!--truncate-->
+
+That is the short answer to "Hudi vs Iceberg for streaming." The rest of this
post walks through the mechanisms behind it: what continuous ingestion actually
stresses in a table format, how each format's write path behaves under it, what
minute-level commits do to each format's metadata, and how each keeps readers
fast while the writers never stop. The goal is a fair, mechanism-level
comparison — both are excellent open table formats, and the differences that
matter here are architectura [...]
+
+## What Continuous Ingestion Stresses in a Table Format
+
+Batch and streaming workloads stress a table format in very different places.
A nightly batch job commits once, writes well-sized files, and leaves a long
quiet window for maintenance. A streaming pipeline inverts every one of those
properties:
+
+- **Commit frequency.** A pipeline targeting minute-level freshness commits
hundreds of times a day, every day. Every structure the format touches per
commit — metadata files, manifests, timeline entries — gets exercised at that
rate.
+- **Small files.** Each micro-commit writes only a sliver of data. Without
countermeasures, a week of minute-level commits leaves behind tens of thousands
of files, and query planning and scan performance degrade with file count.
+- **Write amplification on updates.** Streams from CDC sources are
update-heavy. If applying an update means rewriting the columnar file that
contains the old value, a trickle of changes turns into a torrent of rewritten
bytes.
+- **Concurrent maintenance.** There is no quiet window. Compaction,
clustering, and cleaning must run *while* ingestion continues, without blocking
it and without being starved by it.
+- **Exactly-once delivery.** The format's commit protocol has to compose with
Flink checkpoints or Spark Structured Streaming epochs so that failures and
retries never duplicate or drop records.
+
+These pressures are exactly what a [streaming data
lake](/blog/2026/07/21/what-is-a-streaming-data-lake) has to absorb, and they
are where the two formats' designs part ways.
+
+## The Write Path Compared
+
+**Hudi.** Hudi's [Merge-on-Read table type](/docs/table_types) organizes data
into file groups, each holding a columnar base file plus a set of row-oriented
delta log files. When a streaming update arrives, Hudi's indexes locate the
file group that holds the record and *append* the change to that group's log
file. No base file is rewritten on the hot path — write amplification for a
file group is proportional to the records that changed, not the size of the
files they live in. Inserts ar [...]
+
+**Iceberg.** Iceberg's write path is snapshot-oriented: every commit adds new
immutable data files and produces a new snapshot — a new manifest list, new or
rewritten manifests, and a new table metadata file. For appends this is clean
and cheap per commit. For streaming upserts, Iceberg's merge-on-read mode
writes *delete files* alongside data files: equality deletes (typical for Flink
CDC pipelines) or position deletes mark old rows as superseded, and the new
values land in fresh data f [...]
+
+The structural difference: Hudi routes changes *into* an existing file-group
layout via its indexes and keeps files sized as it writes; Iceberg accretes new
files and metadata per commit and restores layout health through external
compaction.
+
+## Commit Cadence and Its Costs
+
+Consider what one day of minute-level commits — roughly 1,440 of them — does
to each format's metadata.
+
+In Hudi, each commit adds an entry to the [timeline](/docs/timeline), and
file-level statistics and listings land in Hudi's internal metadata table,
which is itself an MOR table designed to absorb frequent small mutations
cheaply. Older timeline entries are archived automatically into a compact
timeline history, so the active timeline that writers and readers consult stays
bounded no matter how long the pipeline runs. Frequent commits are the
*assumed* operating mode — the mechanisms tha [...]
+
+In Iceberg, each commit produces a new snapshot: a metadata JSON file, a
manifest list, and one or more manifests. At minute-level cadence, that is
1,440 snapshots a day, each retaining pointers to the manifests of its
predecessors until `expire_snapshots` runs. Manifests accumulate and fragment,
planning cost grows with manifest count, and the catalog's atomic pointer swap
becomes a serialization point that every commit contends on. None of this is a
correctness problem — Iceberg handle [...]
+
+The honest summary: both formats *can* commit every minute. Hudi's metadata
structures were designed around that rate; Iceberg's were designed around
fewer, larger commits, and sustained high cadence shifts real work onto the
maintenance schedule.
+
+## Keeping Readers Fast While Writing Continuously
+
+Log appends and delete files both defer merge work — the question is who pays
for it, and when.
+
+Hudi gives readers an explicit choice via its [query
types](/docs/table_types#query-types). *Snapshot queries* merge base files with
delta logs at read time and see the freshest data. *Read-optimized queries*
read only the compacted columnar base files, trading a bounded amount of
freshness for pure-Parquet scan performance. Async compaction runs continuously
in the background — scheduled and executed without pausing ingestion,
coordinated through MVCC on the timeline — so the merge debt [...]
+
+Iceberg readers on an upsert stream pay read amplification from delete files:
every scan of a data file must also apply the equality and position deletes
that reference it, until a compaction rewrites the affected files. There is no
built-in equivalent of the read-optimized query — the practical lever is
running `rewrite_data_files` (and delete-file compaction) often enough that
delete-file overhead stays tolerable. Format v3's deletion vectors improve the
mechanics of position deletes c [...]
+
+## Multi-Writer and Table Services Concurrency
+
+Streaming deployments rarely stay single-writer: a backfill job lands next to
the live pipeline, or compaction runs as a separate job from ingestion.
+
+Hudi's [concurrency control](/docs/concurrency_control) distinguishes writers
from table services. A single writer with async compaction, clustering, and
cleaning needs no external locks at all — MVCC coordinates ingestion and
services in-process, which covers the most common streaming deployment with
essentially zero concurrency configuration. For true multi-writer setups, Hudi
offers file-level optimistic concurrency control, and — for streaming semantics
— *non-blocking concurrency co [...]
+
+Iceberg uses optimistic concurrency for everything: each committer builds its
snapshot, then attempts an atomic pointer swap through the catalog; on conflict
it re-validates and retries. This is a clean, well-proven model, and for
writers touching disjoint files retries are cheap metadata operations. The
friction appears under sustained high commit rates: the ingestion job, the
compaction job, and the snapshot-expiry job are all competing committers to the
same table, and a long-running [...]
+
+## Engine Integrations for Streaming
+
+Both projects integrate with the major streaming engines, and it is worth
being factual rather than dismissive on either side.
+
+Iceberg has a mature Flink connector (checkpoint-aligned commits, upserts via
equality deletes), a Spark Structured Streaming sink, and a Kafka Connect sink.
For append-centric event pipelines at moderate checkpoint intervals, these work
well.
+
+Hudi's streaming surface is broader and more opinionated: the Flink writer
with async compaction in the same job, Spark Structured Streaming, the Kafka
Connect sink, and [Hudi Streamer](/docs/hoodie_streaming_ingestion) — a
complete ingestion utility with continuous mode, Kafka/DFS/CDC sources,
checkpoint management, and exactly-once delivery built in. The Flink
integration in particular has seen sustained recent investment: Hudi 1.1
[rebuilt the Flink write path around native RowData](/ [...]
+
+## A Decision Framework
+
+Setting aside benchmarks (run your own, on your workload), the architecture
suggests a straightforward split:
+
+- **Append-only event streams at moderate cadence** — clickstreams or logs
committed every 10–15 minutes, no updates. Either format works; Iceberg is a
perfectly good choice here, especially if your ecosystem is already
Iceberg-centric. Budget for scheduled compaction and snapshot expiry, and size
checkpoint intervals with metadata growth in mind.
+- **Upsert-heavy streams (CDC, mutable entities)** — this is where the
write-path difference compounds. Hudi's indexed log-append writes, delete
handling, and async compaction were built for exactly this pattern.
+- **Sub-10-minute freshness targets** — minute-level commit cadence plays to
Hudi's bounded timeline and mutation-friendly metadata table; on Iceberg it
demands a proportionally aggressive maintenance regimen.
+- **Hands-off operations** — if no team will babysit compaction schedules,
Hudi's self-managing posture (automatic file sizing at write time, async table
services inside the writer process) removes a whole category of orchestration.
+
+## It's Not Either/Or: Interoperability via Apache XTable
+
+The comparison above frames a *writer-side* decision — and it does not have to
constrain your readers. [Apache XTable](https://xtable.apache.org) (incubating)
translates table metadata between Hudi, Iceberg, and Delta Lake without copying
or rewriting data files. A common pattern is to ingest with Hudi — taking the
streaming-native write path, indexing, and async table services — and expose
the same data as an Iceberg table to catalogs and engines that expect Iceberg,
with metadata kept [...]
+
+## Conclusion
+
+Iceberg and Hudi are both capable, openly governed table formats, and both can
sit at the end of a streaming pipeline. The difference is what each was
designed to assume. Iceberg assumes commits are relatively infrequent and
maintenance runs between them; streaming is supported, and works, but small
files, delete files, and snapshot metadata become an operational program you
run alongside the pipeline. Hudi assumes the writers never stop: MOR log
appends keep update costs proportional to [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {question: 'Can Apache Iceberg handle streaming writes?', answer: 'Yes.
Iceberg has a mature Flink connector, a Spark Structured Streaming sink, and a
Kafka Connect sink, and streaming appends work well at moderate commit cadence.
The caveat is operational: every commit creates a new snapshot with new
metadata and manifest files, and upserts add delete files that readers must
merge on every scan, so sustained high-frequency streaming requires regularly
scheduled compaction, manifest re [...]
+ {question: 'Why does Apache Hudi handle small files better than Iceberg?',
answer: 'Hudi addresses small files at write time rather than after the fact.
The writer automatically bin-packs new inserts against existing under-sized
file groups, so every commit works toward well-sized files, and updates are
appended to log files within existing file groups instead of creating new small
files. Iceberg writes new immutable files on every commit and relies on a
separately scheduled rewrite_da [...]
+ {question: 'Which table format works best with Apache Flink for streaming
ingestion?', answer: 'Both have solid Flink connectors, but Hudi offers a
deeper streaming integration: async compaction runs inside the same Flink job,
recent releases rebuilt the write path around Flink-native row types to cut
serialization overhead, and Hudi 1.2 added Record Level Index support so global
upsert lookups use the table-backed index instead of large Flink keyed state.
Iceberg Flink pipelines handl [...]
+ {question: 'Does frequent committing hurt Iceberg tables?', answer:
'Frequent commits are safe for correctness but costly operationally. Each
Iceberg commit produces a new metadata file, manifest list, and manifests, so
minute-level cadence yields well over a thousand snapshots per day whose
metadata accumulates until snapshot expiry runs, and concurrent committers
contend on the catalog pointer swap. This is why Iceberg streaming guidance
typically recommends longer checkpoint interva [...]
+ {question: 'Can I use Hudi for ingestion and still serve Iceberg readers?',
answer: 'Yes. Apache XTable (incubating) translates table metadata between
Hudi, Iceberg, and Delta Lake without copying or rewriting data files, either
as a one-time conversion or as continuous incremental sync. A common pattern is
to ingest with Hudi to get the streaming-native write path and async table
services, then expose the same data files as an Iceberg table to catalogs and
engines that expect Iceberg.'},
+]} />
diff --git
a/website/blog/2026-08-12-hudi-vs-delta-lake-for-write-heavy-workloads.md
b/website/blog/2026-08-12-hudi-vs-delta-lake-for-write-heavy-workloads.md
new file mode 100644
index 000000000000..72a8bba9e287
--- /dev/null
+++ b/website/blog/2026-08-12-hudi-vs-delta-lake-for-write-heavy-workloads.md
@@ -0,0 +1,120 @@
+---
+title: "Apache Hudi vs Delta Lake for Write-Heavy Workloads"
+excerpt: "A mechanism-level comparison of how Hudi and Delta Lake handle
update- and delete-heavy tables: indexing vs file-scan merge planning, MOR logs
vs Parquet rewrites, and how their costs scale as write rates climb."
+description: "How Apache Hudi and Delta Lake handle write-heavy workloads:
indexed upserts and MOR logs vs MERGE joins and file rewrites, compaction,
concurrency, streaming."
+authors: [sivabalan]
+category: deep-dive
+image:
/assets/images/blog/2024-05-27-apache-hudi-vs-delta-lake-choosing-the-right-tool-for-your-data-lake-on-aws.png
+tags:
+- comparison
+- delta lake
+- upsert
+- performance
+---
+
+For update- and delete-heavy tables, the core architectural difference is
this: Apache Hudi locates the records being changed through pluggable
[indexes](/docs/indexes) — including a record-level index that maps each key to
its file group — and can absorb changes into Merge-on-Read delta logs instead
of rewriting columnar files, while Delta Lake locates records by joining the
incoming batch against the table's data files (pruned by file statistics and,
more recently, softened by deletion [...]
+
+This post compares the two systems at the mechanism level: how each plans a
merge, what each does at write time, who cleans up afterward, and how they
behave under concurrency and streaming. It deliberately avoids benchmark
numbers — hardware, versions, and configurations change too fast for them to
stay honest — and focuses instead on the cost model each architecture implies.
+
+## What "Write-Heavy" Actually Means
+
+"Write-heavy" is not one workload; it is a cluster of characteristics that
stress a table format in different ways:
+
+- **High update/delete ratios.** The batch being written is mostly *changes to
existing records* rather than new inserts — CDC streams mirroring an OLTP
database, order-status updates, inventory corrections, GDPR deletes.
Insert-only workloads are easy for every format; it is mutation that separates
them.
+- **Frequent commits.** Writes land every few minutes (or faster) rather than
every few hours, so per-commit overheads — merge planning, file rewriting,
metadata churn — are paid dozens or hundreds of times a day.
+- **Random-key updates vs recent-partition updates.** If updates cluster in
the newest partitions (event tables), any format can prune its way to a small
working set. If updates scatter across the whole keyspace (dimension tables,
user profiles, unpartitioned tables), pruning breaks down and the mechanism for
*finding* records dominates cost.
+- **Concurrent writers.** Multiple pipelines — an ingestion job, a backfill, a
GDPR delete job — writing the same table at once, where conflict handling
decides how much work gets thrown away and retried.
+
+Keep these four dimensions in mind; the two systems diverge on each of them by
different amounts.
+
+## The MERGE Path, Compared
+
+**Delta Lake** expresses row-level mutation primarily through `MERGE INTO`.
The engine joins the source batch against the target table to identify which
data files contain matching rows, using file-level min/max statistics (and data
layout from Z-ordering or liquid clustering) to prune files where possible.
Matched files are then rewritten: a file containing even one matched row is
read, merged, and written out again as a new Parquet file. Deletion vectors
(Delta's merge-on-read-style fe [...]
+
+**Hudi** treats [upsert as a first-class write
operation](/docs/write_operations), not a SQL statement compiled into a join.
Incoming records carry a record key; the write path runs an [index
lookup](/docs/indexes) to tag each record as an insert or an update and route
it to the file group that already holds its key. What happens next depends on
the [table type](/docs/table_types): a Copy-on-Write (COW) table rewrites the
affected base files, much like Delta; a Merge-on-Read (MOR) table [...]
+
+The structural difference: Delta's merge planning cost lives in a join against
data files; Hudi's lives in an index lookup against metadata. As table size
grows and update keys scatter, the first grows with the table; the second grows
with the change batch (for non-global indexes) or stays a sharded point-lookup
(for the record-level index).
+
+## Indexing: The Core Difference
+
+Hudi maintains a persistent mapping from record keys to file groups and lets
you [pick the index](/docs/indexes) that matches your write pattern:
+
+- **Bloom index**: bloom filters plus key-range pruning, stored in file
footers or centrally in the metadata table. Excellent when keys have ordering
(e.g., timestamp-prefixed event keys), so most files are pruned before any data
is read.
+- **Record-level index (RLI)**: an exact key-to-file-group mapping in Hudi's
metadata table, hash-sharded for scale. It turns "which file holds this key?"
into a point lookup, which matters most on large tables with random-key updates
— precisely where probabilistic pruning fails.
+- **Bucket index**: hashes keys directly to file groups, eliminating the
lookup step entirely at the cost of a fixed (or consistent-hashed) bucket
layout. This is the workhorse for very high-throughput streaming upserts,
especially with Flink.
+- **Simple/global variants**: lean joins against existing keys, with global
versions that enforce uniqueness across partitions.
+
+Delta Lake's counterpart — and it is a real counterpart, not an absence — is
**data skipping**: per-file min/max statistics collected in the transaction
log, made more effective by clustering the data so related keys co-locate
(Z-ordering historically, liquid clustering more recently). When keys correlate
with layout, statistics prune candidate files well and MERGE touches little.
The honest framing is that Delta prunes *files by value ranges*, while Hudi
locates *records by key*. The fo [...]
+
+## Write Amplification and File Management
+
+Once records are located, the second question is how many bytes hit storage
per byte of change.
+
+On **Delta**, a MERGE that touches a file rewrites the file (deletion vectors
defer this, as noted). Write amplification is therefore proportional to the
number of *files* touched, and random small updates against large files are the
worst case. Compaction is handled by `OPTIMIZE` (with Z-order/clustering
options) and auto-compaction/optimized-writes features that coalesce small
files; on Databricks much of this is managed for you (predictive optimization),
while open-source deployments [...]
+
+On **Hudi**, a COW table has the same rewrite profile — with one mitigation:
Hudi's automatic file sizing packs inserts into under-sized file groups on
every write, keeping file counts and rewrite units under control without a
separate job. A MOR table changes the equation more fundamentally: updates are
appended to logs, so write amplification per commit is proportional to *records
changed*, not files touched; the rewrite cost is batched and amortized by
[compaction](/docs/compaction), [...]
+
+The operational difference is who runs the cleanup and whether writers wait
for it. Hudi ships cleaning, compaction, clustering, and file sizing as
self-managing services wired into the write path; Delta provides the
equivalents as commands and (on Databricks) managed features. For a write-heavy
table, the key property is that MOR moves the expensive columnar rewrite *off*
the ingestion path entirely — a batch of deletes trickling in all day can be
absorbed cheaply in logs and reconciled [...]
+
+## Concurrency Under Heavy Writes
+
+Both systems use optimistic concurrency control (OCC) between writers, and
both resolve conflicts at commit time — but the failure modes under heavy write
traffic differ.
+
+**Delta** detects conflicts by transaction-log versions: concurrent commits
that logically overlap (e.g., two MERGEs that may have read files the other
rewrote) surface as `ConcurrentAppend`/`ConcurrentDeleteRead`-class exceptions,
and the losing transaction retries. Partitioning and careful predicate scoping
reduce collisions, and this works well when writers touch disjoint partitions.
+
+**Hudi** offers file-level OCC with the same optimistic semantics (two writers
touching disjoint file groups both succeed), plus early conflict detection that
can abort a doomed writer mid-write rather than at commit, saving the wasted
compute. More distinctively, Hudi's [concurrency
model](/docs/concurrency_control) separates *writers* from *table services*:
compaction and clustering run under MVCC without competing in the OCC conflict
path, so a heavy ingestion writer is not fighting i [...]
+
+## Streaming Writes
+
+Both formats ingest from Spark Structured Streaming, and both can sustain
minute-level micro-batches. The differences are in what surrounds that path.
Hudi grew up as a streaming ingestion system (built at Uber to keep lake tables
in sync with upstream databases), and it shows in the toolchain: Hudi Streamer
is a ready-made continuous ingestion service with source connectors,
schema-registry integration, checkpointing, and async table services in one
process; MOR tables give streaming wr [...]
+
+For a dated but methodologically transparent data point on batch write/query
performance, see the 2022 [TPC-DS benchmark
comparison](/blog/2022/06/29/Apache-Hudi-vs-Delta-Lake-transparent-tpc-ds-lakehouse-performance-benchmarks)
— treat it strictly as a historical reference, since both systems have shipped
major releases since then.
+
+## A Decision Framework
+
+Delta Lake is a workable choice when:
+
+- You are **Databricks-centric** — the managed platform runs the
compaction/clustering machinery for you, and the ecosystem integration is
tightest there.
+- **Update rates are moderate** and mostly cluster in recent partitions, so
statistics-based pruning keeps MERGE cheap.
+- Your mutation pattern is **periodic batch MERGE** (hourly/daily), where
per-commit overhead is paid rarely.
+
+Hudi pulls ahead as workloads get write-heavier:
+
+- **Random-key upserts at scale** — dimension tables, user profiles,
unpartitioned tables — where the record-level or bucket index bounds locate
cost and MOR bounds rewrite cost.
+- **CDC replication** of operational databases, where upsert-as-a-primitive,
ordering-field-based merging of out-of-order events, and incremental pulls map
directly onto the problem.
+- **Sub-hour freshness targets** with frequent commits, where async compaction
keeps the ingestion path light.
+- **Open/non-Databricks deployments** — Spark, Flink, EMR, Dataproc, on-prem —
where Hudi's self-managing table services replace what a managed platform would
otherwise do for you.
+- **Multi-writer mutation**, where NBCC avoids OCC retry churn.
+
+## It's Not Either/Or: Interoperability via Apache XTable
+
+Choosing Hudi as the write layer does not cut you off from Delta-reading
tools. [Apache XTable](https://xtable.apache.org/) (incubating) translates
table metadata between Hudi, Delta Lake, and Iceberg over the *same* Parquet
data files — no data copying, just an additional metadata representation. You
can write with Hudi's indexed, MOR-backed ingestion path and expose the table
as Delta for Databricks SQL, or as Iceberg for engines that prefer it, keeping
a one-copy story much like Delta [...]
+
+## Conclusion
+
+Delta Lake and Apache Hudi can both run update-heavy tables; the question is
how their costs scale as "update-heavy" gets heavier. Delta plans merges by
joining against data files and applies them by rewriting Parquet — a model that
stays cheap while pruning works and update rates are moderate, and that
Databricks increasingly automates. Hudi plans merges through indexes and can
apply them as log appends — a model whose write cost tracks the size of the
change rather than the size of the [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {
+ question: 'Is Hudi faster than Delta Lake for updates?',
+ answer: 'It depends on the update pattern, so be wary of any blanket
claim. For updates that scatter randomly across a large table, Hudi\'s
record-level and bucket indexes locate affected file groups without joining
against data files, and Merge-on-Read absorbs changes as log appends instead of
Parquet rewrites, which typically means lower write latency and less compute
per commit. For moderate update rates clustered in recent partitions, Delta\'s
statistics-based pruning keeps MERGE [...]
+ },
+ {
+ question: 'Does Delta Lake have an index?',
+ answer: 'Delta Lake does not maintain a record-level index that maps keys
to files. It relies on per-file min/max statistics in the transaction log for
data skipping, made more effective by clustering the data with Z-ordering or
liquid clustering, and deletion vectors reduce immediate file rewrites for
deletes and updates. This works well when update keys correlate with data
layout, but degrades toward scanning key columns across the table when updates
are random. Hudi instead mainta [...]
+ },
+ {
+ question: 'Can I use Apache Hudi outside Databricks?',
+ answer: 'Yes, and that is one of its main draws. Hudi is a community-run
Apache project that works with open-source Spark, Flink, Presto, Trino, and
cloud services like Amazon EMR, Google Dataproc, and AWS Glue, with no managed
platform required. Its table services, such as compaction, clustering,
cleaning, and file sizing, are built into the writers and run automatically,
covering the operational work a managed platform would otherwise do. Delta Lake
is also open source, but several [...]
+ },
+ {
+ question: 'What is the difference between Hudi Merge-on-Read and Delta
deletion vectors?',
+ answer: 'Both defer expensive file rewrites, but at different granularity.
Deletion vectors mark individual rows in a Parquet file as removed, so a delete
or update avoids an immediate rewrite, while the new version of an updated row
is still written to a data file and the marked file is eventually rewritten by
OPTIMIZE. Hudi\'s Merge-on-Read appends the full changed records to delta log
files attached to each file group, so arbitrary updates, not just row removals,
are absorbed chea [...]
+ },
+ {
+ question: 'Can Delta Lake tools read a Hudi table?',
+ answer: 'Yes, through Apache XTable, an incubating project that translates
table metadata between Hudi, Delta Lake, and Iceberg without copying data
files. You write with Hudi and run XTable sync to produce Delta transaction log
metadata over the same Parquet files, which Delta-compatible engines can then
read like any Delta table. This gives a one-copy, multi-format story similar to
Delta\'s UniForm feature, while keeping Hudi\'s indexing and Merge-on-Read
machinery on the write path.',
+ },
+]} />
diff --git a/website/blog/2026-08-13-hudi-vs-iceberg-performance-benchmarks.md
b/website/blog/2026-08-13-hudi-vs-iceberg-performance-benchmarks.md
new file mode 100644
index 000000000000..e7a45806fe7c
--- /dev/null
+++ b/website/blog/2026-08-13-hudi-vs-iceberg-performance-benchmarks.md
@@ -0,0 +1,109 @@
+---
+title: "Apache Hudi vs Apache Iceberg Performance: What Benchmarks Show, and
What We Measured"
+excerpt: "A Hudi maintainer's guide to the Hudi vs Iceberg benchmark record:
what published TPC-DS and upsert benchmarks show, plus reproducible LakeLoader
results for incremental writes."
+description: "What published Hudi vs Iceberg benchmarks show, why results
differ, reproducible LakeLoader incremental-write results, and how to run a
fair benchmark yourself."
+authors: [sivabalan]
+category: deep-dive
+image:
/assets/images/blog/2023-08-28-Delta-Hudi-Iceberg-A-Benchmark-Compilation.png
+tags:
+- comparison
+- apache iceberg
+- performance
+---
+
+If you searched "Hudi vs Iceberg performance benchmark" hoping for a single
number, here is the honest answer: no single published benchmark settles the
question, because results depend overwhelmingly on workload shape (append-only
scans versus update-heavy ingestion), on configuration parity between the
systems, and on the versions tested. What the record *does* show consistently
is that on read-dominated benchmarks like TPC-DS, table-format overheads
converge to within noise of each ot [...]
+
+This post is written from a Hudi maintainer's seat, so let us be direct about
the design-level conclusion we hold: **for incremental, update-heavy writes, we
believe Hudi will give you the best write performance of the major open table
formats.** Its record-level indexing and merge-on-read design bound write cost
to the size of the change rather than the size of the table, and its
concurrency model blocks writers less when table services run alongside
ingestion — a real advantage if you [...]
+
+## Why Lakehouse Benchmarks Are Hard to Get Right
+
+Table-format benchmarks are unusually easy to get wrong, and the failure modes
recur so reliably that they are worth naming up front.
+
+**Configuration parity is almost never achieved.** The formats ship with
different defaults because they optimize for different things. Hudi's default
write operation is `upsert`, which pays for index lookups and record merging so
mutations are cheap later; Iceberg and Delta default to append-style writes.
Benchmark a bulk load on defaults and you are not measuring format efficiency —
you are measuring whose defaults happen to match the workload. Vinoth Chandar's
April 2022 ["Corrections [...]
+
+**Default-versus-tuned is an editorial choice, not a neutral one.** Out of the
box, Hudi turns on table services and indexing that cost something at write
time and pay off at read and update time. A benchmark that measures only the
write pays the cost and never collects the payoff. Whether the right comparison
is "both on defaults" or "both tuned by experts" is a judgment call — a
credible benchmark states which it made and why.
+
+**Versions age fast.** Every benchmark below names specific versions — Hudi
0.11, Hudi 0.14, Delta 1.2 — that have since been superseded by releases with
materially different write paths (Hudi 1.x being the obvious example). A 2022
result is a historical data point, not a current one.
+
+**Provenance is context, not a verdict.** Most published lakehouse benchmarks
— including the ones in this post — come from people close to one of the
projects, because the people motivated to measure a system are usually the
people building on it. That is normal; it just means reproducibility is the
thing to check. Published code, configs, versions, and hardware let you verify
a result instead of taking it on trust, and that standard applies to our
numbers below as much as anyone else's.
+
+## What TPC-DS-Style Benchmarks Show
+
+TPC-DS is a decision-support benchmark: load the data once, then run about a
hundred analytical queries. It is scan- and query-dominated, which means it
exercises the part of a table format where the formats are most alike — listing
files, pruning with statistics, and reading Parquet.
+
+The most instructive public episode is from mid-2022. A benchmark by
Databeans, shown during a Databricks keynote, reported Hudi dramatically slower
than Delta and Iceberg on TPC-DS. Onehouse's June 2022 response, ["Transparent
TPC-DS Lakehouse Performance
Benchmarks"](/blog/2022/06/29/Apache-Hudi-vs-Delta-Lake-transparent-tpc-ds-lakehouse-performance-benchmarks)
by Alexey Kudinkin, re-ran the comparison on EMR 6.6.0 with Spark 3.2.0,
testing Hudi 0.11.1 against Delta 1.2.0 and 2.0.0rc1 [...]
+
+That episode is dated — those version numbers are ancient now — but its
structural finding has held: on read-heavy TPC-DS-style workloads, comparably
configured table formats land within noise of each other, because the query
engine and the Parquet scan dominate, not the table metadata layer. Kyle
Weller's 2023 compilation (next section) reached the same conclusion across
multiple 2022 TPC-DS runs, noting that Iceberg trailed in those particular
tests while Hudi and Delta were comparable [...]
+
+If TPC-DS is your workload — append-only loads, heavy analytical scans — the
published evidence says the format choice is unlikely to be your performance
bottleneck, and you should weigh operational and ecosystem factors instead.
+
+## What Mutation-Heavy Comparisons Show
+
+The picture inverts once the workload mutates data continuously, because now
the write path — how a format finds the records to update, absorbs deletes, and
amortizes merge costs — is the whole game, and here the formats differ
architecturally rather than cosmetically.
+
+The public record on this side is thinner but pointed. Kyle Weller's August
2023 ["Delta, Hudi, Iceberg — A Benchmark
Compilation"](/blog/2023/08/28/Delta-Hudi-Iceberg-A-Benchmark-Compilation)
recounts a Walmart Global Tech evaluation on two real production scenarios
rather than synthetic queries: a late-arriving-data workload with heavy
read/write amplification, and row-level upserts fed by CDC from Cassandra. In
Walmart's runs (on the then-current, now-old versions they tested), Delta [...]
+
+Why the divergence? Mechanism. To update a record, a writer must first find
it. Hudi ships a [multi-modal indexing subsystem](/docs/indexes) for exactly
this, and the impact of index choice alone is measurable: Soumil Shah's October
2023 [upsert evaluation on Hudi 0.14 with Spark
3.4.1](/blog/2023/10/29/UPSERT-Performance-Evaluation-of-Hudi-0-14-and-Spark-3-4-1-Record-Level-Index-Global-Bloom-Global-Simple-Indexes)
measured the same upsert workload under three of Hudi's global index opti [...]
+
+## The Benchmark Landscape, Compiled
+
+Weller's 2023 compilation remains the best single map of the public record. In
brief, the entries and their provenance:
+
+- **Databeans TPC-DS (June 2022)**, surfaced at a Databricks keynote; showed
Hudi far behind. Corrected by Onehouse for the configuration issues described
above; corrected results showed Hudi and Delta comparable, Iceberg trailing on
those versions.
+- **Brooklyn Data TPC-DS (2022)**, commissioned by Databricks covering Delta
and Iceberg; Onehouse extended it with Hudi. Same pattern on 2022 versions:
Hudi and Delta comparable, Iceberg slower in that run.
+- **Microsoft LST-Bench (paper published May 2023)**, a research effort
notable for methodology rather than a scoreboard: it extends TPC-DS with
mutations, concurrency, and maintenance phases, and proposes longevity metrics
— a direct acknowledgment that load-then-query benchmarks miss what matters for
these systems.
+- **Walmart Global Tech's production-scenario evaluation**, described above —
the clearest published mutation-heavy comparison, and the one whose results
diverge most sharply from the TPC-DS consensus.
+
+Where the results conflict — Iceberg competitive-to-trailing on scans in some
runs, not completing Walmart's write workloads — the conflict itself is the
finding: the benchmarks were measuring different mechanisms. Weller's own
bottom line is that performance benchmarks rarely represent real-life workloads
and users should run their own. We agree, and we took that advice ourselves.
+
+## What We Measured Ourselves: Incremental Writes with LakeLoader
+
+The biggest gap in the public record is mutation-heavy data on current
versions, so we ran our own comparison. All workloads used
[LakeLoader](https://github.com/onehouseinc/lake-loader), an open-source,
format-agnostic benchmarking framework that generates parameterized change
streams (scale, skew, update/insert mix) and applies identical rounds of
changes to each table format — the workload definition is shared across
systems, so differences in results come from the format's write path [...]
+
+In our runs, we observed:
+
+| Workload | What it models | What we observed |
+|---|---|---|
+| FACT, 10 TB, partitioned, skewed updates (Zipfian, 90% updates / 10%
inserts) | Time-partitioned fact tables (events, transactions) with
late-arriving updates | Hudi averaged ~4x lower incremental write latency than
Iceberg and ~6x lower than Delta Lake |
+| Merge-on-read, wide 100-column table, sparse partial updates (~20% of
columns per update) | Dimension tables — user profiles, account masters — where
updates touch a few columns | Hudi averaged ~8x lower steady-state latency than
Iceberg and ~5x lower than Delta Lake, by appending only column-level deltas |
+| Merge-on-read ingestion with concurrent async compaction | Streaming/CDC
pipelines with background maintenance under ingestion SLAs | Hudi sustained
continuous ingestion throughout; Iceberg and Delta Lake writes hit
snapshot-conflict retries during compaction windows |
+
+The mechanism behind the FACT numbers is the one this post keeps returning to:
Hudi's [record-level index](/docs/indexes) resolves which file groups each
batch touches directly, so per-commit shuffle stays on the order of the
incremental input (hundreds of megabytes in these runs), while the join-based
tagging paths in the other two formats scanned and shuffled hundreds of
gigabytes of the target table per commit. That is the same write-path
architecture difference the Walmart evaluation [...]
+
+Apply the same discipline to these numbers that this post asks of every other
benchmark. They are specific to these versions, this cluster shape, and these
workload parameters, and they say nothing about scan-heavy or append-only
workloads, where we expect the formats to land close together. The difference
is that you do not have to take our word for anything: the workload definitions
and run parameters live in LakeLoader, and you can rerun these workloads
yourself on your own hardware, [...]
+
+## How to Run a Benchmark That Actually Means Something
+
+The evergreen part of this post. If you benchmark Hudi against Iceberg for a
real decision, the published record suggests this checklist:
+
+1. **Benchmark your workload shape, not TPC-DS.** If your tables mutate, your
benchmark must mutate — run sustained upsert/delete phases against a loaded
table, not just the initial load.
+2. **Enforce configuration parity.** Same compression codec, same target file
sizes, same Spark/engine version and cluster. Use each format's *documented
recommendation* for the operation at hand (e.g., Hudi `bulk_insert` for
one-time loads, `upsert` for mutation phases).
+3. **State the index and delete strategy.** For Hudi, which index; for
Iceberg, copy-on-write vs merge-on-read and the delete-file mode. These are the
largest levers on the write path.
+4. **Include table-service costs.** Run compaction, clustering, and cleaning
inside the measured window — or report them separately — for both systems. A
write time that excludes the compaction it necessitates is fiction.
+5. **Separate cold and warm runs**, and run queries *during* and *after*
ingestion, not only on a freshly optimized table.
+6. **Measure over time, not one pass.** The LST-Bench insight: performance
after 100 commits differs from performance after one.
+7. **Publish versions, configs, and code.** If you cannot reproduce it,
neither can anyone who disagrees with it.
+
+## Questions to Ask of Any Published Benchmark
+
+The recurring gaps above explain most of the disagreement between published
results, so check for them systematically — in our numbers as much as anyone
else's. Were both systems given documented, workload-appropriate
configurations, or defaults for one and tuning for the other? Are versions
stated, and were they current at publication? Does a "write performance" test
include a sustained mutation phase, or only the initial load? Does the measured
window include the compaction the writes [...]
+
+## The Honest Bottom Line
+
+Choose by workload mechanism fit, then validate with your own benchmark. If
your workload is append-mostly analytics, the published record says format
overheads converge and your decision should rest on ecosystem, catalogs, and
operations. If it is mutation-heavy — CDC, streaming upserts, deletes at scale
— the write-path architecture differs materially, and the published record, the
design analysis, and our own reproducible runs all point the same way: Hudi's
indexed write path and non- [...]
+
+One more option the 2022-era benchmark wars did not have: you no longer have
to choose blind. [Apache XTable](https://xtable.apache.org) (incubating)
translates table metadata between Hudi and Iceberg without rewriting data
files, so you can write with one format and expose the other — or A/B both
against the same data — as described in [our XTable interoperability
post](/blog/2026/07/28/using-hudi-with-apache-iceberg-via-xtable).
+
+## Conclusion
+
+The published Hudi-versus-Iceberg record is smaller and older than the volume
of opinion written on top of it. What it supports: read-heavy benchmarks show
comparably configured formats within noise of each other (Onehouse, 2022, on
2022 versions); mutation-heavy evaluations show large, mechanism-driven
differences favoring purpose-built write paths (Walmart via Weller, 2023, on
the versions then tested); configuration parity gaps have distorted results in
both directions (Chandar, 2022) [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {question: 'Is Apache Hudi faster than Apache Iceberg?', answer: 'For
incremental, update-heavy writes — CDC, streaming upserts, mutable tables — yes
in our reproducible runs: LakeLoader benchmarks on Spark 3.5 testing Hudi 1.1.1
against Iceberg 1.10.0 measured roughly 4x lower incremental write latency for
Hudi on a 10 TB skewed fact-table workload, driven by Hudi\'s record-level
index avoiding full-table-scan joins, with larger gaps on merge-on-read tables
with sparse column-level up [...]
+ {question: 'Why do Hudi vs Iceberg benchmark results disagree so much?',
answer: 'Because they measure different things under different conditions.
Scan-heavy benchmarks exercise the layer where the formats are most alike,
while update-heavy benchmarks exercise indexing, merge-on-read, and delete
handling, where they differ architecturally. Configuration disparities, such as
mismatched compression codecs or using Hudi\'s upsert where bulk_insert is
recommended, have also skewed publish [...]
+ {question: 'What is the best benchmark for comparing table formats?',
answer: 'One that matches your workload shape and is reproducible. TPC-DS alone
is a poor fit because it never mutates data after loading; Microsoft\'s
LST-Bench (2023) improves on it by adding mutation, concurrency, and
maintenance phases, and the open-source LakeLoader framework applies identical
parameterized change streams to each format for incremental-write comparisons.
The most meaningful test is your own pipe [...]
+ {question: 'Do Hudi\'s indexes really change upsert performance that much?',
answer: 'Yes, measurably. Soumil Shah\'s October 2023 evaluation on Hudi 0.14
found the record-level index roughly 45% faster than the global bloom index and
about 25% faster than the global simple index on the same upsert workload. That
within-format spread is why cross-format benchmarks that do not state their
index or delete-strategy configuration are hard to interpret meaningfully.'},
+ {question: 'Can I test both formats without maintaining two copies of my
data?', answer: 'Yes. Apache XTable (incubating) translates table metadata
between Hudi and Iceberg without rewriting the underlying data files, so you
can write with one format and query through the other, or run comparative tests
against the same physical data. That makes the format decision testable rather
than a leap of faith.'},
+]} />
diff --git
a/website/blog/2026-08-14-hudi-for-read-heavy-workloads-point-lookups-with-indexes.md
b/website/blog/2026-08-14-hudi-for-read-heavy-workloads-point-lookups-with-indexes.md
new file mode 100644
index 000000000000..b06a3e001d87
--- /dev/null
+++
b/website/blog/2026-08-14-hudi-for-read-heavy-workloads-point-lookups-with-indexes.md
@@ -0,0 +1,146 @@
+---
+title: "Point Lookups on the Lakehouse: How Hudi Indexes Accelerate Read-Heavy
Workloads"
+excerpt: "Partition pruning and min/max file statistics stop helping when
queries filter on high-cardinality columns. Hudi's record-level, secondary, and
expression indexes prune point lookups down to the handful of files that
actually contain matching rows."
+description: "How Apache Hudi's record-level, secondary, and expression
indexes prune files at query planning time to accelerate point lookups and
selective reads."
+authors: [sivabalan]
+category: deep-dive
+image:
/assets/images/blog/2024-03-30-record-level-indexing-apache-hudi-delivers-70-faster-point.png
+tags:
+- indexing
+- performance
+- querying
+- data skipping
+---
+
+Analytical scans are not the only workload a lakehouse table serves. In many
production query mixes, a large share of queries are needle-in-haystack reads:
fetch one order by `order_id`, pull a user's profile by `user_id`, trace a
request by `uuid`, list all events for one `customer_id`. These queries touch a
few rows out of billions — and they are exactly where the lake's two standard
pruning tools, partition pruning and min/max file statistics, stop helping.
Apache Hudi answers this wi [...]
+
+This post is the read-side companion to our [write-heavy
comparison](/blog/2026/08/12/hudi-vs-delta-lake-for-write-heavy-workloads).
Same approach: mechanisms first, and only benchmark numbers that have already
been published, with dates and sources.
+
+## Why Selective Queries Are Hard on a Data Lake
+
+Lakehouse tables are laid out for scans: large immutable columnar files,
grouped into partitions, described by file-level statistics. Every standard
read optimization prunes at one of those granularities, and each one has a
blind spot for selective predicates on high-cardinality columns:
+
+- **Partition pruning** only helps if the filter column is the partition
column. Nobody partitions by `user_id` or `uuid` — the cardinality is far too
high — so a point lookup on such a column matches *every* partition.
+- **Min/max file statistics** help when values correlate with file layout. A
filter on an ingestion timestamp prunes beautifully, because each file covers a
narrow time range. But a random key like a UUID is uniformly spread: every
file's min/max range spans nearly the whole keyspace, so every file "might"
contain the value and nothing is pruned. Sorting or clustering the data can
rescue statistics for *one* column, but a table can only be physically ordered
one way.
+- **Parquet footer bloom filters and page indexes** operate per file — the
engine still has to open every candidate file to consult them, which at
thousands of files is itself the bottleneck.
+
+The result is familiar to anyone who has run `SELECT * FROM events WHERE
request_id = '...'` on a large table: a full scan of the key column across the
table, minutes of compute, and (on scan-priced engines) a bill proportional to
table size rather than result size. What the query needed was a database-style
answer to "which files contain this value?" — an index.
+
+## How Hudi Indexes Serve Reads
+
+Hudi has maintained indexes since its inception, originally to make upserts
and deletes fast — the write side of the same problem, as covered in the
[write-heavy
post](/blog/2026/08/12/hudi-vs-delta-lake-for-write-heavy-workloads). With the
[multi-modal indexing subsystem](/docs/indexes#multi-modal-indexing), those
same structures are consulted at *query planning time*. The indexes live as
partitions of Hudi's metadata table — itself a Merge-on-Read Hudi table using
an HFile format optim [...]
+
+- **Record-level index (RLI)** — an exact mapping from record key to file
location, hash-sharded across file groups to scale to very large keyspaces. A
query with an equality predicate on the record key (`WHERE uuid = '...'`)
resolves directly to the file group holding that key; only that file is scanned.
+- **Secondary index** — introduced in [Hudi
1.0](/blog/2025/04/02/secondary-index), an index on any non-key column. It maps
secondary key values (e.g., `city`, `driver`, `customer_id`) to the record keys
that carry them; the record index then maps those keys to file locations.
Equality and `IN` predicates on indexed columns prune to exactly the files
containing matches.
+- **Expression index** — an index on a *function* of a column, in two flavors:
column-stats over transformed values (e.g., `from_unixtime(ts)` for date
filters on epoch columns) and bloom filters over transformed values for
equality matching on high-cardinality columns.
+- **Bloom filter index** — bloom filters for all data files stored centrally
in the metadata table, so candidate files can be eliminated without touching
each file's footer.
+- **Column stats and partition stats indexes** — the min/max statistics story,
but stored in the scalable metadata table and usable for range predicates and
partition-level skipping.
+
+The planning flow for a secondary-index lookup, as described in the [indexing
deep
dive](/blog/2025/11/12/deep-dive-into-hudis-indexing-subsystem-part-2-of-2):
the engine pushes the equality predicate down to Hudi's integration layer, the
secondary index returns the matching record keys, the record index returns the
enclosing file locations, and the engine plans a scan over just those files.
Two point lookups against compact metadata replace a scan over the table — the
same shape a datab [...]
+
+This combination of write-side and read-side indexing is one of the [things
that distinguish Hudi
architecturally](/blog/2025/03/05/hudi-21-unique-differentiators): the storage
format deliberately spends extra space on indexes to serve both record-level
mutation and selective reads, rather than optimizing for vanilla scans alone.
+
+## Using It from Spark SQL
+
+Index-accelerated reads are plain SQL. Create a table with the record index
enabled (secondary indexes require it, along with a primary key and the
`COMMIT_TIME_ORDERING` merge mode), then create indexes with `CREATE INDEX`:
+
+```sql
+CREATE TABLE hudi_table (
+ ts BIGINT,
+ uuid STRING,
+ rider STRING,
+ driver STRING,
+ fare DOUBLE,
+ city STRING
+) USING hudi
+OPTIONS (
+ primaryKey = 'uuid',
+ hoodie.metadata.record.index.enable = 'true',
+ hoodie.write.record.merge.mode = 'COMMIT_TIME_ORDERING'
+)
+PARTITIONED BY (city);
+
+-- record index first; secondary indexes build on it
+CREATE INDEX record_index ON hudi_table (uuid);
+-- secondary index on a non-key, high-cardinality column
+CREATE INDEX idx_rider ON hudi_table (rider);
+```
+
+Queries need no hints — equality predicates on indexed columns are pruned
automatically during planning:
+
+```sql
+-- point lookup on the record key, served by the record-level index
+SELECT * FROM hudi_table
+WHERE uuid = 'c8abbe79-8d89-47ea-b4ce-4d224bae5bfa';
+
+-- selective filter on a non-key column, served by the secondary index
+SELECT * FROM hudi_table WHERE rider = 'rider-B';
+```
+
+In the walkthrough in the [SQL queries documentation](/docs/sql_queries), the
second query scans one file instead of three after the index is created — on
the toy table that is the whole point demonstrated; on production tables the
pruning ratio scales with file count. Expression indexes cover predicates with
inline transformations, and bloom-filter expression indexes handle equality
matching where an exact mapping would be overkill:
+
+```sql
+-- date filters on an epoch column
+CREATE INDEX idx_column_ts ON hudi_table
+ USING column_stats(ts) OPTIONS(expr='from_unixtime', format='yyyy-MM-dd');
+
+-- bloom-filter pruning for equality predicates on driver
+CREATE INDEX idx_bloom_driver ON hudi_table
+ USING bloom_filters(driver) OPTIONS(expr='identity');
+```
+
+`SHOW INDEXES FROM hudi_table` lists what exists, `DROP INDEX` removes one,
and session settings such as `SET hoodie.metadata.record.index.enable=true` and
`SET hoodie.metadata.column.stats.enable=true` control which indexes the reader
consults — see [SQL queries](/docs/sql_queries) for the full set.
+
+## What About Trino?
+
+Honestly stated: index-based pruning through the record-level, secondary, and
expression indexes is a **Spark SQL capability today**. When secondary indexes
shipped in Hudi 1.0, [support was planned for Flink, Presto, and
Trino](/blog/2025/04/02/secondary-index) in a subsequent release; that work
rides on the fact that the indexes are engine-neutral storage structures —
partitions of the metadata table on disk, not Spark-private state — so an
engine integration implements the lookup agai [...]
+
+What Trino supports today, per the [query engine
documentation](/docs/sql_queries#trino): Hudi tables are queried through the
native Hudi connector (Trino 398 onward) or via the Hive connector with table
redirection (Trino 411 onward, using `hive.hudi-catalog-name=hudi`). Both paths
support snapshot queries on Copy-on-Write tables and read-optimized queries on
Merge-on-Read tables, with MOR snapshot query support in progress in the Trino
community. So a Trino-fronted deployment gets Hudi [...]
+
+## What Published Results Show
+
+Two data points, both from Hudi's own published material, both with setups
disclosed:
+
+- **Record-level index** ([indexing deep dive, November
2025](/blog/2025/11/12/deep-dive-into-hudis-indexing-subsystem-part-2-of-2)):
on a 400 GB synthetic Hudi table with 20,000 file groups, a query filtering on
a single record key dropped from 977 seconds to 12 seconds — a 98% reduction —
with the record index in use.
+- **Secondary index** ([secondary index announcement, April
2025](/blog/2025/04/02/secondary-index)): on the TPC-DS 1 TB dataset (Hudi
1.0.1, Spark 3.5.5 on EMR, 10 executors), a join query with a customer-id
lookup on `web_sales` ran ~33% faster on the first run and ~58% faster on a
warm second run with a secondary index on `ws_ship_customer_sk`. Data scanned
fell ~90% — from 67 GB across 5,000 files to 7 GB across 521 files, and from
719M rows scanned to 75M.
+
+The scan reduction is the number to internalize: latency gains vary with
cluster and cache state, but reading 90% fewer bytes is an architectural
outcome, and on engines priced per byte scanned it translates directly to cost.
For reproducing this class of measurement on your own keys and data
distribution, the open-source
[LakeLoader](https://github.com/onehouseinc/lake-loader) framework exists
precisely to generate controlled, repeatable lakehouse workloads.
+
+## How This Compares Architecturally
+
+Lakehouse table formats broadly take one of two positions on selective reads.
One position is *file-statistics-only pruning*: keep per-file min/max
statistics (plus partition values) in table metadata, and make them effective
by physically clustering data so that values correlate with files. This is
metadata that is cheap to maintain and works well when queries filter on the
clustering dimensions — but a table can only be clustered one way, and
predicates on other high-cardinality column [...]
+
+Hudi's position is *queryable index metadata*: spend additional storage and
write-path work maintaining exact value-to-location mappings (record-level and
secondary indexes) and auxiliary structures (bloom filters, expression indexes)
in a scalable, transactionally-updated metadata table, so that pruning for
equality predicates is an index lookup rather than a statistics estimate — on
as many columns as you choose to index. The trade is explicit: index storage
and maintenance cost in exc [...]
+
+## Operational Notes
+
+Two things keep this practical in production. First, indexes are maintained
**transactionally with each commit** — a query planned against the secondary
index sees results consistent with the latest completed write, not a lagging
sidecar. Second, adding an index to a table that is already ingesting does not
require stopping it: Hudi's [async indexing](/docs/metadata_indexing) builds a
new index in the background while writers keep committing, then reconciles the
seam — the mechanics and [...]
+
+## Conclusion
+
+Point lookups and selective filters on high-cardinality columns are a real,
often dominant slice of production query traffic, and they are precisely the
queries that partition pruning and min/max statistics cannot save. Hudi's
answer is the one databases settled on decades ago — indexes — rebuilt for lake
storage as transactional partitions of a scalable metadata table: a
record-level index for key equality, secondary indexes for non-key columns,
expression indexes for transformed predic [...]
+
+## FAQ
+
+<PostFAQ heading={null} items={[
+ {
+ question: 'Can you do point lookups on a data lake?',
+ answer: 'Yes, with a table format that maintains indexes. On a plain
columnar lake, a point lookup on a high-cardinality column degrades to scanning
that column across all files, because partition pruning and min/max statistics
cannot narrow random key values. Apache Hudi maintains a record-level index
mapping each record key to its file location and secondary indexes for non-key
columns inside its metadata table, so an equality predicate resolves to the few
files containing matches. [...]
+ },
+ {
+ question: 'What is a secondary index in Apache Hudi?',
+ answer: 'A secondary index, introduced in Hudi 1.0, is an index on any
column other than the record key. It stores mappings from secondary key values
to record keys in the metadata table; at query time the matched record keys are
resolved to file locations through the record-level index, and only those files
are scanned. It is created with plain SQL, for example CREATE INDEX idx_city ON
hudi_table(city), and requires the record index to be enabled on the table.',
+ },
+ {
+ question: 'Does Trino use Hudi indexes?',
+ answer: 'Not yet for record-level and secondary index pruning — that
acceleration is available from Spark SQL today. Trino queries Hudi tables
through the native Hudi connector or Hive connector redirection, supporting
snapshot queries on Copy-on-Write tables and read-optimized queries on
Merge-on-Read tables. Because Hudi\'s indexes are engine-neutral structures
stored in the metadata table rather than Spark-private state, engine
integrations can adopt them, and support for Presto, [...]
+ },
+ {
+ question: 'How much faster are queries with Hudi\'s indexes?',
+ answer: 'Per Hudi\'s published measurements: a record-key lookup on a 400
GB synthetic table with 20,000 file groups fell from 977 seconds to 12 seconds
with the record-level index (a 98% reduction), and a TPC-DS 1 TB join query
with a customer-id filter ran about 33-58% faster with a secondary index while
scanning roughly 90% less data — 7 GB across 521 files instead of 67 GB across
5,000 files. Actual gains depend on data distribution and cluster setup, so
test with your own workload.',
+ },
+ {
+ question: 'Does maintaining indexes for reads slow down ingestion?',
+ answer: 'Indexes are updated transactionally with each commit, which adds
bounded write-path work per index — one reason to index only the columns that
appear in selective predicates. Adding a new index to a live table does not
require stopping ingestion: Hudi\'s async indexing service builds the index in
the background while writers keep committing, then reconciles concurrent
changes, keeping the index timeline-consistent with the table.',
+ },
+]} />
diff --git a/website/static/llms.txt b/website/static/llms.txt
index f8fbfb4df255..f407d633bc50 100644
--- a/website/static/llms.txt
+++ b/website/static/llms.txt
@@ -59,6 +59,10 @@ It can also synchronize your data to half dozen data
catalogs to keep table cons
## Blog — Comparisons
- [Hudi vs Iceberg for CDC
Workloads](https://hudi.apache.org/blog/2026/08/06/hudi-vs-iceberg-for-cdc-workloads):
Criteria-driven comparison for change-data-capture ingestion.
+- [Hudi vs Iceberg for Streaming
Ingestion](https://hudi.apache.org/blog/2026/08/11/hudi-vs-iceberg-for-streaming-ingestion):
Which table format handles continuous, high-frequency writes better.
+- [Hudi vs Delta Lake for Write-Heavy
Workloads](https://hudi.apache.org/blog/2026/08/12/hudi-vs-delta-lake-for-write-heavy-workloads):
Update-intensive and mutable workloads compared across both formats.
+- [Hudi vs Iceberg Performance
Benchmarks](https://hudi.apache.org/blog/2026/08/13/hudi-vs-iceberg-performance-benchmarks):
A methodology-aware compilation of published benchmarks.
+- [Point Lookups on the
Lakehouse](https://hudi.apache.org/blog/2026/08/14/hudi-for-read-heavy-workloads-point-lookups-with-indexes):
How Hudi indexes accelerate read-heavy workloads and selective queries on
high-cardinality columns.
- [Hudi vs Delta Lake vs Iceberg Feature
Comparison](https://hudi.apache.org/blog/2023/01/11/Apache-Hudi-vs-Delta-Lake-vs-Apache-Iceberg-Lakehouse-Feature-Comparison):
Feature-by-feature comparison of the three open table formats.
## Blog — Deep dives
