This is an automated email from the ASF dual-hosted git repository.
danny0405 pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new f2e4bd68a7cf docs(blog): why MOR enables async indexing (MOR series
part 3) (#19070)
f2e4bd68a7cf is described below
commit f2e4bd68a7cf66745303e33303a0a65907b7b68c
Author: Sivabalan Narayanan <[email protected]>
AuthorDate: Sun Jun 28 21:38:24 2026 -0700
docs(blog): why MOR enables async indexing (MOR series part 3) (#19070)
Third post in the MOR-as-architectural-shift series. Walks through
RFC-45's async indexing design and explains why the three primitives
it depends on (stable file-group container, INDEXING timeline action,
two-stage partition lifecycle) only exist because MOR put them in
place years earlier.
Also adds 'async indexing' to the blog tag allowlist.
---
.github/scripts/validate-blog.py | 1 +
...6-06-25-building-indexes-on-a-moving-target.mdx | 189 +++++++++++++++++++++
.../async-indexing-architectural-consequence.png | Bin 0 -> 2981809 bytes
.../async-indexing-end-end-flow.png | Bin 0 -> 2354420 bytes
4 files changed, 190 insertions(+)
diff --git a/.github/scripts/validate-blog.py b/.github/scripts/validate-blog.py
index 847afbb43f38..63cee6129d2e 100644
--- a/.github/scripts/validate-blog.py
+++ b/.github/scripts/validate-blog.py
@@ -49,6 +49,7 @@ ALLOWED_TAGS = {
'bytearray', 'best practices', 'backfilling', 'architecture',
'apicurio registry', 'apache zeppelin', 'apache orc', 'apache
dolphinscheduler',
'apache avro', 'apache', 'access control', 'lakehouse', 'merge on read',
'record level index','rli', 'penn interactive', 'southwest airlines', 'metica',
+ 'async indexing',
}
# Tags that should not be used
diff --git a/website/blog/2026-06-25-building-indexes-on-a-moving-target.mdx
b/website/blog/2026-06-25-building-indexes-on-a-moving-target.mdx
new file mode 100644
index 000000000000..58cdfc6101f6
--- /dev/null
+++ b/website/blog/2026-06-25-building-indexes-on-a-moving-target.mdx
@@ -0,0 +1,189 @@
+---
+title: "Building Indexes on a Moving Target"
+excerpt: "Async indexing in Hudi lets you add a new index to a multi-petabyte
table without stopping ingestion — and the design only works because MOR put
the right primitives in place years earlier."
+authors: [sivabalan]
+category: deep-dive
+image:
/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-architectural-consequence.png
+tags:
+- mor
+- merge on read
+- indexing
+- async indexing
+- metadata
+- architecture
+- data lakehouse
+---
+
+---
+
+_This is the third post in a series on Merge-On-Read as an architectural shift
in Apache Hudi. The [first
post](/blog/2026/05/14/mor-isnt-a-storage-optimization-its-an-architectural-shift)
made the broad case; the
[second](/blog/2026/06/05/why-metadata-has-to-be-mutation-friendly) showed how
the metadata table inherits the same append-first model; this one follows the
pattern one step further — into how new indexes can be added to a live table
without stopping it._
+
+---
+
+<div style={{ fontSize: '1.15rem', lineHeight: '1.75' }}>
+
+The first post in this series argued that Merge-On-Read is not a storage
optimization but an [architectural
shift](/blog/2026/05/14/mor-isnt-a-storage-optimization-its-an-architectural-shift)
— a strategy for time-shifting work in systems where mutation and analytical
scan run at different rhythms. The second grounded that argument in Hudi's
metadata table, showing how metadata itself became another [append-first
workload](/blog/2026/06/05/why-metadata-has-to-be-mutation-friendly). [Asyn
[...]
+
+This article assumes some familiarity with Hudi's metadata table and its
indexing capabilities. If you're new to the topic, the motivation for metadata
indexes, their role in query planning, and the evolution of the metadata table
are covered in detail in
[RFC-45](https://github.com/apache/hudi/blob/master/rfc/rfc-45/rfc-45.md) as
well as our earlier articles on the [metadata table](/docs/metadata) and
[mutation-friendly
metadata](/blog/2026/06/05/why-metadata-has-to-be-mutation-friendly [...]
+
+The title is literal. In Hudi, you can build a new index — record-level,
secondary, expression, column stats, bloom filter — over an existing
multi-petabyte table that doesn't stop while you build it. Ingest writers keep
committing. No outage. No drained pipeline. No swap window. The bootstrap reads
historical state into a freshly initialized metadata partition; concurrent
writers route new mutations into the same partition's file groups; a
reconciliation step at the end stitches the seam.
+
+That capability does not exist in any other open lakehouse format at the spec
level — not because nobody implemented it, but because the storage layer every
other format is built on cannot express the pattern. Async indexing is not a
feature added to Hudi; it is a consequence of decisions Hudi made in 2017. The
only systems that can build indexes this way are the ones whose mutation model
is already append-first.
+
+[RFC-45](https://github.com/apache/hudi/blob/master/rfc/rfc-45/rfc-45.md) is
the design doc, and the [Onehouse engineering
blog](https://www.onehouse.ai/blog/asynchronous-indexing-using-hudi) walks
through the mechanics. Those explain *what* async indexing does and *how* it's
wired in. This post is about why the design works at all — what architectural
primitives have to exist underneath before something like it becomes a coherent
thing to build.
+
+---
+
+## The Shape of the Problem
+
+Adding a new index over an existing large table has always been an
operationally expensive thing for analytical storage to do. Scan every row,
project the indexed columns or compute the indexed expression, materialize the
index. The trouble is that "scan every row" of a multi-petabyte table takes
hours, and during those hours the table either has to stop accepting writes or
has to coordinate with a table that won't sit still.
+
+Two responses dominate the lakehouse conversation, and both inherit the
limitations of their foundation.
+
+The first is to bind the index to the write path: compute the value at
ingestion time and write it alongside the data — column-level statistics,
min/max metadata, file summaries. This is what Iceberg and Delta do by default.
It works, but it cannot retroactively add an index. If the table was written
for two years without it, the write path can't help; the data is already on
disk in its final form.
+
+The second is to lock-and-rebuild: global or per-partition lock, scan,
materialize, atomic swap, release. Fine at small scale. At petabyte scale on a
continuously ingesting table the lock duration is hours, the ingest pipeline
has to stop or buffer indefinitely, and any failure during the rebuild discards
the work.
+
+Hudi's predicament when
[RFC-15](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427331)
introduced the metadata table was that MDT was going to host indexes —
`files`, `column_stats`, `bloom_filters`, `record_index`, `partition_stats`,
`secondary_index`, `expression_index`. Every one of those would eventually need
to be added over an existing table by someone who'd been running Hudi for a
year before the index existed. If index addition required taking the table offl
[...]
+
+---
+
+## What COW Makes Impossible — Not Hard, Impossible
+
+Imagine implementing async indexing on a Copy-On-Write storage layer — a
generic columnar lakehouse format where the unit of mutation is the file
rewrite. Bootstrap scans history, computes, materializes. Concurrent writers
continue ingesting. Index has to be queryable at the end.
+
+The bootstrap is slow under any architecture; that's not what breaks. What
breaks is that there is no place for concurrent writers to put their
contribution to the new index while the bootstrap is running.
+
+Under COW, every index update is a rewrite of a finalized index file. There is
no append channel. If the indexer has materialized `idx_000.parquet` for
history up to `t`, and a writer commits at `t+1`, the writer cannot contribute
to `idx_000.parquet`. The writer's only option is `idx_001.parquet` — which
means the bootstrap has to coordinate with the writer about which file was
produced, validate it, possibly merge it. That coordination is the
synchronization we were trying to avoid. Th [...]
+
+The workaround in rewrite-oriented systems is *bootstrap-and-swap*: build
against a frozen snapshot at `t`; writers ingest normally but don't touch the
new index; at `t + Δ`, replay all writes between `t` and `t + Δ` into the
index, then atomically swap. This works, but the replay window has no upper
bound (if ingest exceeds replay rate, catchup never converges), and the index
is unavailable for query during the entire bootstrap-plus-replay duration.
+
+Hudi doesn't need bootstrap-and-swap because the foundation offers something
neither Iceberg nor Delta has: a stable file group with an append channel, and
a timeline that expresses index construction as a first-class action with its
own reconciliation semantics. Both primitives are downstream of MOR.
+
+Before examining the architectural primitives individually, it's helpful to
look at the end-to-end lifecycle of an async index build. The figure below
intentionally presents the high-level execution flow rather than the
implementation details. For readers interested in the complete design,
state-machine transitions, and failure handling,
[RFC-45](https://github.com/apache/hudi/blob/master/rfc/rfc-45/rfc-45.md)
provides the authoritative specification. Here, we focus on the architectural
[...]
+
+<div style={{ textAlign: 'center' }}>
+ <img
src="/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-end-end-flow.png"
alt="Async indexing end-to-end flow" width="1000"/>
+</div>
+
+---
+
+## The Three Primitives, All of Them MOR Consequences
+
+Async indexing in Hudi rests on three substrate-level primitives. Each one is
something the MOR design forced into existence for other reasons, and each one
is what makes RFC-45's design coherent rather than a heroic engineering effort
against the storage layout.
+
+### Primitive 1: The File Group as a Stable Mutation Container
+
+The [file group](/docs/storage_layouts) is a stable logical mutation
container. Once initialized, multiple independent processes — not just ingest
writers, but background services such as index builders — can deterministically
contribute mutations into the same logical object while deferring physical
reconciliation. Async indexing is one application of this primitive. Later in
the series, we'll see the same stable mutation containers reused by other Hudi
capabilities, including async com [...]
+
+When the indexer initializes a new metadata partition, it creates empty file
groups for that partition. From that moment, they exist on disk and are
addressable. A writer that sees the partition is inflight (via table config)
can immediately append log blocks into those file groups, even though base
files haven't been written yet. The writer doesn't need to know whether the
indexer has finished the base file or coordinate physical layout at all. It
needs only to know which file group to [...]
+
+Under COW this primitive doesn't exist. There's no "container the indexer
prepares and writers append into" — only files, produced atomically and
consumed read-only. The closest analog is a shared directory both processes
write new files into; but that's a namespace, not a container, and the
synchronization problem returns immediately.
+
+### Primitive 2: INDEXING as a First-Class Timeline Action
+
+[Hudi's timeline](/docs/timeline) is an append-only event log: commits,
deltacommits, cleans, compactions, restores, rollbacks. RFC-45 adds
**INDEXING_ACTION**. An index build is not a side process; it is represented as
its own sequence of timeline instants: `<t>.indexing.requested`,
`<t>.indexing.inflight`, and `<t>.indexing` (completed), alongside data writes
and other table services.
+
+That placement matters. Because the action lives on the timeline, concurrent
writers discover it through the same timeline refresh they already perform for
commits and table services; no separate coordination channel is required.
Because it follows the standard requested → inflight → completed lifecycle,
existing mechanisms for recovery, rollback, and lock-protected state
transitions apply without introducing a parallel state machine. Async indexing
is built by extending the timeline rat [...]
+
+More importantly, the timeline and table configuration together govern
**visibility**, not just execution. Bootstrap and Catchup may execute for
minutes while the table continues accepting writes, but the new index remains
invisible to readers throughout that period.
+
+Internally, the indexing action progresses through the standard `requested →
inflight → completed` lifecycle on the timeline, while the table configuration
independently tracks whether the metadata partition is still under construction
(`metadata.partitions.inflight`) or has become queryable
(`metadata.partitions`). Readers observe the new index only after both
transitions have completed. This separation is subtle but fundamental.
Bootstrap and Catchup describe **how** the index is const [...]
+
+Although async indexing predates the completion-time semantics introduced
later for non-blocking concurrency control, it already exhibits the same
architectural pattern: background work progresses independently, reconciliation
completes asynchronously, and publication is deferred until the system can
present a consistent view. Later in the series, we'll see how Hudi generalizes
this idea for concurrent writers using completion-time ordering.
+
+### Primitive 3: Two-Stage Partition Lifecycle in Table Config
+
+The subtlest of the three, and the one most directly inherited from MOR.
`hoodie.properties` carries two related fields:
`hoodie.table.metadata.partitions.inflight` lists partitions initialized and
receiving writes but not yet queryable; `hoodie.table.metadata.partitions`
lists partitions fully built and safe to read.
+
+That split is what makes async indexing safe under split-brain reader/writer
views. When scheduling completes — file groups initialized, plan persisted —
the partition is added to `*_INFLIGHT`. Writers see it and start routing log
blocks. Readers see it too, and ignore the partition: table config is the
source of truth for query readiness. Only after the bootstrap base is written,
Catchup reconciles any missing metadata updates from concurrent commits, and
the indexing action completes d [...]
+
+The two-stage config is MOR's base-file-vs-log-file separation lifted to the
partition level. Log files are writable before the next base file is
materialized; readers reconcile both at query time. Inflight partitions are
writable before the bootstrap completes; readers ignore them until done. Same
idea, different scope. You can see this in code at
`RunIndexActionExecutor.updateTableConfigAndTimeline`: the partition
transitions from inflight to active as part of the indexing completion s [...]
+
+The abort path at `RunIndexActionExecutor.abort` removes the partition from
both lists and deletes the physical partition; on any build failure, the
partition reverts cleanly to "doesn't exist."
+
+Because both scheduling and completion execute under Hudi's timeline lock
provider, these metadata transitions remain serialized even though the
bootstrap itself proceeds concurrently with foreground writers.
+
+Interestingly, these primitives turn out to be useful well beyond index
construction. Stable mutation containers, timeline-ordered visibility, and
deferred reconciliation reappear repeatedly throughout Hudi's architecture.
Async indexing is one application. Later in the series, we'll see the same
foundations support background optimization and eventually reshape how
concurrent writers coordinate against the same data.
+
+<div style={{ textAlign: 'center' }}>
+ <img
src="/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-architectural-consequence.png"
alt="Async indexing as an architectural consequence of MOR" width="1000"/>
+</div>
+
+---
+
+## Indexing Catchup as Deferred Reconciliation, Generalized
+
+The three primitives set up the most interesting piece of the design: indexing
catchup. The implementation lives in `AbstractIndexingCatchupTask`,
orchestrated by `RunIndexActionExecutor`. It's where the MOR-as-foundation
argument becomes concrete.
+
+The indexing plan is scheduled against a bootstrap target — the latest
completed data-table instant at scheduling time. Bootstrap builds the
historical portion of the index by scanning the table up to that point.
Meanwhile, the data timeline continues advancing.
+
+A common misconception is that Bootstrap alone constructs the index. In
reality, every commit that arrives while Bootstrap is running is equally part
of the index build — it simply reaches the metadata partition through the
incremental write path rather than the historical scan. Writers that observe
the indexing action after it becomes inflight begin appending metadata updates
into the new file groups. By the time Bootstrap completes, the metadata
partition consists of a historical base [...]
+
+The indexer cannot simply assume that every eligible commit has been reflected
in the metadata timeline. Some writers may have started before the indexing
action became visible. Others may still be inflight. Catchup exists to
reconcile that uncertainty before the index becomes visible.
+
+Catchup walks the relevant portion of the data timeline and compares each
eligible data-table instant against the metadata timeline using the indexing
action's requested-time identifier. If the corresponding metadata update
already exists, the writer has already contributed its portion of the index.
Otherwise, Catchup synthesizes the missing metadata update on the writer's
behalf.
+
+Inflight writers receive special treatment. Rather than immediately treating
them as missing, Catchup waits for active writers to complete while monitoring
their heartbeats. Writers that complete normally are incorporated through the
usual reconciliation path. Writers whose heartbeats have expired are treated as
abandoned and skipped, allowing index construction to converge without waiting
indefinitely. The wait itself is bounded by
`hoodie.metadata.index.check.timeout.seconds` (900 seco [...]
+
+This is deferred reconciliation in its purest form. Bootstrap constructs
historical state. Concurrent writers continue making forward progress. Catchup
reconciles any remaining divergence between the data timeline and metadata
timeline before publication. Readers never observe intermediate state.
+
+> The pattern is exactly what Merge-On-Read readers already do for every file
group: reconcile a stable base with subsequent incremental mutations to produce
a consistent view. Async indexing applies the same architectural idea at the
scope of an entire metadata partition.
+
+The mechanism is different, but the underlying principle is identical:
foreground progress continues independently, while background reconciliation
converges the system before publication.
+
+---
+
+## The Races, and How the Architecture Absorbs Them
+
+RFC-45 enumerates four error cases. Each one is a place where synchronous
coordination would have required a lock, and each one is handled instead by
deferred reconciliation against timeline state. They're worth walking through
because they make the architectural argument concrete.
+
+**Writer fails while the indexer is inflight.** The writer crashed before
logging its update. The indexer doesn't care. Catchup observes the failed
instant, sees no metadata deltacommit, and either skips it (if data-table
rollback already cleaned up) or fills it in. The metadata partition isn't
corrupted because reader visibility hasn't flipped yet.
+
+**Indexer fails while a writer is inflight.** The writer continues logging
into the new file groups. The partition stays in `*_INFLIGHT` because no commit
metadata was written. Readers continue ignoring it. When the indexer is
re-triggered, it sees the partition is inflight, picks up the build (idempotent
by partition name in `ScheduleIndexActionExecutor`), and proceeds. The writer's
log files aren't lost — they're already in the file group, waiting for catchup.
+
+**Writer goes inflight just after the indexing request, before the indexer
executes.** A writer begins after the indexing action becomes visible and
therefore participates in index construction from the outset. By bootstrap
completion, log files already exist. Catchup observes them, sees the
corresponding deltacommit, moves on. No coordination — writer and indexer were
independently pointing at the same stable container.
+
+**Inflight writer about to commit but indexing just completed.** The trickiest
race. The writer started knowing the partition was inflight; during its
execution the partition flipped to completed. The writer commits, producing a
log file in the now-completed partition, and the indexer missed it. Because the
reader engine is designed from day one to merge base + logs on the fly, **the
reader's merge layer inherently immunizes the system against race conditions.**
The storage layout itself [...]
+
+In each case, the response shape is the same: stable container, timeline
record, reconciliation at the next quiet moment. None of this is
async-indexing-specific. It's what every MOR table service does.
+
+Notice that none of these cases require the bootstrap itself to synchronize
with foreground writers. Synchronization is reserved for timeline state
transitions, while correctness is achieved through deferred reconciliation
before publication.
+
+---
+
+## Why This Isn't Expressible in Rewrite-Oriented Formats
+
+Can async indexing be added to Iceberg or Delta with sufficient engineering
effort? It can be approximated, but the approximation either gives up the "no
synchronization with writers" property or requires inventing the three
primitives above on a foundation that doesn't naturally support them.
+
+Iceberg has manifests, manifest lists, snapshots — no stable logical container
with an append channel. Each commit produces new manifest files; there's no
"manifest the indexer prepares and writers contribute to." Row lineage and v3
deletion vectors give stable row identity, but that's not the same primitive as
a stable append container. Adding async indexing to Iceberg would require a new
manifest layout where indexers stage skeletons that subsequent writers
contribute to — a spec chang [...]
+
+Delta has the transaction log, deletion vectors, and (in Databricks Runtime)
row-level concurrency for non-overlapping MERGE/UPDATE/DELETE. None provide an
append container stable across commits. Coordinated commits in 4.0 improve
multi-engine atomic commit, but the unit of commit is still a set of new files;
there's no notion of writers appending into a file the indexer staged. Delta's
data skipping and statistics are written during the data commit; they cannot be
retroactively built wi [...]
+
+Async indexing isn't a missing feature in those formats; it's missing because
the foundation cannot express the pattern. Adding it would require lifting
append-first mutation into the storage model itself. That is most of what MOR
is.
+
+The "async-indexing-ish" capabilities those formats have shipped are either
write-path additions (compute during writes, can't retroactively add) or
batch-and-swap (block readers during a window, schedule a maintenance window).
Neither is what Hudi async indexing actually does.
+
+---
+
+## What This Unlocks Operationally
+
+The architectural argument lands on a small set of capabilities that change
how teams operate Hudi tables.
+
+The most immediate: *adding indexes to existing tables without a maintenance
window.* RLI on a 1 PB table for primary-key lookup. Secondary index on
`customer_email` for a query pattern that wasn't important when the table was
created. Expression index on `LOWER(country_code)` because the workload
changed. Issue `CREATE INDEX` (or run `HoodieIndexer`) against the running
table. Bootstrap fills history. Writers continue. Catchup closes the seam.
+
+Second: *index re-initialization as a routine operation.* If an index gets
into a bad state — schema evolution mismatch, corrupted base file, missed
catchup window — recovery is "delete the partition, re-run the indexer." Not a
migration event. The same `ScheduleIndexActionExecutor.abort` path handles
cleanup before re-initialization.
+
+Third: *staged index rollouts.* You don't have to enable every metadata
partition at table creation. Enable `files` and `column_stats` initially, add
`record_index` later when point-lookup load justifies it, `secondary_index`
later still when a new query pattern emerges. The metadata table's footprint
grows with the workload's actual needs. A table running for years can adopt a
new index without dropping a write.
+
+None of these are theoretical. Take any one primitive away — make the file
group non-append, take INDEXING off the timeline, collapse the two-stage
partition config — and the operational story collapses with it.
+
+---
+
+## What MOR Ultimately Made Possible
+
+A single architectural decision — append-first mutation with deferred
reconciliation — produces a family of capabilities that look like independent
features but are the same idea applied to different scopes. The metadata table
is MOR because the commit cadence demands it. Async indexing is possible
because the same stable mutation containers can be initialized empty and
evolved by concurrent contributors.
+
+Looked at in isolation, each capability has its own moving parts: RFC-15 for
MDT, RFC-45 for async indexing, and later in the series we'll examine how Hudi
extends the same architectural principles — stable mutation containers,
deferred reconciliation, and timeline-mediated publication — to support
non-blocking concurrency control. The descriptions are all correct. They miss
the bigger point, which is that the moving parts exist because MOR established
the underlying mutation model — and [...]
+
+</div>
diff --git
a/website/static/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-architectural-consequence.png
b/website/static/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-architectural-consequence.png
new file mode 100644
index 000000000000..397dc1b48700
Binary files /dev/null and
b/website/static/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-architectural-consequence.png
differ
diff --git
a/website/static/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-end-end-flow.png
b/website/static/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-end-end-flow.png
new file mode 100644
index 000000000000..ebe3b56fe2c3
Binary files /dev/null and
b/website/static/assets/images/blog/2026-06-25-building-indexes-on-a-moving-target/async-indexing-end-end-flow.png
differ
