This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 4d83ac93024d docs(tech-specs): secondary index and index versions for
table version 9 (#19650)
4d83ac93024d is described below
commit 4d83ac93024da07687cfe6d23f33b22394c082ea
Author: deepakpanda93 <[email protected]>
AuthorDate: Mon Aug 24 15:21:38 2026 +0530
docs(tech-specs): secondary index and index versions for table version 9
(#19650)
* docs(tech-specs): secondary index and index versions for table version 9
Closes #17068. (JIRA: HUDI-9550.)
Corrects an inverted statement already published in the tech spec, and
documents
the table version 9 index changes the issue asks for.
The existing text claimed that V2 "shards records by the primary (record)
key
rather than by the secondary key", at the cost of "secondary-key range scans
having to visit more file groups". Both halves are backwards.
MetadataPartitionType
dispatches with getSecondaryKeyToFileGroupMappingFunction(indexVersion >=
HoodieIndexVersion.V2), and that function hashes the unescaped secondary-key
prefix when the flag is set and the full record key otherwise. So V1 hashes
the
whole <secondary>$<primary> key and scatters one secondary value across
every
file group, while V2 hashes only the <secondary>$ prefix so that a lookup by
secondary value alone reads a single file group. V2 is the improvement for
secondary-key lookups, not a regression.
Also adds, all verified against release-1.2.0:
- The index definition JSON shape under hoodie.table.index.defs.path. The
path
and the property were already documented but the schema was not, and the
secondary index section pointed at #indexing-functions for "the on-disk
shape",
which is a two-line RFC stub. That cross-reference now resolves to the new
section.
- The version attribute, and which version a new index gets per index type.
Worth stating explicitly because it is not secondary-index specific:
column_stats, partition_stats and expr_index also default to V2 at table
version 9, while record_index, bloom_filters and files stay V1.
- Upgrade and downgrade behaviour. The issue describes this as
"downgrade/upgrade
will drop new/old SI index", but only downgrade drops. Upgrading 8 to 9
stamps
definitions that lack a version with V1 and drops nothing
(EightToNineUpgradeHandler#populateIndexVersionIfMissing). Downgrading 9
to 8
drops partitions whose version is above V1, plus partition_stats when a V2
column_stats partition goes
(UpgradeDowngradeUtils#dropNonV1IndexPartitions).
- Secondary index limitations: one indexed column only; the supported column
types; and that an indexed column's type cannot change while the index
exists.
Two facts differ from the abandoned PR #13713, which was the starting
reference:
- Its supported-type list, "string, double, timestamp and any integral
types", is
wrong. HoodieIndexUtils#isSecondaryIndexSupportedType admits string, int,
long,
float, double, date and time, and admits timestamp only when
isUtcAdjusted, so
local timestamps are rejected.
- Its claim that schema evolution is not allowed on an indexed column is too
broad. HoodieTable throws SchemaCompatibilityException on a column type
change,
but explicitly permits a change that only alters nullability.
That PR's .gitignore change is not carried over, being unrelated to the
docs, and
neither is its "Indexes" to "Indices" heading rename.
npm run build passes with the warning set byte-identical to a baseline
built at
the same base commit.
* docs(tech-specs): cover the integral SQL types a secondary index accepts
Review feedback on apache/hudi#19650: the supported-types list omitted
smallint
and tinyint.
The reviewer is right, and the reason is worth stating rather than just
patching
the list. HoodieIndexUtils#isSecondaryIndexSupportedType is an allow-list
over
schema types, and it has no SHORT or BYTE entries because Avro has neither.
Spark's ShortType and ByteType serialize to Avro INT (AvroSerializer has
explicit
"case (ByteType, INT)" and "case (ShortType, INT)" branches), so a smallint
or
tinyint column reaches the INT branch of the allow-list and is accepted.
Listing
only "int" was therefore accurate about the schema type but misleading to
anyone
reading it as SQL.
TestSecondaryIndexDataTypes confirms the behaviour from the other side: it
asserts index creation succeeds on col_string, col_int, col_bigint,
col_long,
col_smallint, col_tinyint, col_timestamp, col_date, col_float and
col_double, and
fails on col_decimal, col_boolean, col_binary, col_array, col_map and
col_struct.
The bullet now gives both vocabularies, and adds the rejected types, which
the
earlier wording only covered as "all other types".
Build passes with the warning set still byte-identical to a baseline at the
same
base commit.
* docs(tech-specs): describe indexFunction by what actually populates it
Review feedback on apache/hudi#19650. The bullet read "indexFunction is the
transform applied to the source column, identity unless the index is an
expression index", which is wrong in both directions: an expression index
created
without an explicit function also gets identity, so identity does not
distinguish
the two, and non-expression indexes do not reliably get identity either.
Traced all four sites that build a HoodieIndexDefinition, and the field is
populated inconsistently across them:
- HoodieTableMetadataUtil#getIndexPartitionsToInit calls withIndexFunction
only
when the partition name prefix is expr_index_, so a secondary index
registered
through that path is left empty.
- The other builder in HoodieTableMetadataUtil never calls
withIndexFunction, so
built-in partitions are left empty.
- HoodieIndexUtils#getSecondaryOrExpressionIndexDefinition, which serves SQL
CREATE INDEX for both secondary and expression indexes, uses
options.getOrDefault(EXPRESSION_OPTION, IDENTITY_TRANSFORM), so a
secondary
index created that way gets identity.
- HoodieSparkIndexClient#createOrUpdateColumnStatsIndexDefinition passes
withIndexFunction(PARTITION_NAME_COLUMN_STATS), so that definition carries
"column_stats" rather than an empty value or identity.
Unset values are stored as the empty string, not identity:
HoodieIndexDefinition
normalises with nonEmpty(indexFunction) ? indexFunction : EMPTY_STRING.
The bullet now says the field carries the expression-index transform and
defaults
to identity for expression indexes, that it is not meaningful for other
index
types, what each registration path leaves behind, and that indexType is the
field
to identify an index by. This is a case where the honest description is
that the
value is inconsistent, rather than inventing a rule the code does not
follow.
Build passes with the warning set still byte-identical to a baseline at the
same
base commit.
---
website/learn/tech-specs.md | 104 +++++++++++++++++++++++++++++++++++++++++---
1 file changed, 97 insertions(+), 7 deletions(-)
diff --git a/website/learn/tech-specs.md b/website/learn/tech-specs.md
index f542c4c5b651..8f8c653d459a 100644
--- a/website/learn/tech-specs.md
+++ b/website/learn/tech-specs.md
@@ -463,6 +463,74 @@ The other fields can also be optional for writers
depending on whether protectio
### Naming
Indexes are stored under `.hoodie/metadata` storage path, with separate
partitions of the form `<index_type>_<index_name>`.
+### Index Definitions
+
+Every index carries a definition, serialized to JSON under the path in
`hoodie.table.index.defs.path`
+(default `.hoodie/.index_defs/index.json`). All definitions for a table live
in a single file, keyed by the
+metadata-table partition name:
+
+```json
+{
+ "indexDefinitions": {
+ "<partition_name>": {
+ "indexName": "<partition_name>",
+ "indexType": "<index_type>",
+ "indexFunction": "<index_function>",
+ "version": "<index_version>",
+ "sourceFields": ["<column_1>", "<column_2>"],
+ "indexOptions": {}
+ }
+ }
+}
+```
+
+* `indexType` is one of `files`, `column_stats`, `partition_stats`,
`bloom_filters`, `record_index`,
+ `secondary_index` or `expr_index`.
+* `indexFunction` carries the transform for an expression index, and
defaults to `identity` when an expression index
+ is created without an explicit function. For every other index type the
field is not meaningful, and the value it
+ ends up with depends on the code path that registered the definition:
empty when the built-in initialisation path
+ registers it, `identity` for a secondary index created through SQL `CREATE
INDEX`, and the partition name for the
+ column-stats registration path. Use `indexType` to identify an index, not
this field.
+* `sourceFields` are the data-table columns the index is derived from.
Metadata columns are permitted.
+* `indexOptions` carries index-type-specific options, such as the
expression-index function arguments.
+* `version` is the storage-layout version of the index, described below.
+
+#### Index Versions
+
+`version` holds a `HoodieIndexVersion`, an index-level attribute introduced in
table version 9. It allows the physical
+layout of an index to change without forcing a table-version upgrade or
downgrade, so that readers and writers built
+against different releases agree on how to interpret a given index partition.
Values take the form `V1`, `V2` and so on.
+
+A table version 8 definition may omit `version` entirely. From table version 9
onward a definition without a version is
+rejected as invalid, and secondary indexes on a table still at version 8 must
be `V1`.
+
+Which version a newly created index gets depends on the index type and the
table version:
+
+| Index type | Table version 8 | Table version 9 |
+|---|---|---|
+| `secondary_index` | `V1` | **`V2`** |
+| `column_stats` | `V1` | **`V2`** |
+| `partition_stats` | `V1` | **`V2`** |
+| `expr_index` | `V1` | **`V2`** |
+| `record_index` | `V1` | `V1` |
+| `bloom_filters` | `V1` | `V1` |
+| `files` | `V1` | `V1` |
+
+Only the layout changes; the logical contents of an index are unaffected by
its version.
+
+#### Table Upgrade and Downgrade
+
+Index versions are reconciled when the table version changes.
+
+* **Upgrading from table version 8 to 9** does not rebuild or drop any
index. Definitions that carry no `version` are
+ stamped `V1`, so existing indexes keep their on-disk layout and continue
to be read as `V1`. Indexes created after
+ the upgrade pick up the table version 9 defaults above.
+* **Downgrading from table version 9 to 8** drops every metadata partition
whose index version is newer than `V1`,
+ because table version 8 readers cannot interpret those layouts.
Additionally, if a `V2` `column_stats` partition is
+ dropped, `partition_stats` is dropped with it, since partition stats may
have no definition of their own to inspect.
+
+Dropped indexes have to be rebuilt after the downgrade.
+
### Bloom Filter Index
The bloom filter index is used to accelerate 'presence checks' — validating
whether a particular record is present in a file — which is used during
merging, hash-based joins, point-lookup queries, etc.
@@ -506,7 +574,7 @@ Hudi supports near-standard [SQL
syntax](/docs/sql_ddl#create-index) for creatin
via Spark SQL, along with an asynchronous indexing table service that builds
indexes without interrupting writers.
A secondary index definition is serialized to JSON and saved at the path
specified by `hoodie.table.index.defs.path`
-(see [Indexing Functions / Index Definitions](#indexing-functions) for the
on-disk shape).
+(see [Index Definitions](#index-definitions) for the on-disk shape).
The index itself is stored in the Hudi metadata table under the partition
`secondary_index_<index_name>`. As with other
metadata partitions the entry is a key/value, but the encoding is a little
more nuanced.
@@ -532,12 +600,34 @@ For example, a secondary index on the `city` column, for
a record with `city = C
chennai$id1 -> {"isDeleted": false}
```
-Each secondary-index partition is tagged with a `HoodieIndexVersion` (stored
on the corresponding `HoodieIndexDefinition`).
-Table version 8 constrained every secondary-index partition to `V1` (the
encoding described above). Table version 9
-introduces `V2`, which shards records by the primary (record) key rather than
by the secondary key. This makes secondary-index
-updates cheaper on writes with skewed secondary values, at the cost of
secondary-key range scans having to visit more file
-groups. Readers pick their scan strategy from the per-partition
`HoodieIndexVersion`. New tables created on version 9
-default to `V2` for secondary indexes; existing `V1` partitions from upgraded
tables continue to be read with the V1 encoding.
+**Partitioning** decides which file group of the index partition an entry is
written to. Hudi hashes a portion of the
+record key and takes that value modulo the number of file groups. Which
portion is hashed is governed by the
+[`HoodieIndexVersion`](#index-versions) recorded on the index's
`HoodieIndexDefinition`:
+
+* **`V1`**, the default for table version 8, hashes the whole
`<escaped-secondary-key>$<escaped-primary-key>` key.
+ Entries sharing a secondary value are distributed across all file groups,
so resolving a secondary value to its
+ records reads every file group unless the primary key is already known.
+* **`V2`**, the default for table version 9, hashes only the leading
`<escaped-secondary-key>$` portion. All entries
+ sharing a secondary value therefore reside in one file group, and a lookup
by secondary value alone reads that
+ single file group.
+
+The strategy is selected per partition from its recorded version, so `V1`
partitions on an upgraded table continue to be
+read as `V1` while indexes created afterwards on the same table use `V2`.
+
+#### Limitations
+
+* A secondary index may be defined on **exactly one column**. Attempting
more fails with
+ `Only one column can be indexed for functional or secondary index.`
+* The indexed column must resolve to one of the schema types `string`,
`int`, `long`, `float`, `double`, `date`,
+ `time`, or a **UTC-adjusted** `timestamp`. In Spark SQL terms that covers
`string`, `tinyint`, `smallint`, `int`,
+ `bigint`, `float`, `double`, `date` and `timestamp`, since `tinyint` and
`smallint` are represented as `int`.
+ Rejected are `decimal`, `boolean`, `binary`, the nested types `array`,
`map` and `struct`, and local
+ (non-UTC-adjusted) timestamps. A nullable column is supported when its
non-null branch is a supported type.
+* The **type of an indexed column cannot be changed** while the index
exists. Both a SQL
+ `ALTER TABLE ... ALTER COLUMN ... TYPE ...` and a write that evolves the
column through schema-on-read fail with
+ `Column '<column>' has secondary index '<index>' and cannot evolve from
schema '<old>' to '<new>'`. Drop the index,
+ change the type, then rebuild the index. Changing only a column's
nullability is permitted and does not require
+ dropping the index.
### Expression Indexes
