hudi-agent commented on code in PR #19650:
URL: https://github.com/apache/hudi/pull/19650#discussion_r3797373295
##########
website/learn/tech-specs.md:
##########
@@ -463,6 +463,70 @@ The other fields can also be optional for writers
depending on whether protectio
### Naming
Indexes are stored under `.hoodie/metadata` storage path, with separate
partitions of the form `<index_type>_<index_name>`.
+### Index Definitions
+
+Every index carries a definition, serialized to JSON under the path in
`hoodie.table.index.defs.path`
+(default `.hoodie/.index_defs/index.json`). All definitions for a table live
in a single file, keyed by the
+metadata-table partition name:
+
+```json
+{
+ "indexDefinitions": {
+ "<partition_name>": {
+ "indexName": "<partition_name>",
+ "indexType": "<index_type>",
+ "indexFunction": "<index_function>",
+ "version": "<index_version>",
+ "sourceFields": ["<column_1>", "<column_2>"],
+ "indexOptions": {}
+ }
+ }
+}
+```
+
+* `indexType` is one of `files`, `column_stats`, `partition_stats`,
`bloom_filters`, `record_index`,
+ `secondary_index` or `expr_index`.
+* `indexFunction` is the transform applied to the source column, `identity`
unless the index is an expression index.
Review Comment:
🤖 This bullet could be more precise about how `indexFunction` is actually
populated. As written it implies every non-expression index stores `identity`,
but that doesn't match the definition builders: built-in partitions created via
`HoodieTableMetadataUtil` (e.g. `record_index`) never call
`withIndexFunction(...)`, so `indexFunction` ends up as the empty string, not
`identity`. Meanwhile secondary indexes go through the same `HoodieIndexUtils`
builder as expression indexes and default to `identity`
(`options.getOrDefault(EXPRESSION_OPTION, IDENTITY_TRANSFORM)`), and an
expression index likewise shows `identity` when no explicit function is given.
So `identity` isn't the marker of a non-expression index — it might help to say
instead that `indexFunction` holds the expression-index transform (defaulting
to `identity`), is `identity` for secondary indexes, and is empty for the
built-in index types.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
