linliu-code opened a new pull request, #20054:
URL: https://github.com/apache/hudi/pull/20054

   ### Describe the issue this Pull Request addresses
   
   issue: #20053
   
   Token search over a text column (`WHERE message has 'disk' and 'quota'`) 
cannot use any Hudi index today, so it reads every data file. This draft is a 
copy-on-write prototype of a full-text index in the metadata table, meant to 
inform an RFC; it is not intended to merge as is.
   
   ### Summary and Changelog
   
   Adds `CREATE INDEX idx ON t USING full_text(col)` for COW tables and five 
SQL predicates (`hudi_has_token`, `hudi_has_all_tokens`, `hudi_has_any_tokens`, 
`hudi_has_token_prefix`, `hudi_has_phrase`). The file index uses the new index 
to skip base files that hold no row satisfying the full-text filter, including 
predicates combined with AND, OR and NOT.
   
   - New MDT partition type `FULL_TEXT_INDEX` and a nullable 
`FullTextIndexMetadata` field in `HoodieMetadata.avsc`. Per (term, base file) 
it stores a presence entry with the file's row count and the number of rows 
holding the term and, unless the term is dense in the file, a roaring bitmap of 
the row ordinals holding it.
   - `FullTextIndexer` indexes each base file a commit writes and removes the 
entries of base files it replaces (COW rewrites, clustering, insert overwrite).
   - `FullTextIndexSupport` in `HoodieFileIndex` prunes a slice only when a 
per-file coverage marker for its exact base file is visible in the index view 
being read, evaluates the pushed filters per file on row positions (AND as 
intersection, OR as union, NOT as the complement of rows that surely match, so 
a file where `disk` and `quota` only occur in different rows is pruned for 
either `hudi_has_all_tokens(m, 'disk quota')` or `hudi_has_token(m, 'disk') AND 
hudi_has_token(m, 'quota')`), keeps slices with log files, and is not used for 
incremental or older time-travel reads. The marker makes pruning safe when the 
reader's file slices and the metadata table view are at different instants.
   - Tokenizer and key layout in `FullTextIndexUtils`; DDL wiring in 
`CreateIndexCommand`, `HoodieSparkIndexClient`, `HoodieIndexUtils` and index 
scheduling.
   
   ### Impact
   
   New public SQL syntax and functions, and a new metadata table partition and 
record field (additive; existing partitions are unchanged). Measured one commit 
before the coverage-marker change, on MS MARCO (8.84M passages in 16 COW base 
files, 200 real queries), queries read 2.08 of 16 files on average with 
identical results to a full scan, and median latency dropped from 6.41 s to 
0.78 s. The index is 0.68 GB. The cost is on writes: a 1% upsert that rewrites 
every file took 216 s with the index versus 22.5 s without.
   
   ### Risk Level
   
   medium. It adds a metadata table record type and field; nothing changes for 
tables without the index. Verified with `TestFullTextIndex` (co-occurrence 
fixture, COW upsert with and without entry removal, a reader and index view at 
different instants, a column added by schema evolution, time travel, 
clustering, insert overwrite, rollback, CUSTOM merge mode rejection, keyless 
tables, AND/OR/NOT, prefix and phrase predicates, and a randomized test over 
random predicate trees, at both a low and a full dense ratio, with null texts 
and with files whose every row holds a phrase's words in other orders, that no 
file with a match is ever pruned; the randomized test fails on the previous 
commit, which pruned matching files for `NOT hudi_has_phrase`), the existing 
`TestSecondaryIndex` and `TestHoodieMetadataPayload`, and mutation checks on 
the tokenizer, the bitmap intersection, the base-instant check, the coverage 
check and the NOT complement.
   
   Known limitations of this prototype: COW only; pruning is per file: the 
stored row ordinals and per-file counts decide which files are read, but are 
not yet used to skip rows within a file or to answer counts without reading 
data, and dense terms store no positions; tokenization follows the JVM's 
Unicode tables, so writers and readers on different JDK versions could disagree 
on rare characters; renaming or dropping an indexed column is not blocked (base 
files written before the column existed, or under its old name, are simply 
never pruned); when the full-text index prunes some files, the other index 
supports are not consulted for the query's remaining predicates; a very short 
`hudi_has_token_prefix` prefix reads every matching presence entry into the 
driver.
   
   ### Documentation Update
   
   none for this draft. A merged version would need the index documented on the 
website and an RFC.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Enough context is provided in the sections above
   - [ ] Adequate tests were added if applicable
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to