vinothchandar commented on code in PR #19613: URL: https://github.com/apache/hudi/pull/19613#discussion_r3775829259
########## rfc/rfc-110/rfc-110.md: ########## @@ -0,0 +1,639 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--> + +# RFC-110: Native Full-Text Search Index + +## Proposers + +- @danny0405 + +## Approvers + +- TBD + +## Status + +Issue: TBD + +> The RFC number is provisional until the community assigns an issue and +> accepts the proposal. RFC-109 is the highest numbered proposal in this +> checkout, so this draft uses RFC-110 to make repository review practical. + +## Abstract + +This RFC proposes a native, relevance-ranked full-text index for Apache Hudi. +It supports token and phrase search over string columns, exposes search through +a Spark table-valued function (TVF), and builds and maintains indexes through +the Hudi metadata table (MDT) indexing lifecycle. + +The search engine follows Lance's useful architectural choices without making +Lance a storage dependency: immutable segments, a compact term dictionary, +compressed posting lists, document-length statistics, BM25 ranking, positions, +and block-max WAND. Hudi owns the analyzer contract and on-disk format. The +format and hot search path are implemented in a Rust crate kept in the Hudi +repository and called through a narrow Java native boundary. + +The MDT remains authoritative for index definitions, visibility, coverage, +rollbacks, and cleaning. Large immutable posting payloads are sidecar files in +an auxiliary directory owned by the MDT rather than values embedded in HFiles. +An MDT commit atomically publishes descriptors for already durable payloads. + +Queries are snapshot-safe. The default `complete` mode combines native index +results with a raw scan of source file slices not covered by a compatible +segment. An opt-in `fast` mode searches only covered data and reports that it +may omit matches. + +## Background + +Hudi indexes currently answer questions such as which files might contain a +record key or a value range. Full-text search has a different contract: analyze +free text into terms, locate matching documents, optionally verify positions, +and rank the best documents. Sending this workload to Elasticsearch or +OpenSearch is effective, but creates a second ingestion pipeline and a second +source of snapshot and retention truth. + +This proposal builds on: + +- [RFC-45](../rfc-45/rfc-45.md), which introduced asynchronous MDT indexing; +- [RFC-77](../rfc-77/rfc-77.md), which established dynamically named secondary + index partitions and index definitions; +- [RFC-102](../rfc-102/rfc-102.md), whose vector-search TVF provides a useful + SQL precedent; and +- RFC-109, the native vector-index proposal listed in the RFC catalog. Text and + vector search should eventually share native artifact packaging, storage + adapters, and top-k execution utilities, but their persistent formats remain + independent. + +### Design principles + +1. The Hudi timeline is the source of snapshot truth. +2. Index creation, visibility, rollback, and cleaning use MDT components. +3. Immutable payloads support object-store range reads and safe caching. +4. Ranking is independent of how the index is physically partitioned. +5. Freshness and incompleteness are explicit query properties. +6. The Java/native interface is small enough to replace without changing SQL. + +### Goals + +- Rank token, boolean, prefix, fuzzy, and phrase queries on one string column. +- Support copy-on-write (COW) and merge-on-read (MOR) tables. +- Build asynchronously and incrementally using the MDT indexer lifecycle. +- Guarantee snapshot-correct results in the default search mode. +- Provide an object-store-friendly native format with bounded memory usage. +- Keep the initial Rust implementation inside the Hudi repository. + +### Non-goals + +- Elasticsearch API, aggregation, highlighting, or percolator compatibility. +- Multi-column relevance models in the first format version. +- Updating posting lists in place. +- Replacing SQL predicate indexes or the record index. +- A general Rust rewrite of Hudi readers. + +### Alternatives considered + +**External Elasticsearch/OpenSearch.** This remains a valid integration, but it +requires change-data-capture coordination, separate retention, and explicit +mapping between external documents and a Hudi snapshot. + +**Embedding Tantivy.** Tantivy is mature and Rust-native. Its archive and +directory abstractions, however, become a second persistent compatibility +contract. A smaller Hudi-owned format gives the project control over source +file-slice identity, range-read layout, and MDT publication semantics. + +**Posting lists as MDT record values.** This would make MDT storage atomic, but +multi-gigabyte postings, merges, and random term reads fight the metadata +table's record-oriented HFile/MOR strengths. Small authoritative descriptors in +the MDT plus immutable sidecars preserve the lifecycle benefits without that +cost. + +## Implementation + +### Terminology + +| Term | Meaning | +| --- | --- | +| Logical index | A named SQL index and its immutable analyzer configuration. | +| MDT index partition | Dynamic `text_index_<name>` metadata partition containing authoritative control records. | +| Segment | An immutable set of documents and term postings built together. | +| Payload partition | A shard within a segment whose local document identifiers are `u32`. | +| Source slice | A Hudi base file and its ordered log files at a snapshot. | +| Coverage | Proof that a segment represents a particular source-slice fingerprint. | +| Raw tail | Eligible source slices not covered by compatible visible segments. | +| Document | One Hudi record, identified by record key and source information. | + +### SQL interface + +Creation follows Hudi's secondary-index syntax: + +```sql +CREATE INDEX article_body_fts +ON articles +USING text_index (body) +OPTIONS ( + 'base_tokenizer' = 'simple', + 'lower_case' = 'true', + 'language' = 'und', + 'with_position' = 'true', + 'posting_block_size' = '128' +); +``` + +The existing `HoodieIndexDefinition` is populated as follows: + +```json +{ + "indexName": "article_body_fts", + "indexType": "text_index", + "sourceFields": ["body"], + "indexFunction": "tokenize", + "indexOptions": { + "base_tokenizer": "simple", + "lower_case": "true", + "language": "und", + "with_position": "true", + "posting_block_size": "128" + } +} +``` + +Search is exposed as a Spark TVF rather than a boolean predicate because score Review Comment: how standard is this experience? If we are adding an index, why not implement this as SQL predicate? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
