chrevanthreddy opened a new pull request, #19109:
URL: https://github.com/apache/hudi/pull/19109
## Summary
Add the initial Spark DDL and option scaffolding for vector indexes.
This PR is intentionally thin and focuses on the `#19096` scope only:
- `CREATE INDEX ... USING VECTOR`
- vector option surface and validation
- `SHOW INDEXES` / `DROP INDEX` scaffolding for vector index definitions
- minimal metadata partition naming/version plumbing for vector index
definitions
- Spark SQL and metadata/versioning test coverage for the new scaffolding
## What this PR includes
- Adds a minimal `VectorIndexOptions` contract for vector DDL options,
including:
- `vector.dimension`
- `vector.metric`
- `vector.algorithm`
- `vector.quantizer`
- probe / cluster / refine / iteration settings
- `vector.rabitq.total_bits`
- Adds vector index definition construction and early validation:
- exactly one indexed column
- indexed column must exist
- indexed column must be `VECTOR`
- configured dimension must match the schema dimension
- Wires `USING VECTOR` through Spark SQL command handling
- Registers vector index definitions through the Spark index client
- Makes `SHOW INDEXES` include vector index definitions even before full
metadata partition materialization
- Adds minimal metadata naming/version support for `vector_index_*`
## What this PR does not include
To keep review scope small, this PR does **not** include:
- MDT payload/schema changes for vector postings
- bootstrap / clustering / posting generation logic
- vector query planning or read path execution
- approximate candidate generation or exact rerank
- rebuild / cleanup semantics beyond thin DDL definition handling
Those pieces are intended for follow-up PRs in the RFC-104 sequence.
## Validation
### Completed
- added SQL parser/analyzer + execution coverage in `TestIndexSyntax`
- added metadata partition coverage in `TestMetadataPartitionType`
- added versioning coverage in `TestHoodieIndexVersion`
- verified local Maven/JDK setup under JDK 17
### Blocked / known repo-local validation issues
Full Maven validation in this worktree is currently blocked by unrelated
pre-existing issues in the local branch/worktree environment, including:
- unrelated compile failures in existing modules outside this PR scope
- missing local snapshot dependency resolution for some Spark artifacts when
compiling leaf modules directly
Because of that, this PR is being opened with the code scoped and committed,
but without a clean end-to-end local reactor pass from this environment.
## Related
- Parent: #19094
- Child scope: #19096
- RFC umbrella: #18676
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
