rangareddy commented on issue #17308:
URL: https://github.com/apache/hudi/issues/17308#issuecomment-5365819865
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-8517).
**Findings: confirmed, and the constraint is at the schema level - this is a
payload format change, not a code tweak.**
Your premise checks out exactly.
`hudi-common/src/main/avro/HoodieMetadata.avsc` defines the secondary index
value as:
```json
{
"name": "SecondaryIndexMetadata",
"doc": "Metadata Index that contains information about secondary keys
and the corresponding record keys in the dataset",
"type": [
"null",
{
"type": "record",
"name": "HoodieSecondaryIndexInfo",
"fields": [
{
"name": "isDeleted",
"type": "boolean",
"doc": "True if this entry has been deleted"
}
]
}
],
"default" : null
}
```
One field. There is nowhere to record which partition a record key lives in,
which is exactly why the index only works against a global RLI.
So the work is an MDT payload schema change with on-disk compatibility
consequences: existing secondary-index partitions contain the one-field record,
and readers must keep handling them. That probably means riding an
index-version bump rather than evolving in place - and usefully,
`SECONDARY_INDEX` already has version branching in
`HoodieIndexVersion.getCurrentVersion`, returning `V2` at table version 9 and
above and `V1` below. That is the natural seam for this.
The delete handling the description mentions gets harder too: with partition
in the value, a delete has to identify the right (secondary key, record key,
partition) tuple rather than assuming global uniqueness.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
