yashmayya opened a new pull request, #18447:
URL: https://github.com/apache/pinot/pull/18447
## Summary
Pre-1.6.0 segments don't write the `lengthOfLongestElement` metadata key.
The loader in `ColumnMetadataImpl.Builder.build()` backfills it from
`dictionaryElementSize`, but only when the latter is strictly positive. For
dictionary-encoded STRING columns where every value is empty (e.g. an
empty-string default column added via schema evolution),
`dictionaryElementSize` is `0`, so the fallback is skipped and
`lengthOfLongestElement` is left at the `UNAVAILABLE` sentinel (`-1`). That
`-1` then propagates into `StringDictionary` as `numBytesPerValue`, and
`BaseImmutableDictionary.getBuffer()` throws `NegativeArraySizeException` at
query time:
```
java.lang.NegativeArraySizeException: -1
at BaseImmutableDictionary.getBuffer(BaseImmutableDictionary.java:295)
at StringDictionary.getBuffer(StringDictionary.java:85)
at StringDictionary.readStringValues(StringDictionary.java:164)
```
Regression introduced in #18280, which switched `DictionaryIndexType` from
reading the legacy `lengthOfEachEntry` (always present) to the new
`lengthOfLongestElement` (only present on segments built >= 1.6.0).
Fix: relax the backfill guard to `>= 0` so a zero-length longest entry still
canonicalizes correctly. The added test reproduces the exact production stack
trace before the fix and passes after.
## Test plan
- [x] New
`ImmutableDictionaryTest#testStringDictionaryReadOnPre16OldSegmentMetadata`
reproduces the original `NegativeArraySizeException` on the pre-fix code path
and passes after the fix
- [x] Full `ImmutableDictionaryTest`, `ColumnMetadataImplTest`,
`ColumnMetadataTest` pass
- [x] spotless / checkstyle / license clean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
