Jackie-Jiang commented on code in PR #19072:
URL: https://github.com/apache/pinot/pull/19072#discussion_r3648894675
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/nullvalue/NullValueIndexType.java:
##########
@@ -110,20 +116,53 @@ protected IndexReaderFactory<NullValueVectorReader>
createReaderFactory() {
return ReaderFactory.INSTANCE;
}
+ @Override
+ public void validate(FieldIndexConfigs indexConfigs, FieldSpec fieldSpec,
TableConfig tableConfig) {
+ if (indexConfigs.getConfig(this).isBackfill()) {
+ // Backfill reconstructs nulls by comparing each stored value against
the column's default null value, which is
+ // only meaningful for scalar stored types. MAP (and other complex
types) are not supported because:
+ // - the default null value for a MAP is an empty map — an ordinary
value rather than a rare sentinel — so
+ // treating every empty map as null would be far too lossy to be
safe; and
+ // - an OPEN_STRUCT-backed MAP is materialized into child columns with
no single scannable parent forward
+ // index, so there is nothing coherent to scan for the parent column.
+ // TODO: Revisit MAP/complex backfill if complex-type null handling
matures and a safe (non-occurring) sentinel
+ // default null value becomes available.
Review Comment:
Keeping the TODO deliberately. It sits directly under the paragraph
explaining *why* MAP/complex types are unsupported (the default null value for
a MAP is the empty map — an ordinary value rather than a rare sentinel — and an
OPEN_STRUCT-backed MAP has no single scannable parent forward index), so it
reads as a marker on documented design rationale rather than an untracked task.
Un-linked `TODO`s of exactly this shape are the existing convention in this
module, e.g. `SegmentPreProcessor` ("TODO: Reduce segment metadata reload...")
and `SegmentMetadataUtils` ("TODO: Revisit if we can save the overhead...").
Happy to swap in an issue link if a committer would prefer one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
