Jackie-Jiang opened a new pull request, #19072:
URL: https://github.com/apache/pinot/pull/19072
## Summary
Adds an **opt-in, per-column** reload-time backfill that reconstructs a
column's null value vector for segments that were created before null handling
was enabled, by treating every stored value equal to the column's default null
value as null. This lets operators enable null handling on a table and have `IS
NULL` / null-aware queries work on already-ingested data without re-ingesting.
The reconstruction mirrors how ingestion records nulls:
- single-value: a doc is null when its stored value equals the column's
default null value;
- multi-value: a doc is null when its entry is a single-element array
holding the default null value.
### Opt-in and safety
The backfill is **lossy** — a genuine value that happens to equal the
default null value is also marked null — so it is opt-in per column, via the
null value vector index config:
```json
"fieldConfigList": [
{ "name": "status", "indexes": { "null": { "backfill": true } } }
]
```
- A new `NullValueVectorConfig` carries the `backfill` flag; its `enabled`
state is still derived from null handling (column-based `isNullable` or the
table-level `nullHandlingEnabled`), and the two are merged in
`NullValueIndexType`'s deserializer.
- Enable it only for columns whose default null value is a non-occurring
sentinel (e.g. a dimension's `MIN_VALUE`), not for metric/boolean columns whose
default is a common value like `0`.
- `MAP`/complex types are rejected at table-config validation time
(`NullValueIndexType.validate`), since the default (empty map) is an ordinary
value and `OPEN_STRUCT`-backed maps have no single scannable parent forward
index. A `TODO` is left to revisit if complex-type null handling matures.
### `nonNull` column-metadata flag
A new `nonNull` column-metadata property records the "null handling enabled,
but no null values" case. A bitmap file is written only when a column actually
has nulls (unchanged from today); the no-null case is recorded via this flag
instead. This keeps behavior **symmetric between segment creation and
backfill** — no empty bitmap files — and makes the backfill **idempotent** (a
flagged column is not re-scanned on subsequent reloads) while distinguishing a
null-handled-but-empty column from one that was never null-handled.
### Backward / mixed-version compatibility
- The metadata flag defaults to `false` and is additive; older segments are
unaffected.
- The feature is off unless a column explicitly opts in.
- Rolling-upgrade note: once a column opts in (writes `indexes.null` to the
table config), a node still running the pre-change code will fail to resolve
that table's index configs (the old null-vector deserializer treats the config
as an exclusive alternative and every column already gets a derived config).
Enable `backfill` only after the whole cluster is upgraded; a rollback with the
config still set will fail on that table.
### Tests
- `NullValueVectorHandlerTest` — backfill across SV/MV and dict/raw scalar
types, selectivity (an un-opted column holding the sentinel is left untouched),
the creation-time `nonNull` flag write, and idempotency on reload.
- `NullValueIndexTypeTest` — config resolution (`enabled` derived from null
handling) and `validate` rejecting `MAP` opt-in.
- `TableConfigUtilsTest` — end-to-end validation: scalar opt-in passes,
`MAP` opt-in is rejected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
