walterddr commented on issue #10381:
URL: https://github.com/apache/pinot/issues/10381#issuecomment-1751448407
the discussion here is: the bottom line is this discussion should only be
regarded to query time, no ingestion time behavior change is discussed here.
now with the query time:
config
===
- if QueryOption `enableNullhandling` flag is set, consider all columns to
be nullable (exactly like current v1 behavior)
- if QueryOption is not set, use the schema level column `isNullableField`
to consider whether it is nullable
- this behavior should be honored by both v1 and v2
- for v2, enableNullHandling flag should not be used b/c it is not a
runtime controllable feature
- tables are registered in catelog during server start time. At that
time, column nullability is already loaded
- we need a schema watcher to change table cache in case schema is
changed but that's orthogonal to the discussion here
config interpretation (runtime behavior, only on v1)
===
if a column is configured nullable, we read the null vector -- if it doesn't
exist, there's no null in this segment for this column
if a column is configured as non-nullable, we skip the null vector, event if
it exist, and directly use the forward index value
The main discussion point here is whether we should introduce a "new query
option" specifically for v2, say `enableNullHandlingIfColumnIsNullable` which
- if `enableNullHandling` is set (only on v1) keep current v1 behavior
- if `enableNullHandlingWhenColumnIsNullable` is set, honor schema column
level nullability
- if nothing is set, all columns are non-nullable (which only affects
runtime behavior, at v2 planning time column is still considered nullable, it
will only affect some optimization rule, but final results will be the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
