walterddr commented on issue #10381: URL: https://github.com/apache/pinot/issues/10381#issuecomment-1745823112
> Thanks @walterddr for the doc. > > From a user point of view, I feel we should keep columnLevelNullability as a new feature altogether. +1 on this point, but i would like to keep this issue solely discuss the impact of this new feature on query time. > > So if column level nullability is disabled, things can stay as they are now. "As they are now" is a bit loose, let me explain below: > If column level nullability is enabled, then we ignore table level nullability altogether, and both v1 and v2 engine should honor it in the same consistent way, which is: --> this is the part why i want this to stay as a query-time only flag * regardless of column level nullability is enabled. we use table-nullability flag (which only used during ingestion time) for ingestion * later we might use column level to control table level, but that's a separate issue all together and query-time null can only be enabled if table has enabled null value ingestion flag. > > 1. During ingestion null vectors would be created based on column level nullability (this is independent of query engine anyways) --> see above, we do not do that for now, ingestion doesn't change for this new feature; > 2. On query side both v1 and v2 engine would behave the same way when the null handling query option is enabled/disabled. null handling is always enabled on v2 regardless, b/c there's no way to not handle null, say in a LEFT JOIN situation. so null is always enabled > 3. Null filters like IS NULL/IS NOT NULL should work consistently across the two engines. - currently IS_NULL and IS_NOT_NULL is not supported by v2, so the current behavior on v2 is as if the column is not nullable. which suites the default not-nullable behavior before adding this column level null flag --> if this is what we meant by "behavior doesn't change", then yes the behavior doesn't change (on v2) - i have yet to check on v1 but as long as we dont use these settings in v1 we are fine. unless we want them to match -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
