Yuvraj-cyborg opened a new pull request, #19213:
URL: https://github.com/apache/datafusion/pull/19213
Closes #19169 - spark ascii need to have custom nullability
Rationale for this change
The default implementation of ```return_field_from_args``` always returned
true for the nullable field, which is incorrect. For most scalar functions
including Spark's ```ascii```, the output should only be nullable if any of the
input arguments are nullable. This is the standard null-propagation behavior in
SQL.
For example:
- ```ascii(non_null_column)``` should return a non-nullable result
- ```ascii(nullable_column)``` should return a nullable result
The old behavior caused incorrect schema information to propagate through
the query plan.
Changes in PR :
1. Fixed the default ```return_field_from_args```:
- Changed from always returning ```nullable = true``` to computing
nullability as ```args.arg_fields.iter().any(|f| f.is_nullable())```
- Added documentation explaining the null propagation behavior
- Fixed the example in the doc comment that calculated ```nullable``` but
didn't use it
2. Added a regression test ```test_return_field_nullability_from_args```
that verifies:
- All non-nullable inputs → non-nullable output
- Any nullable input → nullable output
- All nullable inputs → nullable output
- No inputs → non-nullable output
3. Updated existing tests in ```predicate_bounds.rs``` that were relying on
the old incorrect behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
