mhilton opened a new issue, #11905:
URL: https://github.com/apache/datafusion/issues/11905
### Describe the bug
It is not possible to create a `ScalarValue::Dictionary` value where the
resulting arrow array will return `true` on the `is_null` method.
### To Reproduce
The following application shows it quite well.
```rust
use datafusion::{arrow::datatypes::DataType, scalar::ScalarValue};
fn main() {
let v = ScalarValue::Dictionary(Box::new(DataType::Int32),
Box::new(ScalarValue::Utf8(None)));
let arr = v.to_array().unwrap();
assert!(arr.is_null(0));
}
```
### Expected behavior
Calling `is_null` on the resulting `Array` should return `true`.
### Additional context
I am attempting to use `NULL` `ScalarValue` literals to harmonise the
schemas of two tables that are being unioned and then sorted. I would like the
`NULL`s to sort last, but because the arrow sort kernel doesn't think these
values are `NULL` (`is_valid` returns `true`) and the way `Utf8` arrays are
layed out they are processed as zero-length strings, which are sorted first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
