paleolimbot opened a new issue, #26810:
URL: https://github.com/apache/datafusion/issues/26810
### Describe the bug
The `intersect_metadata_for_union()` function, which determines what
metadata is propagated through a UNION for aligned fields, is slightly too
strict: it discards metadata that doesn't agree, which strips metadata if two
`ARROW:extension:metadata`s are, say, `{"width":5}` and `"{ "width" : 5 }`
(despite the types being identical. We came across this problem because
GeoArrow's metadata includes a `"crs"` key where one `"crs"` can be represented
a few ways.
A note that this can be worked around with an optimizer rule (e.g.,
https://github.com/apache/sedona-db/pull/1520 ).
### To Reproduce
```rust
use std::collections::HashMap;
use datafusion_expr::expr::intersect_metadata_for_union;
fn main() {
let metadata = |json: &str| {
HashMap::from([
(
"ARROW:extension:name".to_string(),
"example.width".to_string(),
),
("ARROW:extension:metadata".to_string(), json.to_string()),
])
};
let left = metadata(r#"{"width":5}"#);
let right = metadata(r#"{ "width" : 5 }"#);
let merged = intersect_metadata_for_union([&left, &right]);
println!("{merged:?}");
}
```
### Expected behavior
_No response_
### Additional context
_No response_
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
