paleolimbot commented on code in PR #24865:
URL: https://github.com/apache/datafusion/pull/24865#discussion_r3918956484
##########
datafusion/functions/src/utils.rs:
##########
@@ -69,6 +70,26 @@ macro_rules! get_optimal_return_type {
};
}
+/// Returns the field metadata shared by every argument that can contribute a
+/// value to a conditional function's result.
+///
+/// Fields with a `Null` data type (untyped NULL literals) carry no metadata
+/// and are ignored. If the remaining fields disagree on metadata, the result
+/// carries none: propagating one argument's metadata (e.g. an Arrow extension
+/// type name) would claim a type identity for values that other arguments may
+/// supply without it.
+pub(crate) fn unanimous_metadata<'a>(
Review Comment:
I think this may be too strict for some practical things that can happen
here. Notably, extension types that use JSON for parameters can have non-equal
JSON bytes, and unrelated field metadata that arrived unbeknownst to a user
(e.g., was present in an Arrow file or embedded Arrow schema in a Parquet file)
might not agree and could break existing (and completely valid) queries.
This should probably be a very lenient combination (possibly just merging
all metadata keys or returning the first encountered metadata)...engines can
insert extra constraints here if they would like to but if the return field
drops the extension name, extension aware functions will fail (like ours did!).
As an example, we would probably write an optimizer rule that checks for
equal `SedonaType`s for a case when or ifelse; however, if the return type of
the case when doesn't carry extension metadata, it will fail before the
optimizer rule can run.
This may also be more appropriate next to some related utilities:
https://github.com/apache/datafusion/blob/d02bd786de49f2ad3dc15cb947608fb0f01fe142/datafusion/common/src/metadata.rs#L75-L86
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
