adriangb opened a new issue, #26915: URL: https://github.com/apache/datafusion/issues/26915
### Describe the bug A `UNION` between a column with an Arrow extension type (e.g. `arrow.uuid` on `FixedSizeBinary(16)`) and a value of a different storage type fails with an internal error. `UNION ALL` of the same inputs does not error, but silently drops the extension type from the result. Found while reviewing https://github.com/apache/datafusion/pull/21714, which makes this reachable with plain SQL. It reproduces identically on `main` using the test-only `UUID` type planner registered for `cast_extension_type_metadata.slt`, so it predates that PR. ### To Reproduce On #21714 (or in `cast_extension_type_metadata.slt` on `main`): ```sql CREATE TABLE u (id UUID); INSERT INTO u VALUES (X'00010203040506070809000102030506'); -- 1. UNION: internal error SELECT id FROM u UNION SELECT X'00010203040506070809000102030506'; -- Internal error: Physical input schema should be the same as the one converted from logical input schema. Differences: -- - field metadata at index 0 [id]: (physical) {} vs (logical) {"ARROW:extension:name": "arrow.uuid"}. -- 2. UNION ALL: no error, extension type silently lost SELECT arrow_metadata(id, 'ARROW:extension:name') FROM (SELECT id FROM u UNION ALL SELECT X'00010203040506070809000102030506') LIMIT 1; -- NULL ``` `UNION` / `INTERSECT` between two `UUID` inputs work fine. ### Expected behavior Either a result typed as the extension type, or a planning error saying the types can't be unified. Never an internal error, and never silently dropping the extension type. For comparison: - Postgres: `SELECT id FROM u UNION SELECT '\x00...'::bytea` → `ERROR: UNION could not convert type bytea to uuid`; with a text literal the result is `uuid`. - DuckDB: `UNION` / `UNION ALL` with a text literal gives a `UUID` result. ### Additional context Related: https://github.com/apache/datafusion/issues/26178 (same internal error, different trigger) and https://github.com/apache/datafusion/issues/26810 (`intersect_metadata_for_union` too strict). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
