raghuvanshraj commented on issue #27089:
URL: https://github.com/apache/datafusion/issues/27089#issuecomment-6918251523
Two files, same shape of data, shredded differently:
| file | shredded (`typed_value`) | left in `value` |
|---|---|---|
| `a.parquet` | `id`, `status` | `service` |
| `b.parquet` | `id` | `service`, `status` |
```sql
COPY (
SELECT parse_json(event) AS event FROM (VALUES
('{"id": 1, "service":"api", "status":200}'),
('{"id": 2, "service":"api", "status":500}'),
('{"id": 3, "service":"web", "status":504}')
) AS input(event)
) TO 'mixed_events/a.parquet' STORED AS PARQUET
OPTIONS ('variant.shredding' '{"event": "STRUCT<id BIGINT, status
BIGINT>"}');
COPY (
SELECT parse_json(event) AS event FROM (VALUES
('{"id": 4, "service":"api", "status":200}'),
('{"id": 5, "service":"db", "status":503}'),
('{"id": 6, "service":"web", "status":200}')
) AS input(event)
) TO 'mixed_events/b.parquet' STORED AS PARQUET
OPTIONS ('variant.shredding' '{"event": "STRUCT<id BIGINT>"}');
SELECT event:id, event:status FROM 'mixed_events/' ORDER BY event:id::BIGINT;
```
```text
+------------------------+----------------------------+
| mixed_events/.event:id | mixed_events/.event:status |
+------------------------+----------------------------+
| 1 | 200 |
| 2 | 500 |
| 3 | 504 |
| 4 | null |
| 5 | null |
| 6 | null |
+------------------------+----------------------------+
```
Expected `200, 503, 200` for rows 4-6. `SELECT count(*) FROM 'mixed_events/'
WHERE event:status::BIGINT = 200` returns 1 instead of 3.
The crate side is in: `CREATE EXTERNAL TABLE mixed_events STORED AS PARQUET
LOCATION 'mixed_events/'` over the same two files returns `200, 503, 200`,
because `register_all` wraps the session's `PARQUET` table factory and attaches
`VariantExprAdapterFactory` (datafusion-contrib/datafusion-variant#149) to the
tables it creates. The url-table path above does not go through that factory,
so it still null-fills; @alamb filed that as
datafusion-contrib/datafusion-variant#177. What is left for core is whether a
registered extension type can get its adapter applied without a per-table or
per-factory install.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]