raghuvanshraj commented on issue #27089:
URL: https://github.com/apache/datafusion/issues/27089#issuecomment-6918251523

   Two files, same shape of data, shredded differently:
   
   | file | shredded (`typed_value`) | left in `value` |
   |---|---|---|
   | `a.parquet` | `id`, `status` | `service` |
   | `b.parquet` | `id` | `service`, `status` |
   
   ```sql
   COPY (
     SELECT parse_json(event) AS event FROM (VALUES
         ('{"id": 1, "service":"api", "status":200}'),
         ('{"id": 2, "service":"api", "status":500}'),
         ('{"id": 3, "service":"web", "status":504}')
     ) AS input(event)
   ) TO 'mixed_events/a.parquet' STORED AS PARQUET
   OPTIONS ('variant.shredding' '{"event": "STRUCT<id BIGINT, status 
BIGINT>"}');
   
   COPY (
     SELECT parse_json(event) AS event FROM (VALUES
         ('{"id": 4, "service":"api", "status":200}'),
         ('{"id": 5, "service":"db",  "status":503}'),
         ('{"id": 6, "service":"web", "status":200}')
     ) AS input(event)
   ) TO 'mixed_events/b.parquet' STORED AS PARQUET
   OPTIONS ('variant.shredding' '{"event": "STRUCT<id BIGINT>"}');
   
   SELECT event:id, event:status FROM 'mixed_events/' ORDER BY event:id::BIGINT;
   ```
   
   ```text
   +------------------------+----------------------------+
   | mixed_events/.event:id | mixed_events/.event:status |
   +------------------------+----------------------------+
   | 1                      | 200                        |
   | 2                      | 500                        |
   | 3                      | 504                        |
   | 4                      | null                       |
   | 5                      | null                       |
   | 6                      | null                       |
   +------------------------+----------------------------+
   ```
   
   Expected `200, 503, 200` for rows 4-6. `SELECT count(*) FROM 'mixed_events/' 
WHERE event:status::BIGINT = 200` returns 1 instead of 3.
   
   The crate side is in: `CREATE EXTERNAL TABLE mixed_events STORED AS PARQUET 
LOCATION 'mixed_events/'` over the same two files returns `200, 503, 200`, 
because `register_all` wraps the session's `PARQUET` table factory and attaches 
`VariantExprAdapterFactory` (datafusion-contrib/datafusion-variant#149) to the 
tables it creates. The url-table path above does not go through that factory, 
so it still null-fills; @alamb filed that as 
datafusion-contrib/datafusion-variant#177. What is left for core is whether a 
registered extension type can get its adapter applied without a per-table or 
per-factory install.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to