raghuvanshraj commented on issue #27089:
URL: https://github.com/apache/datafusion/issues/27089#issuecomment-6949297701

   @adriangb agree on both counts. The shredded data is not copied into `value` 
here, so inference that ignores shredded columns would lose `id` and `status` 
for `a.parquet` without the adapter. Its residual `value` is 9 bytes and holds 
only `service`:
   
   ```sql
   SELECT event:id, event['value'] AS residual_value FROM 
'mixed_events/a.parquet';
   ```
   
   ```text
   +---------------------------------+--------------------+
   | mixed_events/a.parquet.event:id | residual_value     |
   +---------------------------------+--------------------+
   | 1                               | 02010100040d617069 |
   | 2                               | 02010100040d617069 |
   | 3                               | 02010100040d776562 |
   +---------------------------------+--------------------+
   ```
   
   With the adapter installed the table shape does not matter. Declaring the 
table as plain `VARIANT`, so no `typed_value` in the table schema at all, over 
the same two files also returns the right rows:
   
   ```sql
   CREATE EXTERNAL TABLE unshredded_schema (event VARIANT) STORED AS PARQUET 
LOCATION 'mixed_events/';
   SELECT arrow_typeof(event) FROM unshredded_schema LIMIT 1;
   SELECT event:id, event:status FROM unshredded_schema ORDER BY 
event:id::BIGINT;
   ```
   
   ```text
   +-----------------------------------------------------------------------+
   | arrow_typeof(unshredded_schema.event)                                 |
   +-----------------------------------------------------------------------+
   | Struct("metadata": non-null BinaryView, "value": non-null BinaryView) |
   +-----------------------------------------------------------------------+
   +----------------------------+--------------------------------+
   | unshredded_schema.event:id | unshredded_schema.event:status |
   +----------------------------+--------------------------------+
   | 1                          | 200                            |
   | 2                          | 500                            |
   | 3                          | 504                            |
   | 4                          | 200                            |
   | 5                          | 503                            |
   | 6                          | 200                            |
   +----------------------------+--------------------------------+
   ```
   
   So the adapter is the piece that has to be there. What inference puts in the 
table schema is a separate choice, and I am fine either way.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to