raghuvanshraj commented on issue #27089:
URL: https://github.com/apache/datafusion/issues/27089#issuecomment-6949297701
@adriangb agree on both counts. The shredded data is not copied into `value`
here, so inference that ignores shredded columns would lose `id` and `status`
for `a.parquet` without the adapter. Its residual `value` is 9 bytes and holds
only `service`:
```sql
SELECT event:id, event['value'] AS residual_value FROM
'mixed_events/a.parquet';
```
```text
+---------------------------------+--------------------+
| mixed_events/a.parquet.event:id | residual_value |
+---------------------------------+--------------------+
| 1 | 02010100040d617069 |
| 2 | 02010100040d617069 |
| 3 | 02010100040d776562 |
+---------------------------------+--------------------+
```
With the adapter installed the table shape does not matter. Declaring the
table as plain `VARIANT`, so no `typed_value` in the table schema at all, over
the same two files also returns the right rows:
```sql
CREATE EXTERNAL TABLE unshredded_schema (event VARIANT) STORED AS PARQUET
LOCATION 'mixed_events/';
SELECT arrow_typeof(event) FROM unshredded_schema LIMIT 1;
SELECT event:id, event:status FROM unshredded_schema ORDER BY
event:id::BIGINT;
```
```text
+-----------------------------------------------------------------------+
| arrow_typeof(unshredded_schema.event) |
+-----------------------------------------------------------------------+
| Struct("metadata": non-null BinaryView, "value": non-null BinaryView) |
+-----------------------------------------------------------------------+
+----------------------------+--------------------------------+
| unshredded_schema.event:id | unshredded_schema.event:status |
+----------------------------+--------------------------------+
| 1 | 200 |
| 2 | 500 |
| 3 | 504 |
| 4 | 200 |
| 5 | 503 |
| 6 | 200 |
+----------------------------+--------------------------------+
```
So the adapter is the piece that has to be there. What inference puts in the
table schema is a separate choice, and I am fine either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]