raghuvanshraj commented on issue #27089:
URL: https://github.com/apache/datafusion/issues/27089#issuecomment-6949236514

   @alamb it is the second one, the per-file cast null-fills 
`typed_value.status` for `b.parquet`. Schema merging is not the problem. It 
does not pick the first file, it unions the two `typed_value` structs, so the 
table has `typed_value.id` and `typed_value.status` whichever file is listed 
first. `mixed_events/` is the directory from the repro above. 
`mixed_events_rev/` holds the same two files with `a.parquet` renamed so it 
sorts after `b.parquet`:
   
   ```shell
   $ ls mixed_events mixed_events_rev
   mixed_events:
   a.parquet  b.parquet
   
   mixed_events_rev:
   b.parquet  z.parquet
   ```
   
   ```sql
   SELECT arrow_typeof(event) FROM 'mixed_events/' LIMIT 1;
   SELECT arrow_typeof(event) FROM 'mixed_events_rev/' LIMIT 1;
   ```
   
   ```text
   Struct("metadata": non-null BinaryView, "value": BinaryView, "typed_value": 
Struct("id": non-null Struct("value": BinaryView, "typed_value": Int64), 
"status": Struct("value": BinaryView, "typed_value": Int64)))
   Struct("metadata": non-null BinaryView, "value": BinaryView, "typed_value": 
Struct("id": non-null Struct("value": BinaryView, "typed_value": Int64), 
"status": Struct("value": BinaryView, "typed_value": Int64)))
   ```
   
   Rows 4-6 are null in both directories. That union is a fine table shape for 
Variant, the shredded file already reads through it.
   
   The wrong rows come from the per-file cast. For `b.parquet` the default 
adapter sees `event` with a different struct type and wraps it in a `CastExpr` 
([schema_rewriter.rs#L774-L778](https://github.com/apache/datafusion/blob/3e55e62388ad60c2302660fe5d728e346ab6309f/datafusion/physical-expr-adapter/src/schema_rewriter.rs#L774-L778)),
 and the struct cast fills the missing `typed_value.status` child with nulls 
([nested_struct.rs#L107-L111](https://github.com/apache/datafusion/blob/3e55e62388ad60c2302660fe5d728e346ab6309f/datafusion/common/src/nested_struct.rs#L107-L111)).
 Under the shredding spec a field whose `value` and `typed_value` are both null 
means "this field does not exist" 
([VariantShredding.md#L195](https://github.com/apache/parquet-format/blob/5cc18cae643835b19f869d3cffb085cd3c04eb15/VariantShredding.md#L195)),
 so `status` is never looked up in the residual `value` bytes where `b.parquet` 
actually has it.
   
   `VariantExprAdapterFactory` replaces that cast with a reshred of the file's 
layout into the table's 
([expr_adapter.rs#L211](https://github.com/datafusion-contrib/datafusion-variant/blob/8141d3f/src/expr_adapter.rs#L211)),
 which is why `CREATE EXTERNAL TABLE` returns the right rows. The url table 
builds its `ListingTable` without any adapter factory 
([dynamic_file.rs#L85-L95](https://github.com/apache/datafusion/blob/3e55e62388ad60c2302660fe5d728e346ab6309f/datafusion/core/src/datasource/dynamic_file.rs#L85-L95)),
 so it gets the default one and the null fill.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to