alamb opened a new issue, #27588:
URL: https://github.com/apache/datafusion/issues/27588

   ## Is your feature request related to a problem or challenge?
   
   - Part of #21308
   - Part of #24119.
   - related to #21306 (describes the usecase)
   
   The usecase is reading a single field from a [shredded 
Variant](https://github.com/apache/parquet-format/blob/master/VariantShredding.md)
 column stored in Parquet, using `variant_get` / `try_variant_get` from 
[datafusion-variant](https://github.com/datafusion-contrib/datafusion-variant):
   
   ```sql
   SELECT sum(try_variant_get(v, '$.x', 'BIGINT')) FROM events;
   ```
   
   If the variant stores `x` shredded as an int64 column, then this query 
should only read the int64 column, and not the variant `value` column (that has 
all the unaccessed fields)
   
   This is the same as happens with a normal struct field access
   
   ```sql
   
   SELECT sum(s['x']) from t;
   ```
    only reads `x` leaf column of a plain struct.
   
   
   So the query above only needs to read the `x` column (and `metadata`),
   
   
   Here is the shredding schema:
   
   ```text
   v: Struct(
     metadata: BinaryView,
     value: BinaryView,                <-- residual fields that are not shredded
     typed_value: Struct(
       x: Struct(value: BinaryView, typed_value: Int64),      < -- `$.x` 
shredded, `value` holds any rows that are not a BIGINT
       y: Struct(value: BinaryView, typed_value: Utf8View),
     )
   )
   ```
   
   DataFusion can already push functons into the Parquet scan (via 
`ScalarUDFImpl::placement`), but it still reads and decodes every leaf of `v` 
and then evaluates the function on the whole column.
   
   The Parquet reader only narrows the read for `get_field` (`GetFieldFunc`), 
which is hardcoded, as described in
   - hardcoded in `PushdownChecker` / `projection_read_plan.rs` as
   
   ### Example
   
   Using `dfv` (the datafusion-variant CLI) from 
https://github.com/datafusion-contrib/datafusion-variant.
   
   The file has 100k rows with the same two fields stored as a shredded Variant 
`v` and as a plain struct `s`:
   
   ```sql
   SET datafusion.execution.target_partitions = 1;
   
   CREATE TABLE t AS
   SELECT
     value AS id,
     parse_json('{"x": ' || value || ', "y": "' || repeat(md5(CAST(value AS 
VARCHAR)), 4) || '"}') AS v,
     named_struct('x', value, 'y', repeat(md5(CAST(value AS VARCHAR)), 4)) AS s
   FROM generate_series(1, 100000);
   
   COPY t TO 'events.parquet' STORED AS PARQUET
   OPTIONS ('variant.shredding' '{"v":"STRUCT<x BIGINT, y STRING>"}');
   
   CREATE EXTERNAL TABLE events STORED AS PARQUET LOCATION 'events.parquet';
   ```
   
   The plans for the struct access and the Variant access have the same shape: 
`get_field` and `try_variant_get` are pushed into the data source
   
   ```sql
   > EXPLAIN SELECT sum(s['x']) FROM events;
   
   ProjectionExec: expr=[sum(__datafusion_extracted_1)@0 as sum(events.s[x])]
     AggregateExec: mode=Single, gby=[], aggr=[sum(__datafusion_extracted_1)]
       DataSourceExec: file_groups={1 group: [[events.parquet]]}, 
projection=[get_field(s@2, x) as __datafusion_extracted_1], file_type=parquet
   
   > EXPLAIN SELECT sum(try_variant_get(v, '$.x', 'BIGINT')) FROM events;
   
   ProjectionExec: expr=[sum(__datafusion_extracted_1)@0 as 
sum(try_variant_get(events.v,Utf8("$.x"),Utf8("BIGINT")))]
     AggregateExec: mode=Single, gby=[], aggr=[sum(__datafusion_extracted_1)]
       DataSourceExec: file_groups={1 group: [[events.parquet]]}, 
projection=[try_variant_get(v@1, $.x, BIGINT) as __datafusion_extracted_1], 
file_type=parquet
   ```
   
   However, `EXPLAIN ANALYZE` shows that only the struct version actually reads 
less
   
   | query | scan `projection` | `bytes_scanned` | scan `elapsed_compute` |
   |---|---|---:|---:|
   | `sum(s['x'])` | `get_field(s@2, x)` | 326.0 KB | 12.07ms |
   | `sum(try_variant_get(v, '$.x', 'BIGINT'))` | `try_variant_get(v@1, $.x, 
BIGINT)` | **2.3 MB** | **72.77ms** |
   | `sum(v['typed_value']['x']['typed_value'])` (hand written access to the 
shredded leaf) | `get_field(v@1, typed_value, x, typed_value)` | 326.0 KB | 
15.03ms |
   
   The last row shows what is desired: reading the shredded leaf of the same 
Variant column directly with `get_field` scans 326 KB rather than 2.3MB
   
   note the last row is only correct when `x` is perfectly shredded
   
   ## Describe the solution you'd like
   
   1. A way for a `ScalarUDF` to tell the Parquet reader which nested leaves of 
its arguments it reads
   
   There are two draft PRs for (1) and (2), with different API shapes:
   - #25013
   - #26282
   
   ## Describe alternatives you've considered
   
   **Rewrite `variant_get` into `get_field` in a `PhysicalExprAdapter`.** For a 
file where the path is shredded with the requested type, the downstream crate 
could rewrite the call per file into `get_field(v, 'typed_value', 'x', 
'typed_value')`, which the reader already understands (third row of the table 
above). This is only correct when there are no fallback values in 
`typed_value.x.value`, which the adapter cannot know today as it is only given 
the schemas and not the file statistics. Any file with a fallback value would 
still read the whole column.
   
   ## Additional context
   
   - #21306
   - #21308
   - #24119
   - #20871
   - https://github.com/datafusion-contrib/datafusion-variant/issues/195 and 
https://github.com/datafusion-contrib/datafusion-variant/pull/196
   - https://github.com/datafusion-contrib/datafusion-variant/issues/199
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to