Re: [PR] Add dynamic filter (bounds) pushdown to HashJoinExec [datafusion]

[via GitHub]

adriangb commented on PR #16445:
URL: https://github.com/apache/datafusion/pull/16445#issuecomment-3141364005

   Here it is in action:
   
   ```sql
   COPY (
       with data as (
           select unnest(generate_series(1, 99999999)) as id
       )
       select 
           id,
           (id / 100000) as partition_id
       from data
   ) TO 'test/t1/' STORED AS parquet PARTITIONED BY (partition_id);
   
   CREATE EXTERNAL TABLE t1 (
       id int
   )
   STORED AS PARQUET
   PARTITIONED BY (partition_id int)
   LOCATION 'test/t1/';
   
   COPY (
       with data as (
           select unnest(generate_series(1, 100)) as id
       )
       select 
           id,
           (id / 100000) as partition_id
       from data
   ) TO 'test/t2/' STORED AS parquet PARTITIONED BY (partition_id);
   
   CREATE EXTERNAL TABLE t2 (
       id int
   )
   STORED AS PARQUET
   PARTITIONED BY (partition_id int)
   LOCATION 'test/t2/';
   
   SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
   
   explain analyze
   SELECT count(*)
   FROM t1
   JOIN t2 USING (id);
   
   SET datafusion.optimizer.enable_dynamic_filter_pushdown = true;
   
   explain analyze
   SELECT count(*)
   FROM t1
   JOIN t2 USING (id);
   ```
   
   
   | dynamic filters | time (ms) | bytes scanned |
   |-----------------|-----------|---------------|
   | on              |        55 |       376,155 |
   | off             | 401       | 296,192,099   |
   
   
   
   <details>
     <summary>Explain plans</summary>
   
   
   ```
   ❯ ./target/release/datafusion-cli
   DataFusion CLI v49.0.0
   > COPY (
       with data as (
           select unnest(generate_series(1, 99999999)) as id
       )
       select 
           id,
           (id / 100000) as partition_id
       from data
   ) TO 'test/t1/' STORED AS parquet PARTITIONED BY (partition_id);
   
   
   CREATE EXTERNAL TABLE t1 (
       id int
   )
   STORED AS PARQUET
   PARTITIONED BY (partition_id int)
   LOCATION 'test/t1/';
   
   
   
   COPY (
       with data as (
           select unnest(generate_series(1, 100)) as id
       )
       select 
           id,
           (id / 100000) as partition_id
       from data
   ) TO 'test/t2/' STORED AS parquet PARTITIONED BY (partition_id);
   
   
   CREATE EXTERNAL TABLE t2 (
       id int
   )
   STORED AS PARQUET
   PARTITIONED BY (partition_id int)
   LOCATION 'test/t2/';
   +----------+
   | count    |
   +----------+
   | 99999999 |
   +----------+
   1 row(s) fetched. 
   Elapsed 16.986 seconds.
   
   0 row(s) fetched. 
   Elapsed 0.015 seconds.
   
   +-------+
   | count |
   +-------+
   | 100   |
   +-------+
   1 row(s) fetched. 
   Elapsed 0.001 seconds.
   
   0 row(s) fetched. 
   Elapsed 0.000 seconds.
   
   > SET datafusion.optimizer.enable_dynamic_filter_pushdown = false;
   
   0 row(s) fetched. 
   Elapsed 0.000 seconds.
   
   > explain analyze
   SELECT count(*)
   FROM t1
   JOIN t2 USING (id);
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | plan_type         | plan                                                   
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                          |
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | Plan with Metrics | ProjectionExec: expr=[count(Int64(1))@0 as count(*)], 
metrics=[output_rows=1, elapsed_compute=417ns]                                  
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                          |
   |                   |   AggregateExec: mode=Final, gby=[], 
aggr=[count(Int64(1))], metrics=[output_rows=1, elapsed_compute=18.874µs]       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                           |
   |                   |     CoalescePartitionsExec, metrics=[output_rows=12, 
elapsed_compute=7.959µs]                                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                           |
   |                   |       AggregateExec: mode=Partial, gby=[], 
aggr=[count(Int64(1))], metrics=[output_rows=12, elapsed_compute=14.54µs]       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                 
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                           |
   |                   |         ProjectionExec: expr=[], 
metrics=[output_rows=100, elapsed_compute=219ns]                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                          |
   |                   |           CoalesceBatchesExec: target_batch_size=8192, 
metrics=[output_rows=100, elapsed_compute=1.570227ms]                           
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                          |
   |                   |             HashJoinExec: mode=CollectLeft, 
join_type=Inner, on=[(id@0, id@0)], metrics=[output_rows=100, 
elapsed_compute=694.331448ms, build_input_batches=1, build_input_rows=100, 
input_batches=13000, input_rows=99999999, output_batches=13000, 
build_mem_used=2680, build_time=1.906461ms, join_time=692.419193ms]             
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                          |
   |                   |               DataSourceExec: file_groups={1 group: 
[[Users/adriangb/GitHub/datafusion/test/t2/partition_id=0/0lwnifc1mVkAu3uv.parquet]]},
 projection=[id], file_type=parquet, metrics=[output_rows=100, 
elapsed_compute=1ns, batches_splitted=0, bytes_scanned=318, file_open_errors=0, 
file_scan_errors=0, files_ranges_pruned_statistics=0, 
num_predicate_creation_errors=0, page_index_rows_matched=0, 
page_index_rows_pruned=0, predicate_evaluation_errors=0, 
pushdown_rows_matched=0, pushdown_rows_pruned=0, 
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, 
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, 
bloom_filter_eval_time=2ns, metadata_load_time=102.71µs, 
page_index_eval_time=2ns, row_pushdown_eval_time=2ns, statistics_eval_time=2ns, 
time_elapsed_opening=1.735375ms, time_elapsed_processing=1.865291ms, 
time_elapsed_scanning_total=224.5µs, 
time_elapsed_scanning_until_data=217.333µs]                                     
            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                             |
   |                   |               DataSourceExec: file_groups={12 groups: 
[[Users/adriangb/GitHub/datafusion/test/t1/partition_id=0/pE27ne7XoO3Ozi2m.parquet:0..332561,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=1/pE27ne7XoO3Ozi2m.parquet:0..296777,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=10/pE27ne7XoO3Ozi2m.parquet:0..296764,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=100/pE27ne7XoO3Ozi2m.parquet:0..296777,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=101/pE27ne7XoO3Ozi2m.parquet:0..296764,
 ...], 
[Users/adriangb/GitHub/datafusion/test/t1/partition_id=173/pE27ne7XoO3Ozi2m.parquet:65945..296772,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=174/pE27ne7XoO3Ozi2m.parquet:0..296774,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=175/pE27ne7XoO3Ozi2m.parquet:0..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=176/pE27ne7XoO3Ozi2m.parquet:0..296771,
 Users/adriangb/GitHub/datafusion/test/t1/partition_id=177/pE27ne7
 XoO3Ozi2m.parquet:0..296764, ...], 
[Users/adriangb/GitHub/datafusion/test/t1/partition_id=248/pE27ne7XoO3Ozi2m.parquet:167870..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=249/pE27ne7XoO3Ozi2m.parquet:0..296774,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=25/pE27ne7XoO3Ozi2m.parquet:0..296767,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=250/pE27ne7XoO3Ozi2m.parquet:0..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=251/pE27ne7XoO3Ozi2m.parquet:0..296775,
 ...], 
[Users/adriangb/GitHub/datafusion/test/t1/partition_id=322/pE27ne7XoO3Ozi2m.parquet:269778..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=323/pE27ne7XoO3Ozi2m.parquet:0..296775,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=324/pE27ne7XoO3Ozi2m.parquet:0..296764,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=325/pE27ne7XoO3Ozi2m.parquet:0..296771,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=326/pE27ne7XoO3Ozi2m.parquet:0..296764,
 ...],
  
[Users/adriangb/GitHub/datafusion/test/t1/partition_id=399/pE27ne7XoO3Ozi2m.parquet:74933..296772,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=4/pE27ne7XoO3Ozi2m.parquet:0..296767,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=40/pE27ne7XoO3Ozi2m.parquet:0..296764,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=400/pE27ne7XoO3Ozi2m.parquet:0..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=401/pE27ne7XoO3Ozi2m.parquet:0..296775,
 ...], ...]}, projection=[id], file_type=parquet, 
metrics=[output_rows=99999999, elapsed_compute=12ns, batches_splitted=0, 
bytes_scanned=296192099, file_open_errors=0, file_scan_errors=0, 
files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, 
page_index_rows_matched=0, page_index_rows_pruned=0, 
predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, 
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, 
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, blo
 om_filter_eval_time=2.022µs, metadata_load_time=597.091275ms, 
page_index_eval_time=2.022µs, row_pushdown_eval_time=2.022µs, 
statistics_eval_time=2.022µs, time_elapsed_opening=9.99707ms, 
time_elapsed_processing=1.943677056s, time_elapsed_scanning_total=3.949963979s, 
time_elapsed_scanning_until_data=2.625103918s] |
   |                   |                                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                          |
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   1 row(s) fetched. 
   Elapsed 0.401 seconds.
   
   > SET datafusion.optimizer.enable_dynamic_filter_pushdown = true;
   
   0 row(s) fetched. 
   Elapsed 0.000 seconds.
   
   > explain analyze
   SELECT count(*)
   FROM t1
   JOIN t2 USING (id);
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | plan_type         | plan                                                   
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                |
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   | Plan with Metrics | ProjectionExec: expr=[count(Int64(1))@0 as count(*)], 
metrics=[output_rows=1, elapsed_compute=333ns]                                  
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                |
   |                   |   AggregateExec: mode=Final, gby=[], 
aggr=[count(Int64(1))], metrics=[output_rows=1, elapsed_compute=4.372µs]        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                 |
   |                   |     CoalescePartitionsExec, metrics=[output_rows=12, 
elapsed_compute=3.166µs]                                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                 |
   |                   |       AggregateExec: mode=Partial, gby=[], 
aggr=[count(Int64(1))], metrics=[output_rows=12, elapsed_compute=7.5µs]         
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                 
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                 |
   |                   |         ProjectionExec: expr=[], 
metrics=[output_rows=100, elapsed_compute=136ns]                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                |
   |                   |           CoalesceBatchesExec: target_batch_size=8192, 
metrics=[output_rows=100, elapsed_compute=2.546µs]                              
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                     
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                 |
   |                   |             HashJoinExec: mode=CollectLeft, 
join_type=Inner, on=[(id@0, id@0)], filter=[id@0 >= 1 AND id@0 <= 100], 
metrics=[output_rows=100, elapsed_compute=811.172µs, build_input_batches=1, 
build_input_rows=100, input_batches=3, input_rows=20480, output_batches=3, 
build_mem_used=2680, build_time=607.5µs, join_time=131.88µs]                    
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                               
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                   |
   |                   |               DataSourceExec: file_groups={1 group: 
[[Users/adriangb/GitHub/datafusion/test/t2/partition_id=0/0lwnifc1mVkAu3uv.parquet]]},
 projection=[id], file_type=parquet, predicate=true, metrics=[output_rows=100, 
elapsed_compute=1ns, batches_splitted=0, bytes_scanned=318, file_open_errors=0, 
file_scan_errors=0, files_ranges_pruned_statistics=0, 
num_predicate_creation_errors=0, page_index_rows_matched=0, 
page_index_rows_pruned=0, predicate_evaluation_errors=0, 
pushdown_rows_matched=0, pushdown_rows_pruned=0, 
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, 
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, 
bloom_filter_eval_time=2ns, metadata_load_time=797.418µs, 
page_index_eval_time=126ns, row_pushdown_eval_time=2ns, 
statistics_eval_time=2ns, time_elapsed_opening=824.667µs, 
time_elapsed_processing=245.335µs, time_elapsed_scanning_total=196.501µs, 
time_elapsed_scanning_until_data=193.583µs]                            
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                     |
   |                   |               DataSourceExec: file_groups={12 groups: 
[[Users/adriangb/GitHub/datafusion/test/t1/partition_id=0/pE27ne7XoO3Ozi2m.parquet:0..332561,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=1/pE27ne7XoO3Ozi2m.parquet:0..296777,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=10/pE27ne7XoO3Ozi2m.parquet:0..296764,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=100/pE27ne7XoO3Ozi2m.parquet:0..296777,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=101/pE27ne7XoO3Ozi2m.parquet:0..296764,
 ...], 
[Users/adriangb/GitHub/datafusion/test/t1/partition_id=173/pE27ne7XoO3Ozi2m.parquet:65945..296772,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=174/pE27ne7XoO3Ozi2m.parquet:0..296774,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=175/pE27ne7XoO3Ozi2m.parquet:0..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=176/pE27ne7XoO3Ozi2m.parquet:0..296771,
 Users/adriangb/GitHub/datafusion/test/t1/partition_id=177/pE27ne7
 XoO3Ozi2m.parquet:0..296764, ...], 
[Users/adriangb/GitHub/datafusion/test/t1/partition_id=248/pE27ne7XoO3Ozi2m.parquet:167870..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=249/pE27ne7XoO3Ozi2m.parquet:0..296774,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=25/pE27ne7XoO3Ozi2m.parquet:0..296767,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=250/pE27ne7XoO3Ozi2m.parquet:0..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=251/pE27ne7XoO3Ozi2m.parquet:0..296775,
 ...], 
[Users/adriangb/GitHub/datafusion/test/t1/partition_id=322/pE27ne7XoO3Ozi2m.parquet:269778..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=323/pE27ne7XoO3Ozi2m.parquet:0..296775,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=324/pE27ne7XoO3Ozi2m.parquet:0..296764,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=325/pE27ne7XoO3Ozi2m.parquet:0..296771,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=326/pE27ne7XoO3Ozi2m.parquet:0..296764,
 ...],
  
[Users/adriangb/GitHub/datafusion/test/t1/partition_id=399/pE27ne7XoO3Ozi2m.parquet:74933..296772,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=4/pE27ne7XoO3Ozi2m.parquet:0..296767,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=40/pE27ne7XoO3Ozi2m.parquet:0..296764,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=400/pE27ne7XoO3Ozi2m.parquet:0..296761,
 
Users/adriangb/GitHub/datafusion/test/t1/partition_id=401/pE27ne7XoO3Ozi2m.parquet:0..296775,
 ...], ...]}, projection=[id], file_type=parquet, 
predicate=DynamicFilterPhysicalExpr [ id@0 >= 1 AND id@0 <= 100 ], 
pruning_predicate=id_null_count@1 != row_count@2 AND id_max@0 >= 1 AND 
id_null_count@1 != row_count@2 AND id_min@3 <= 100, required_guarantees=[] |
   |                   | , metrics=[output_rows=20480, elapsed_compute=12ns, 
batches_splitted=0, bytes_scanned=376155, file_open_errors=0, 
file_scan_errors=0, files_ranges_pruned_statistics=0, 
num_predicate_creation_errors=0, page_index_rows_matched=20480, 
page_index_rows_pruned=79519, predicate_evaluation_errors=0, 
pushdown_rows_matched=0, pushdown_rows_pruned=0, 
row_groups_matched_bloom_filter=0, row_groups_matched_statistics=1, 
row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=999, 
bloom_filter_eval_time=37.804µs, metadata_load_time=298.876746ms, 
page_index_eval_time=132.682µs, row_pushdown_eval_time=2.022µs, 
statistics_eval_time=4.021308ms, time_elapsed_opening=296.060463ms, 
time_elapsed_processing=118.217967ms, time_elapsed_scanning_total=27.708559ms, 
time_elapsed_scanning_until_data=27.457468ms]                                   
                                                                                
                                                       
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                   |
   |                   |                                                        
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                      
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                                                                                
                |
   
+-------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
   1 row(s) fetched. 
   Elapsed 0.055 seconds.
   ```
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org