Neuw84 commented on issue #6264: URL: https://github.com/apache/datafusion-comet/issues/6264#issuecomment-5884746709
@dwsmith1983 I don't think it's the same. SPARK-54554 is about Spark's optimizer never inserting the DPP filter, because the pruning side comes from SHOW PARTITIONS and no plan gets one. Here Spark does insert it. With Spark alone the same plan has dynamicpruningexpression(ss_sold_date_sk IN dynamicpruning#…) backed by a SubqueryBroadcast, and it prunes to 1.66 M rows at SF1. The filter only disappears once the scan becomes CometNativeScanExec, and the other DPP scans in the same query keep it. So it happens after DPP planning, in Comet's scan conversion (or its AQE handling), not in Spark's DPP rule. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
