viirya opened a new pull request, #7692:
URL: https://github.com/apache/datafusion-comet/pull/7692

   ## Which issue does this PR close?
   
   Part of #6499 (step 2). Builds on #6883.
   
   ## Rationale for this change
   
   #6883 runs a query as one local DataFusion graph only when its leaf is a 
range. Step 2 of #6499 admits exchange-free plans over Comet's native Parquet 
scan (scan, filter, project), judged from the plan `CometExecRule` produces, 
whether or not AQE is enabled.
   
   ## What changes are included in this PR?
   
   Admission (`CometLocalRule`)
   - The block's single leaf may now be a `CometNativeScanExec`. The scan is 
declined when it reports partitioning (a bucketed or key-grouped scan) or reads 
encrypted files, since decryption needs the JVM.
   - A plan whose expressions contain a physical subquery (`PlanExpression`, 
for example a scalar subquery or dynamic pruning in a scan's filters) is 
declined. These are not in the serialized plan, so the existing JVM-callback 
check on the plan does not see them.
   - A scan's file partitions are listed only when the query runs, as in 
ordinary execution. So `CometLocalExec` builds its spec at execution time, 
after resolving the scan's subqueries. When there are no partitions, or more 
than 1024, `collect` runs as ordinary Comet execution.
   - The result node copies the block's logical link. Without it, 
`spark.sql.adaptive.forceApply` failed with an internal error when AQE made the 
result its final stage. This also affected range plans from #6883.
   
   Native
   - `createQuery` takes each partition's serialized `NativeScan` with its 
files, which is the same data a Spark task receives. A partition's files are 
injected into a copy of the plan when that partition is planned. The ordinary 
planner and the protobuf are unchanged.
   - `native/local` now plans and starts each partition on the runtime's 
blocking pool. This is needed because creating an S3 object store blocks on 
Comet's runtime, which panics on a worker. The started stream is stored from 
the blocking thread, so cancellation still drops it synchronously.
   - The query records which partition's turn it was when it failed. The JVM 
uses this to name that partition's files when a native read error carries no 
file path, as ordinary execution names the failing task's files.
   
   The contributor guide page `local-execution.md` is updated. One known gap is 
documented there: a local scan reports no task input metrics.
   
   ## How are these changes tested?
   
   - `native/local`: new tests for a partition start that blocks on the 
runtime, a start that panics, and the partition reported for an error.
   - `native/core` (`local::`): each partition reads only its own Parquet 
files, an empty file group yields an empty partition, and file lists that do 
not match the plan are rejected.
   - `CometLocalExecutionSuite`: Parquet scans split into many partitions are 
compared with Spark's results and row order. Further tests cover:
     - partition pruning;
     - scans of no files or of more than 1024 partitions;
     - declined bucketed, encrypted and subquery plans;
     - AQE on, off and `forceApply` (range and scan);
     - cached and RDD partition counts;
     - a corrupt file named in the error;
     - explain before files are listed.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to