viirya opened a new pull request, #7692:
URL: https://github.com/apache/datafusion-comet/pull/7692
## Which issue does this PR close?
Part of #6499 (step 2). Builds on #6883.
## Rationale for this change
#6883 runs a query as one local DataFusion graph only when its leaf is a
range. Step 2 of #6499 admits exchange-free plans over Comet's native Parquet
scan (scan, filter, project), judged from the plan `CometExecRule` produces,
whether or not AQE is enabled.
## What changes are included in this PR?
Admission (`CometLocalRule`)
- The block's single leaf may now be a `CometNativeScanExec`. The scan is
declined when it reports partitioning (a bucketed or key-grouped scan) or reads
encrypted files, since decryption needs the JVM.
- A plan whose expressions contain a physical subquery (`PlanExpression`,
for example a scalar subquery or dynamic pruning in a scan's filters) is
declined. These are not in the serialized plan, so the existing JVM-callback
check on the plan does not see them.
- A scan's file partitions are listed only when the query runs, as in
ordinary execution. So `CometLocalExec` builds its spec at execution time,
after resolving the scan's subqueries. When there are no partitions, or more
than 1024, `collect` runs as ordinary Comet execution.
- The result node copies the block's logical link. Without it,
`spark.sql.adaptive.forceApply` failed with an internal error when AQE made the
result its final stage. This also affected range plans from #6883.
Native
- `createQuery` takes each partition's serialized `NativeScan` with its
files, which is the same data a Spark task receives. A partition's files are
injected into a copy of the plan when that partition is planned. The ordinary
planner and the protobuf are unchanged.
- `native/local` now plans and starts each partition on the runtime's
blocking pool. This is needed because creating an S3 object store blocks on
Comet's runtime, which panics on a worker. The started stream is stored from
the blocking thread, so cancellation still drops it synchronously.
- The query records which partition's turn it was when it failed. The JVM
uses this to name that partition's files when a native read error carries no
file path, as ordinary execution names the failing task's files.
The contributor guide page `local-execution.md` is updated. One known gap is
documented there: a local scan reports no task input metrics.
## How are these changes tested?
- `native/local`: new tests for a partition start that blocks on the
runtime, a start that panics, and the partition reported for an error.
- `native/core` (`local::`): each partition reads only its own Parquet
files, an empty file group yields an empty partition, and file lists that do
not match the plan are rejected.
- `CometLocalExecutionSuite`: Parquet scans split into many partitions are
compared with Spark's results and row order. Further tests cover:
- partition pruning;
- scans of no files or of more than 1024 partitions;
- declined bucketed, encrypted and subquery plans;
- AQE on, off and `forceApply` (range and scan);
- cached and RDD partition counts;
- a corrupt file named in the error;
- explain before files are listed.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]