andygrove opened a new pull request, #3699: URL: https://github.com/apache/datafusion-comet/pull/3699
## Which issue does this PR close? Closes #3314. ## Rationale for this change When `native_datafusion` scan encounters missing files with `ignoreMissingFiles=false`, it throws a `CometNativeException` with an error message like "Object at location ... not found". `CometExecIterator` pattern-matches this message and wraps it in a `SparkException` with a plain `FileNotFoundException` cause. However, Spark SQL tests expect the cause to be `SparkFileNotFoundException` (which is `private[spark]`), so the `.asInstanceOf[SparkFileNotFoundException]` cast fails. ## What changes are included in this PR? - **Rust**: Add `FileNotFound` variant to `SparkError` enum. Detect file-not-found errors in `throw_exception()` (both the `DataFusionError::External` branch and the generic catch-all) and convert them to `SparkError::FileNotFound`, which is serialized as JSON and thrown as `CometQueryExecutionException`. - **Shim layer**: Add `"FileNotFound"` case to all three `ShimSparkErrorConverter` implementations (Spark 3.4, 3.5, 4.0) that calls `QueryExecutionErrors.readCurrentFileNotFoundError()`, producing a proper `SparkFileNotFoundException`. - **CometExecIterator**: Remove the old `fileNotFoundPattern` regex matching from the `CometNativeException` handler since file-not-found errors are now handled through the JSON/shim path. ## How are these changes tested? The fix enables the existing Spark SQL tests (`FileBasedDataSourceSuite` and `SimpleSQLViewSuite`) that were previously skipped for `native_datafusion` to pass. The diff workarounds (`assume()` and `IgnoreCometNativeDataFusion` tags) can be removed in a follow-up once these tests are confirmed passing in CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
