akshaytayal opened a new issue, #13433: URL: https://github.com/apache/gluten/issues/13433
Part of #13181 (sub-theme #13180). Found while re-enabling the Spark 4.2 UTs disabled in #13163. ### Problem The Velox native scan honors only the session conf `spark.sql.files.ignoreMissingFiles`. The per-read data source option `ignoreMissingFiles` (Spark >= 3.4) never reaches native code, so: | SQLConf | read option | Spark | Gluten (native) | |---|---|---|---| | true | (unset) | skips missing file | skips (OK) | | false | `true` | skips missing file | **fails** (Velox `FILE_NOT_FOUND`) | | false | (unset) | `FAILED_READ_FILE.FILE_NOT_EXIST` | fails with wrapped Velox error (different error class) | | true | `false` | `FAILED_READ_FILE.FILE_NOT_EXIST` | **silently skips the missing file** (wrong result) | Disabled tests (`VeloxTestSettings.scala`, spark42): `SPARK-55857: Enabling/disabling ignoreMissingFiles` in `GlutenOrcV1QuerySuite`, `GlutenOrcV2QuerySuite`, `GlutenParquetV1QuerySuite`, `GlutenParquetV2QuerySuite`. ### Root cause - Gluten sends only the session conf: `GlutenConfig.scala` adds `spark.sql.files.ignoreMissingFiles` to the native session conf; `cpp/velox/utils/ConfigExtractor.cc` maps it to `HiveConfig::kIgnoreMissingFilesSession`. - Velox reads it only from session properties (`velox/connectors/hive/FileSplitReader.cpp`, `fileConfig_->ignoreMissingFiles(connectorQueryCtx_->sessionProperties())`), so it can't differ per scan. - Spark resolves it per scan via `FileSourceOptions.ignoreMissingFiles` (option, falling back to SQLConf) in `FileScanRDD` (V1) and `FilePartitionReader` (V2). ### Proposed fix 1. Velox: let a per-table parameter override the session value in `FileSplitReader` (e.g. `tableHandle_->tableParameters()["ignore_missing_files"]`, else the session property). 2. Gluten: pass `new FileSourceOptions(options).ignoreMissingFiles` from `FileSourceScanExecTransformer` / `BatchScanExecTransformer` and set the table parameter in `SubstraitToVeloxPlan.cc` (`HiveTableHandle` construction). 3. Map the native missing-file error to Spark's `FAILED_READ_FILE.FILE_NOT_EXIST` (or add a Gluten test variant for that assertion only), then remove the 4 excludes. Interim option (narrow, Gluten-only): fall back to vanilla scan in `BasicScanExecTransformer` validation when the read option differs from the session conf. This fixes the silent-skip correctness issue without a Velox change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
