rdtr commented on PR #13207:
URL: https://github.com/apache/gluten/pull/13207#issuecomment-6689656954
@philo-he could you take a look when you have time? This falls back the scan
when the `ignoreMissingFiles` read option differs from
`spark.sql.files.ignoreMissingFiles`. Today Gluten ignores the read option.
With the conf set to `true`, `spark.read.option("ignoreMissingFiles", "false")`
returns fewer rows instead of failing like Spark.
facebookincubator/velox#19840 adds a table parameter that overrides
`ignore_missing_files`, and that is the better fix in the long run. Could this
go in as an interim fix until it lands? Once it does, I'll replace the fallback
with passing the read option to Velox (#13433). I rebased this onto main. The
only CI failure is `test that introducing TTL after restart fails query` in
`spark-test-spark42 (1)`, the flaky test tracked in #13463.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
