[
https://issues.apache.org/jira/browse/SPARK-49617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882529#comment-17882529
]
Xiao Li commented on SPARK-49617:
---------------------------------
[~yhosny] when did we introduce this behavior change?
> Stop ignoring auth errors when ignoreCorruptFiles=1 in FileScanRDD
> ------------------------------------------------------------------
>
> Key: SPARK-49617
> URL: https://issues.apache.org/jira/browse/SPARK-49617
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 4.0.0
> Reporter: Yousof Hosny
> Priority: Major
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> We currently lose data when the ignoreCorruptFiles flag is set to true
> because file Authorization Errors are handled as if they were corrupt files.
> We ignore these files despite them having no corrupt records. We’d like to
> fix this bug by failing the stream when authorization errors are found.
> To prevent regressions we will add a SQLConf flag that can be temporarily set
> to revert to previous behavior. The idea is to deprecate the flag at a later
> major version release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
