Linhong Liu created SPARK-61585:
-----------------------------------

             Summary: spark.sql.files.ignoreCorruptFiles swallows a 
SparkUpgradeException and skips the rest of the file instead of raising it
                 Key: SPARK-61585
                 URL: https://issues.apache.org/jira/browse/SPARK-61585
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Linhong Liu


DataSourceUtils.shouldIgnoreCorruptFileException returns true for every 
RuntimeException and IOException, which includes SparkUpgradeException (e.g. 
INCONSISTENT_BEHAVIOR_CROSS_VERSION from a datetime pattern or value only the 
legacy parser accepts). So with spark.sql.files.ignoreCorruptFiles=true, the 
file is truncated at the first row that raises the upgrade error and the query 
silently returns partial data, even though the parsers rethrow the upgrade 
error on purpose rather than turning it into a bad record.

Fix: do not treat a SparkUpgradeException (or a ParquetDecodingException 
wrapping one) as file corruption; raise it. A new conf 
spark.sql.legacy.files.ignoreCorruptFilesSkipsUpgradeErrors (default false) 
restores the previous behavior. Found via correctness analysis.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to