[ https://issues.apache.org/jira/browse/SPARK-31404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607326#comment-17607326 ]
Sachit commented on SPARK-31404: -------------------------------- Hi [~cloud_fan] , Could you please confirm if we need to use below properties to ensure it can read data written by spark2.4x spark.conf.set("spark.sql.parquet.int96RebaseModeInRead","CORRECTED") spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite","CORRECTED") Regards Sachit > file source backward compatibility after calendar switch > -------------------------------------------------------- > > Key: SPARK-31404 > URL: https://issues.apache.org/jira/browse/SPARK-31404 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.0.0 > Reporter: Wenchen Fan > Priority: Blocker > Fix For: 3.0.0 > > Attachments: Switch to Java 8 time API in Spark 3.0.pdf > > > In Spark 3.0, we switch to the Proleptic Gregorian calendar by using the Java > 8 datetime APIs. This makes Spark follow the ISO and SQL standard, but > introduces some backward compatibility problems: > 1. may read wrong data from the data files written by Spark 2.4 > 2. may have perf regression -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org