MaxGekk commented on a change in pull request #30056:
URL: https://github.com/apache/spark/pull/30056#discussion_r507669992
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -2640,6 +2640,20 @@ object SQLConf {
.checkValues(LegacyBehaviorPolicy.values.map(_.toString))
.createWithDefault(LegacyBehaviorPolicy.EXCEPTION.toString)
+ val LEGACY_PARQUET_INT96_REBASE_MODE_IN_WRITE =
+ buildConf("spark.sql.legacy.parquet.int96RebaseModeInWrite")
+ .internal()
+ .doc("When LEGACY, which is the default, Spark will rebase INT96
timestamps from " +
+ "Proleptic Gregorian calendar to the legacy hybrid (Julian +
Gregorian) calendar when " +
+ "writing Parquet files. When CORRECTED, Spark will not do rebase and
write the timestamps" +
+ " as it is. When EXCEPTION, Spark will fail the writing if it sees
ancient timestamps " +
+ "that are ambiguous between the two calendars.")
+ .version("3.1.0")
+ .stringConf
+ .transform(_.toUpperCase(Locale.ROOT))
+ .checkValues(LegacyBehaviorPolicy.values.map(_.toString))
+ .createWithDefault(LegacyBehaviorPolicy.LEGACY.toString)
Review comment:
I am not sure that we can do that in the minor release 3.1. This can
break existing apps. @cloud-fan @HyukjinKwon WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
