This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push: new 8743571 [SPARK-33571][SQL][DOCS][3.0] Add a ref to INT96 config from the doc for `spark.sql.legacy.parquet.datetimeRebaseModeInWrite/Read` 8743571 is described below commit 874357174d8a2c9ef207a4dcd0ca819e658b2881 Author: Max Gekk <max.g...@gmail.com> AuthorDate: Fri Dec 4 15:10:51 2020 -0800 [SPARK-33571][SQL][DOCS][3.0] Add a ref to INT96 config from the doc for `spark.sql.legacy.parquet.datetimeRebaseModeInWrite/Read` ### What changes were proposed in this pull request? For the SQL configs `spark.sql.legacy.parquet.datetimeRebaseModeInWrite` and `spark.sql.legacy.parquet.datetimeRebaseModeInRead`, improve their descriptions by: 1. Explicitly document on which parquet types, those configs influence on 2. Refer to corresponding configs for `INT96` ### Why are the changes needed? To avoid user confusions like reposted in SPARK-33571, and make the config description more precise. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? By running `./dev/scalastyle`. Closes #30604 from MaxGekk/clarify-rebase-docs-3.0. Authored-by: Max Gekk <max.g...@gmail.com> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index 23d1d70..0cf980c 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -2537,7 +2537,9 @@ object SQLConf { "to the legacy hybrid (Julian + Gregorian) calendar when writing Parquet files. " + "When CORRECTED, Spark will not do rebase and write the dates/timestamps as it is. " + "When EXCEPTION, which is the default, Spark will fail the writing if it sees " + - "ancient dates/timestamps that are ambiguous between the two calendars.") + "ancient dates/timestamps that are ambiguous between the two calendars. " + + "This config influences on writes of the following parquet logical types: DATE, " + + "TIMESTAMP_MILLIS, TIMESTAMP_MICROS. Spark always performs rebasing for the INT96 type.") .version("3.0.0") .stringConf .transform(_.toUpperCase(Locale.ROOT)) @@ -2552,7 +2554,9 @@ object SQLConf { "When CORRECTED, Spark will not do rebase and read the dates/timestamps as it is. " + "When EXCEPTION, which is the default, Spark will fail the reading if it sees " + "ancient dates/timestamps that are ambiguous between the two calendars. This config is " + - "only effective if the writer info (like Spark, Hive) of the Parquet files is unknown.") + "only effective if the writer info (like Spark, Hive) of the Parquet files is unknown. " + + "This config influences on reads of the following parquet logical types: DATE, " + + "TIMESTAMP_MILLIS, TIMESTAMP_MICROS. Spark always performs rebasing for the INT96 type.") .version("3.0.0") .stringConf .transform(_.toUpperCase(Locale.ROOT)) --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org