This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 148950f [SPARK-31359][DOC][FOLLOWUP] improve code comments in RebaseDateTime 148950f is described below commit 148950fa2ba7ad03d7536c8c2201ba18b9345c0e Author: Wenchen Fan <wenc...@databricks.com> AuthorDate: Fri Apr 10 03:43:32 2020 +0000 [SPARK-31359][DOC][FOLLOWUP] improve code comments in RebaseDateTime ### What changes were proposed in this pull request? improve the code comment and make them consistent between `rebaseJulianToGregorian*` and `rebaseGregorianToJulian*` ### Why are the changes needed? improve readability. ### Does this PR introduce any user-facing change? no ### How was this patch tested? N/A Closes #28166 from cloud-fan/comment. Authored-by: Wenchen Fan <wenc...@databricks.com> Signed-off-by: Wenchen Fan <wenc...@databricks.com> --- .../spark/sql/catalyst/util/RebaseDateTime.scala | 176 ++++++++++++--------- 1 file changed, 98 insertions(+), 78 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala index 5efde37..766eee9 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/RebaseDateTime.scala @@ -32,20 +32,20 @@ import org.apache.spark.sql.catalyst.util.DateTimeUtils._ /** * The collection of functions for rebasing days and microseconds from/to the hybrid calendar * (Julian + Gregorian since 1582-10-15) which is used by Spark 2.4 and earlier versions - * to/from Proleptic Gregorian calendar used in Spark SQL since version 3.0, see SPARK-26651. + * to/from Proleptic Gregorian calendar which is used by Spark since version 3.0. See SPARK-26651. */ object RebaseDateTime { /** * Rebases days since the epoch from an original to an target calendar, for instance, * from a hybrid (Julian + Gregorian) to Proleptic Gregorian calendar. * - * It finds the latest switch day which is less than `value`, and adds the difference - * in days associated with the switch days to the given `value`. + * It finds the latest switch day which is less than the given `days`, and adds the difference + * in days associated with the switch days to the given `days`. * The function is based on linear search which starts from the most recent switch days. * This allows to perform less comparisons for modern dates. * - * @param switches The days when difference in days between original - * and target calendar was changed. + * @param switches The days when difference in days between original and target calendar + * was changed. * @param diffs The differences in days between calendars. * @param days The number of days since the epoch 1970-01-01 to be rebased * to the target calendar. @@ -71,12 +71,20 @@ object RebaseDateTime { -390745, -354220, -317695, -244645, -208120, -171595, -141427) /** - * Converts the given number of days since the epoch day 1970-01-01 to - * a local date in Julian calendar, interprets the result as a local - * date in Proleptic Gregorian calendar, and take the number of days - * since the epoch from the Gregorian date. + * Converts the given number of days since the epoch day 1970-01-01 to a local date in Julian + * calendar, interprets the result as a local date in Proleptic Gregorian calendar, and takes the + * number of days since the epoch from the Gregorian local date. * - * @param days The number of days since the epoch in Julian calendar. + * This is used to guarantee backward compatibility, as Spark 2.4 and earlier versions use + * Julian calendar for dates before 1582-10-15, while Spark 3.0 and later use Proleptic Gregorian + * calendar. See SPARK-26651. + * + * For example: + * Julian calendar: 1582-01-01 -> -141704 + * Proleptic Gregorian calendar: 1582-01-01 -> -141714 + * The code below converts -141704 to -141714. + * + * @param days The number of days since the epoch in Julian calendar. It can be negative. * @return The rebased number of days in Gregorian calendar. */ def rebaseJulianToGregorianDays(days: Int): Int = { @@ -97,51 +105,26 @@ object RebaseDateTime { -390750, -354226, -317702, -244653, -208129, -171605, -141427) /** - * Rebasing days since the epoch to store the same number of days - * as by Spark 2.4 and earlier versions. Spark 3.0 switched to - * Proleptic Gregorian calendar (see SPARK-26651), and as a consequence of that, - * this affects dates before 1582-10-15. Spark 2.4 and earlier versions use - * Julian calendar for dates before 1582-10-15. So, the same local date may - * be mapped to different number of days since the epoch in different calendars. + * Converts the given number of days since the epoch day 1970-01-01 to a local date in Proleptic + * Gregorian calendar, interprets the result as a local date in Julian calendar, and takes the + * number of days since the epoch from the Julian local date. + * + * This is used to guarantee backward compatibility, as Spark 2.4 and earlier versions use + * Julian calendar for dates before 1582-10-15, while Spark 3.0 and later use Proleptic Gregorian + * calendar. See SPARK-26651. * * For example: * Proleptic Gregorian calendar: 1582-01-01 -> -141714 * Julian calendar: 1582-01-01 -> -141704 * The code below converts -141714 to -141704. * - * @param days The number of days since the epoch 1970-01-01. It can be negative. + * @param days The number of days since the epoch in Gregorian calendar. It can be negative. * @return The rebased number of days since the epoch in Julian calendar. */ def rebaseGregorianToJulianDays(days: Int): Int = { rebaseDays(gregJulianDiffSwitchDay, gregJulianDiffs, days) } - /** - * Converts the given microseconds to a local date-time in UTC time zone in Proleptic Gregorian - * calendar, interprets the result as a local date-time in Julian calendar in UTC time zone. - * And takes microseconds since the epoch from the Julian timestamp. - * - * @param zoneId The time zone ID at which the rebasing should be performed. - * @param micros The number of microseconds since the epoch '1970-01-01T00:00:00Z' - * in Proleptic Gregorian calendar. It can be negative. - * @return The rebased microseconds since the epoch in Julian calendar. - */ - private[sql] def rebaseGregorianToJulianMicros(zoneId: ZoneId, micros: Long): Long = { - val instant = microsToInstant(micros) - val ldt = instant.atZone(zoneId).toLocalDateTime - val cal = new Calendar.Builder() - // `gregory` is a hybrid calendar that supports both - // the Julian and Gregorian calendar systems - .setCalendarType("gregory") - .setDate(ldt.getYear, ldt.getMonthValue - 1, ldt.getDayOfMonth) - .setTimeOfDay(ldt.getHour, ldt.getMinute, ldt.getSecond) - // Local time-line can overlaps, such as at an autumn daylight savings cutover. - // This setting selects the original local timestamp mapped to the given `micros`. - .set(Calendar.DST_OFFSET, zoneId.getRules.getDaylightSavings(instant).toMillis.toInt) - .build() - millisToMicros(cal.getTimeInMillis) + ldt.get(ChronoField.MICRO_OF_SECOND) - } - /** * The class describes JSON records with microseconds rebasing info. @@ -176,8 +159,8 @@ object RebaseDateTime { * Rebases micros since the epoch from an original to an target calendar, for instance, * from a hybrid (Julian + Gregorian) to Proleptic Gregorian calendar. * - * It finds the latest switch micros which is less than `value`, and adds the difference - * in micros associated with the switch micros to the given `value`. + * It finds the latest switch micros which is less than the given `micros`, and adds the + * difference in micros associated with the switch micros to the given `micros`. * The function is based on linear search which starts from the most recent switch micros. * This allows to perform less comparisons for modern timestamps. * @@ -227,22 +210,53 @@ object RebaseDateTime { private val gregJulianRebaseMap = loadRebaseRecords("gregorian-julian-rebase-micros.json") /** - * Rebases the given `micros` to the number of microseconds since the epoch via a local - * timestamp that have the same date-time fields in Proleptic Gregorian and in the hybrid - * calendars. + * Converts the given number of microseconds since the epoch '1970-01-01T00:00:00Z', to a local + * date-time in Proleptic Gregorian calendar with timezone identified by `zoneId`, interprets the + * result as a local date-time in Julian calendar with the same timezone, and takes microseconds + * since the epoch from the Julian local date-time. + * + * This is used to guarantee backward compatibility, as Spark 2.4 and earlier versions use + * Julian calendar for dates before 1582-10-15, while Spark 3.0 and later use Proleptic Gregorian + * calendar. See SPARK-26651. * - * The function may optimize the rebasing by using pre-calculated rebasing maps. If the maps - * don't contains information about the current JVM system time zone, the functions falls back - * to regular conversion mechanism via building local timestamps. + * For example: + * Proleptic Gregorian calendar: 1582-01-01 00:00:00.123456 -> -12244061221876544 + * Julian calendar: 1582-01-01 00:00:00.123456 -> -12243196799876544 + * The code below converts -12244061221876544 to -12243196799876544. + * + * @param zoneId The time zone ID at which the rebasing should be performed. + * @param micros The number of microseconds since the epoch '1970-01-01T00:00:00Z' + * in Proleptic Gregorian calendar. It can be negative. + * @return The rebased microseconds since the epoch in Julian calendar. + */ + private[sql] def rebaseGregorianToJulianMicros(zoneId: ZoneId, micros: Long): Long = { + val instant = microsToInstant(micros) + val ldt = instant.atZone(zoneId).toLocalDateTime + val cal = new Calendar.Builder() + // `gregory` is a hybrid calendar that supports both + // the Julian and Gregorian calendar systems + .setCalendarType("gregory") + .setDate(ldt.getYear, ldt.getMonthValue - 1, ldt.getDayOfMonth) + .setTimeOfDay(ldt.getHour, ldt.getMinute, ldt.getSecond) + // Local time-line can overlaps, such as at an autumn daylight savings cutover. + // This setting selects the original local timestamp mapped to the given `micros`. + .set(Calendar.DST_OFFSET, zoneId.getRules.getDaylightSavings(instant).toMillis.toInt) + .build() + millisToMicros(cal.getTimeInMillis) + ldt.get(ChronoField.MICRO_OF_SECOND) + } + + /** + * An optimized version of [[rebaseGregorianToJulianMicros(ZoneId, Long)]]. This method leverages + * the pre-calculated rebasing maps to save calculation. If the rebasing map doesn't contain + * information about the current JVM system time zone, the function falls back to the regular + * unoptimized version. * * Note: The function assumes that the input micros was derived from a local timestamp * at the default system JVM time zone in Proleptic Gregorian calendar. * - * @param micros The microseconds since the epoch 1970-01-01T00:00:00Z + * @param micros The number of microseconds since the epoch '1970-01-01T00:00:00Z' * in Proleptic Gregorian calendar. It can be negative. - * @return The microseconds since the epoch that have the same local timestamp representation - * in the hybrid calendar (Julian + Gregorian) as the original `micros` in - * Proleptic Gregorian calendar. + * @return The rebased microseconds since the epoch in Julian calendar. */ def rebaseGregorianToJulianMicros(micros: Long): Long = { val timeZone = TimeZone.getDefault @@ -256,14 +270,23 @@ object RebaseDateTime { } /** - * Converts the given microseconds to a local date-time in UTC time zone in the hybrid - * calendar (Julian + Gregorian since 1582-10-15), interprets the result as a local date-time - * in Proleptic Gregorian calendar in UTC time zone. And takes microseconds since the epoch - * from the Gregorian timestamp. + * Converts the given number of microseconds since the epoch '1970-01-01T00:00:00Z', to a local + * date-time in Julian calendar with timezone identified by `zoneId`, interprets the result as a + * local date-time in Proleptic Gregorian calendar with the same timezone, and takes microseconds + * since the epoch from the Gregorian local date-time. + * + * This is used to guarantee backward compatibility, as Spark 2.4 and earlier versions use + * Julian calendar for dates before 1582-10-15, while Spark 3.0 and later use Proleptic Gregorian + * calendar. See SPARK-26651. + * + * For example: + * Julian calendar: 1582-01-01 00:00:00.123456 -> -12243196799876544 + * Proleptic Gregorian calendar: 1582-01-01 00:00:00.123456 -> -12244061221876544 + * The code below converts -12243196799876544 to -12244061221876544. * * @param zoneId The time zone ID at which the rebasing should be performed. * @param micros The number of microseconds since the epoch '1970-01-01T00:00:00Z' - * in the hybrid calendar (Julian + Gregorian). It can be negative. + * in the Julian calendar. It can be negative. * @return The rebased microseconds since the epoch in Proleptic Gregorian calendar. */ private[sql] def rebaseJulianToGregorianMicros(zoneId: ZoneId, micros: Long): Long = { @@ -287,9 +310,13 @@ object RebaseDateTime { (Math.floorMod(micros, MICROS_PER_SECOND) * NANOS_PER_MICROS).toInt) .plusDays(cal.get(Calendar.DAY_OF_MONTH) - 1) val zonedDateTime = localDateTime.atZone(zoneId) - // Zero DST offset means that local clocks have switched to the winter time already. - // So, clocks go back one hour. We should correct zoned date-time and change - // the zone offset to the later of the two valid offsets at a local time-line overlap. + // Assuming the daylight saving switchover time is 2:00, the local clock will go back to + // 2:00 after hitting 2:59. This means the local time between [2:00, 3:00) appears twice, and + // can map to two different physical times (seconds from the UTC epoch). + // Java 8 time API resolves the ambiguity by picking the earlier physical time. This is the same + // as Java 7 time API, except for 2:00 where Java 7 picks the later physical time. + // Here we detect the "2:00" case and pick the latter physical time, to be compatible with the + // Java 7 date-time. val adjustedZdt = if (cal.get(Calendar.DST_OFFSET) == 0) { zonedDateTime.withLaterOffsetAtOverlap() } else { @@ -306,24 +333,17 @@ object RebaseDateTime { private val julianGregRebaseMap = loadRebaseRecords("julian-gregorian-rebase-micros.json") /** - * This is an opposite to `rebaseGregorianToJulianMicros` function which rebases the given - * microseconds since the epoch 1970-01-01T00:00:00Z via local timestamps from the - * hybrid calendar (Julian + Gregorian) to Proleptic Gregorian calendar. - * For example, the input `micros` -12243196799876544 is mapped to 1582-01-01 00:00:00.123456 in - * Julian calendar. The same local timestamp in Proleptic Gregorian calendar is mapped to - * the different number of micros -12244061221876544 since the epoch. Semantically, the function - * performs such conversion either via direct conversion to local timestamps, - * or via pre-calculated rebasing tables. + * An optimized version of [[rebaseJulianToGregorianMicros(ZoneId, Long)]]. This method leverages + * the pre-calculated rebasing maps to save calculation. If the rebasing map doesn't contain + * information about the current JVM system time zone, the function falls back to the regular + * unoptimized version. * * Note: The function assumes that the input micros was derived from a local timestamp - * at the default system JVM time zone in the hybrid calendar (Julian and Gregorian - * since 1582-10-15) + * at the default system JVM time zone in the Julian calendar. * - * @param micros The number of microseconds since the epoch 1970-01-01T00:00:00Z - * in the hybrid calendar (Julian + Gregorian). It can be negative. - * @return The rebased number of microseconds since the epoch which is mapped to the same - * local timestamp in Proleptic Gregorian calendar as `micros` in the hybrid calendar - * at the system time zone. + * @param micros The number of microseconds since the epoch '1970-01-01T00:00:00Z' + * in the Julian calendar. It can be negative. + * @return The rebased microseconds since the epoch in Proleptic Gregorian calendar. */ def rebaseJulianToGregorianMicros(micros: Long): Long = { val timeZone = TimeZone.getDefault --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org