turcsanyip commented on a change in pull request #4781: URL: https://github.com/apache/nifi/pull/4781#discussion_r564092023
########## File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/ResultSetRecordSet.java ########## @@ -26,12 +27,12 @@ import java.math.BigDecimal; import java.math.BigInteger; import java.sql.Array; +import java.sql.Date; Review comment: It was intentional because it seemed to be a bug. `ResultSet.getMetaData().getColumnClassName()` would never return `java.util.Date` but the SQL date/time types. However, now I see that this code section is related to QueryRecord and it may use `java.util.Date`. I'll revert it back. Thanks for the heads up that made me double check it in more detail. ########## File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java ########## @@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName); } + /** + * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet) + * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day). + * + * @param date java.sql.Date with local time zone 00:00:00 + * @return java.sql.Date with UTC 00:00:00 + */ + public static Date convertDateToUTC(Date date) { + return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli()); Review comment: `atStartOfDay()` has `ZoneId` parameter only. In my understanding `ZoneOffset` is DST-agnostic, so the `ZoneId` is the right choice in any case. ########## File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java ########## @@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio testConvertToAvroStreamForDateTime(options, (record, date) -> { + java.sql.Date expected = DataTypeUtils.convertDateToUTC(date); final int daysSinceEpoch = (int) record.get("date"); final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS); - java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate()); - LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual); - assertEquals(date, actual); + java.sql.Date actual = new java.sql.Date(millisSinceEpoch); + LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual); + assertEquals(expected, actual); Review comment: These tests (historically) do not assert the raw value (int / long) from the Avro record but reconstruct the Date/Time/Timestamp objects and compare them with the original objects (which were the input of the test case). I believe the reason was to avoid to calculate the days/millis/etc. from the test input data which would be a similarly complicated method. As the test inputs are literals, it is not needed to calculate the expected value from code, but it can be determined once and hard coded as literals too. What do you think? I mean ``` int expectedDaysSinceEpoch = 17444 assertDate.accept(record, expectedDaysSinceEpoch); ``` ########## File path: nifi-nar-bundles/nifi-extension-utils/nifi-database-utils/src/test/java/org/apache/nifi/util/db/TestJdbcCommon.java ########## @@ -679,11 +680,12 @@ public void testConvertToAvroStreamForDateTimeAsLogicalType() throws SQLExceptio testConvertToAvroStreamForDateTime(options, (record, date) -> { + java.sql.Date expected = DataTypeUtils.convertDateToUTC(date); final int daysSinceEpoch = (int) record.get("date"); final long millisSinceEpoch = TimeUnit.MILLISECONDS.convert(daysSinceEpoch, TimeUnit.DAYS); - java.sql.Date actual = java.sql.Date.valueOf(Instant.ofEpochMilli(millisSinceEpoch).atZone(ZoneOffset.UTC).toLocalDate()); - LOGGER.debug("comparing dates, expecting '{}', actual '{}'", date, actual); - assertEquals(date, actual); + java.sql.Date actual = new java.sql.Date(millisSinceEpoch); + LOGGER.debug("comparing dates, expecting '{}', actual '{}'", expected, actual); + assertEquals(expected, actual); Review comment: These tests (historically) do not assert the raw value (int / long) from the Avro record but reconstruct the Date/Time/Timestamp objects and compare them with the original objects (which were the inputs of the test cases). I believe the reason was to avoid to calculate the days/millis/etc. from the test input data which would be a similarly complicated method. As the test inputs are literals, it is not needed to calculate the expected value from code, but it can be determined once and hard coded as literals too. What do you think? I mean: ``` int expectedDaysSinceEpoch = 17444; assertDate.accept(record, expectedDaysSinceEpoch); ``` ########## File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java ########## @@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName); } + /** + * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet) + * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day). + * + * @param date java.sql.Date with local time zone 00:00:00 + * @return java.sql.Date with UTC 00:00:00 + */ + public static Date convertDateToUTC(Date date) { + return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli()); + } + + /** + * Converts a java.sql.Date object with 00:00:00 in UTC + * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day). + * + * @param date java.sql.Date with UTC 00:00:00 + * @return java.sql.Date with local time zone 00:00:00 + */ + public static Date convertDateToLocalTZ(Date date) { + return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate()); Review comment: Indeed... what do you think about this? ``` public static Date convertDateToUTC(Date dateLocalTZ) { ZonedDateTime zdtLocalTZ = ZonedDateTime.ofInstant(Instant.ofEpochMilli(dateLocalTZ.getTime()), ZoneId.systemDefault()); ZonedDateTime zdtUTC = zdtLocalTZ.withZoneSameLocal(ZoneId.of("UTC")); Date dateUTC = new Date(zdtUTC.toInstant().toEpochMilli()); return dateUTC; } ``` No `LocaDate` and I believe `withZoneSameLocal()` in the middle of the method really grasps what is going on here. ########## File path: nifi-commons/nifi-record/src/main/java/org/apache/nifi/serialization/record/util/DataTypeUtils.java ########## @@ -1085,6 +1087,28 @@ private static Object toEnum(Object value, EnumDataType dataType, String fieldNa throw new IllegalTypeConversionException("Cannot convert value [" + value + "] of type " + value.getClass() + " to Date for field " + fieldName); } + /** + * Converts a java.sql.Date object with 00:00:00 in local time zone (typically coming from a java.sql.ResultSet) + * to UTC normalized form (storing epoch corresponding to UTC 00:00:00 on the given day). + * + * @param date java.sql.Date with local time zone 00:00:00 + * @return java.sql.Date with UTC 00:00:00 + */ + public static Date convertDateToUTC(Date date) { + return new Date(date.toLocalDate().atStartOfDay(ZoneId.of("UTC")).toInstant().toEpochMilli()); + } + + /** + * Converts a java.sql.Date object with 00:00:00 in UTC + * to local time zone normalized form (storing epoch corresponding to 00:00:00 in local time zone on the given day). + * + * @param date java.sql.Date with UTC 00:00:00 + * @return java.sql.Date with local time zone 00:00:00 + */ + public static Date convertDateToLocalTZ(Date date) { + return Date.valueOf(Instant.ofEpochMilli(date.getTime()).atZone(ZoneId.of("UTC")).toLocalDate()); Review comment: Indeed... what do you think about this? ``` public static Date convertDateToUTC(Date dateLocalTZ) { ZonedDateTime zdtLocalTZ = ZonedDateTime.ofInstant(Instant.ofEpochMilli(dateLocalTZ.getTime()), ZoneId.systemDefault()); ZonedDateTime zdtUTC = zdtLocalTZ.withZoneSameLocal(ZoneId.of("UTC")); Date dateUTC = new Date(zdtUTC.toInstant().toEpochMilli()); return dateUTC; } ``` No `LocaDate` and I believe `withZoneSameLocal()` in the middle of the method really grasps what is going on here. The same can be applied to `convertDateToLocalTZ()` similarly. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org