fifteencai has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16869 )
Change subject: IMPALA-9922: A better approach to deal with date's sub-second fractions ...................................................................... IMPALA-9922: A better approach to deal with date's sub-second fractions In this patch, we relax the constraint on date format of to_timestamp(). Previously, the sub-second fraction width should never exceed the data, otherwise, the conversion will fail. This constraint is stricter than many query engines like Hive, Presto, ClickHouse etc. Hence we've encountered inconsistent conversions on dataset with malformed values. To relax this constraint, we made these modifications: 1) Introduced a new parameter `desired_length` alongside with `tok_len`, the former parameter refers to the fraction width we want. 2) When fraction part is a single `S`, the `desired_length` is set to actual width of data, making the output consistent with older logic. 3) In `DatetimeSimpleDateFormatParser`, move a pointer computation out of the loop, avoiding unnecessary code execution. ======================================================================== Example: > Before: select to_timestamp("2020-01-01 18:00:00.12","yyyy-MM-dd HH:mm:ss.SSS") NULL ------------------------------------------------------------------------ > After: select to_timestamp("2020-01-01 18:00:00.12","yyyy-MM-dd HH:mm:ss.SSS") 2020-01-01 18:00:00.0120000000 ======================================================================== Testing: 1. Added 3 test cases in `timestamp-test.cc`. 2. Passed all backend tests Change-Id: I8e870bb8ad8fd14d388f37dfc5175589ecf9a5a7 --- M be/src/runtime/datetime-iso-sql-format-parser.cc M be/src/runtime/datetime-parser-common.cc M be/src/runtime/datetime-parser-common.h M be/src/runtime/datetime-simple-date-format-parser.cc M be/src/runtime/timestamp-test.cc 5 files changed, 36 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/69/16869/2 -- To view, visit http://gerrit.cloudera.org:8080/16869 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8e870bb8ad8fd14d388f37dfc5175589ecf9a5a7 Gerrit-Change-Number: 16869 Gerrit-PatchSet: 2 Gerrit-Owner: fifteencai <fifteen...@tencent.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>