[ https://issues.apache.org/jira/browse/HIVE-25449?focusedWorklogId=640598&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-640598 ]
ASF GitHub Bot logged work on HIVE-25449: ----------------------------------------- Author: ASF GitHub Bot Created on: 23/Aug/21 07:09 Start Date: 23/Aug/21 07:09 Worklog Time Spent: 10m Work Description: abstractdog commented on pull request #2591: URL: https://github.com/apache/hive/pull/2591#issuecomment-903502845 +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 640598) Time Spent: 1h (was: 50m) > datediff() gives wrong output when run in a tez task with some non-UTC > timezone > ------------------------------------------------------------------------------- > > Key: HIVE-25449 > URL: https://issues.apache.org/jira/browse/HIVE-25449 > Project: Hive > Issue Type: Bug > Components: UDF > Reporter: Shubham Chaurasia > Assignee: Shubham Chaurasia > Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Repro (thanks Qiaosong Dong) - > Add -Duser.timezone=GMT+8 to {{tez.task.launch.cmd-opts}} > {code} > create external table test_dt(id string, dt date); > insert into test_dt values('11', '2021-07-06'), ('22', '2021-07-07'); > select datediff(dt1.dt, '2021-07-01') from test_dt dt1 left join test_dt dt > on dt1.id = dt.id; > +------+ > | _c0 | > +------+ > | 6 | > | 7 | > +------+ > {code} > Expected output - > {code} > +------+ > | _c0 | > +------+ > | 5 | > | 6 | > +------+ > {code} > *Cause* > This happens because in {{VectorUDFDateDiffColScalar}} class > 1. For second argument(scalar) , we use {{java.text.SimpleDateFormat}} to > parse the date strings which interprets it to be in local timezone. > 2. For first column we get a column vector which represents the date as epoch > day. This is always in UTC. > *Solution* > We need to check other variants of datediff UDFs as well and change the > parsing mechanism to always interpret date strings in UTC. > > I did a quick change in {{VectorUDFDateDiffColScalar}} which fixes the issue. > {code} > - date.setTime(formatter.parse(new String(bytesValue, > "UTF-8")).getTime()); > - baseDate = DateWritableV2.dateToDays(date); > + org.apache.hadoop.hive.common.type.Date hiveDate > + = org.apache.hadoop.hive.common.type.Date.valueOf(new > String(bytesValue, "UTF-8")); > + date.setTime(hiveDate.toEpochMilli()); > + baseDate = hiveDate.toEpochDay(); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)