[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022804#comment-15022804 ] Yin Huai commented on HIVE-3454: OK. Thanks. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Fix For: 1.2.0 > > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022785#comment-15022785
]
Aihua Xu commented on HIVE-3454:
[~yhuai] In 1.2.1, you need to {{set
hive.int.timestamp.conversion.in.seconds=true;}} to get the correct behavior,
see HIVE-9917. We kept the existing behavior for backward compatibility. In
2.0.0, we default hive.int.timestamp.conversion.in.seconds to true.
> Problem with CAST(BIGINT as TIMESTAMP)
> --
>
> Key: HIVE-3454
> URL: https://issues.apache.org/jira/browse/HIVE-3454
> Project: Hive
> Issue Type: Bug
> Components: Types, UDF
>Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0,
> 0.13.1
>Reporter: Ryan Harris
>Assignee: Aihua Xu
> Labels: newbie, newdev, patch
> Fix For: 1.2.0
>
> Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch
>
>
> Ran into an issue while working with timestamp conversion.
> CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current
> time from the BIGINT returned by unix_timestamp()
> Instead, however, a 1970-01-16 timestamp is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022420#comment-15022420 ] Yin Huai commented on HIVE-3454: I was using 1.2.1. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Fix For: 1.2.0 > > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022402#comment-15022402 ] Aihua Xu commented on HIVE-3454: [~yhuai] Which version are you running on? I tried on the latest and it seems to be fine. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Fix For: 1.2.0 > > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15014538#comment-15014538
]
Yin Huai commented on HIVE-3454:
Seems this issue has not been fixe completely. I still see
{code}
hive> SELECT CAST(CAST(-1200.0 AS TIMESTAMP) AS DOUBLE);
OK
-1200.0
Time taken: 0.047 seconds, Fetched: 1 row(s)
hive> SELECT CAST(CAST(-1200 AS TIMESTAMP) AS INT);
OK
-2
Time taken: 0.044 seconds, Fetched: 1 row(s)
{code}
> Problem with CAST(BIGINT as TIMESTAMP)
> --
>
> Key: HIVE-3454
> URL: https://issues.apache.org/jira/browse/HIVE-3454
> Project: Hive
> Issue Type: Bug
> Components: Types, UDF
>Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0,
> 0.13.1
>Reporter: Ryan Harris
>Assignee: Aihua Xu
> Labels: newbie, newdev, patch
> Fix For: 1.2.0
>
> Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch
>
>
> Ran into an issue while working with timestamp conversion.
> CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current
> time from the BIGINT returned by unix_timestamp()
> Instead, however, a 1970-01-16 timestamp is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378367#comment-14378367 ] Aihua Xu commented on HIVE-3454: Thanks [~csun] and [~jdere]. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Fix For: 1.2.0 > > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376863#comment-14376863 ] Jason Dere commented on HIVE-3454: -- +1 > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14375082#comment-14375082 ] Aihua Xu commented on HIVE-3454: This error is unrelated to the change and seems random. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14374774#comment-14374774
]
Hive QA commented on HIVE-3454:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12706362/HIVE-3454.3.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7819 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3107/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3107/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3107/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12706362 - PreCommit-HIVE-TRUNK-Build
> Problem with CAST(BIGINT as TIMESTAMP)
> --
>
> Key: HIVE-3454
> URL: https://issues.apache.org/jira/browse/HIVE-3454
> Project: Hive
> Issue Type: Bug
> Components: Types, UDF
>Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0,
> 0.13.1
>Reporter: Ryan Harris
>Assignee: Aihua Xu
> Labels: newbie, newdev, patch
> Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch
>
>
> Ran into an issue while working with timestamp conversion.
> CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current
> time from the BIGINT returned by unix_timestamp()
> Instead, however, a 1970-01-16 timestamp is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372897#comment-14372897
]
Hive QA commented on HIVE-3454:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12706162/HIVE-3454.3.patch
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 7819 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_between_in
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastLongToTimestamp
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3105/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3105/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3105/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12706162 - PreCommit-HIVE-TRUNK-Build
> Problem with CAST(BIGINT as TIMESTAMP)
> --
>
> Key: HIVE-3454
> URL: https://issues.apache.org/jira/browse/HIVE-3454
> Project: Hive
> Issue Type: Bug
> Components: Types, UDF
>Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0,
> 0.13.1
>Reporter: Ryan Harris
>Assignee: Aihua Xu
> Labels: newbie, newdev, patch
> Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch
>
>
> Ran into an issue while working with timestamp conversion.
> CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current
> time from the BIGINT returned by unix_timestamp()
> Instead, however, a 1970-01-16 timestamp is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372840#comment-14372840 ] Aihua Xu commented on HIVE-3454: Attached should be the right patch now. Maybe I uploaded the right patch first but I worried test run was not started (but seems the test was run against the right patch), so I uploaded again but the wrong one. Thanks for catching it. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372089#comment-14372089 ] Jason Dere commented on HIVE-3454: -- So this patch did not require any changes to MathExpr? How does the behavior of the vectorized cast change? > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14371672#comment-14371672
]
Hive QA commented on HIVE-3454:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12705949/HIVE-3454.3.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7820 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3091/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3091/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3091/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12705949 - PreCommit-HIVE-TRUNK-Build
> Problem with CAST(BIGINT as TIMESTAMP)
> --
>
> Key: HIVE-3454
> URL: https://issues.apache.org/jira/browse/HIVE-3454
> Project: Hive
> Issue Type: Bug
> Components: Types, UDF
>Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0,
> 0.13.1
>Reporter: Ryan Harris
>Assignee: Aihua Xu
> Labels: newbie, newdev, patch
> Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch
>
>
> Ran into an issue while working with timestamp conversion.
> CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current
> time from the BIGINT returned by unix_timestamp()
> Instead, however, a 1970-01-16 timestamp is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363650#comment-14363650 ] Jason Dere commented on HIVE-3454: -- Whoops, yes it is longToTimestamp() that should be fixed, since the point of this is to correct the int/long to timestamp behavior. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363355#comment-14363355 ] Aihua Xu commented on HIVE-3454: Thanks [~jdere]. We need to correct MathErpr.longToTimestamp() function since we have inconsistency with the int/long type, Correct? > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359502#comment-14359502 ] Jason Dere commented on HIVE-3454: -- I've just started looking at vectorized code .. looks like the corresponding change to make for the vectorized path will be in MathExpr.doubleToTimestamp(). CC'ing [~mmccline] in case there is any more details to add here. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359406#comment-14359406 ] Aihua Xu commented on HIVE-3454: Yes. That's right. That patch is to change in the correct way to interpreting all the datatypes as seconds. Thanks for looking. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359383#comment-14359383 ] Jason Dere commented on HIVE-3454: -- This has been marked patch available - which one should we be looking at - HIVE-3454.3.patch? > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359144#comment-14359144 ] Jason Dere commented on HIVE-3454: -- That's fine to break into separate tasks, we'll just need to make sure the followup config task gets in before the next release. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14358988#comment-14358988 ] Chao commented on HIVE-3454: +1. Breaking into separate tasks makes sense to me, but it's better to have others' inputs (I'm no expert on this matter), before we commit this. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355789#comment-14355789 ] Aihua Xu commented on HIVE-3454: [~brocknoland], [~jdere] and [~hagleitn] I created HIVE-9917 for the followup change. I will mark the current change as incompatible. How does that sound? > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355763#comment-14355763 ] Aihua Xu commented on HIVE-3454: I'm looking into the solution [~hagleitn] suggested. I will break the tasks into separate tasks, so that this jira will focus on the right/simple behavior and a followup to focus on making it configurable so that we can easily revert in the future and also focus on that only. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348023#comment-14348023 ] Aihua Xu commented on HIVE-3454: Of course, since we need to support LLAP, UDF approach seems to be the right approach. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347905#comment-14347905 ] Aihua Xu commented on HIVE-3454: I agree that for non thread-safe statics, it will cause problems for multi-threading, while for constants or immutable objects, I guess you would share across threads rather than creating each copy for each thread. This is the case of a java primitive that we initialize once and use/read later by threads. Regarding UDF approach, I looked into that and it seems promising, while the current approach actually only evaluates the value when the type is timestamp. I guess it would be the same as UDF approach. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347582#comment-14347582 ] Gunther Hagleitner commented on HIVE-3454: -- I agree with [~sershe]. These statics aren't just a pain w/ multi threading, they also hurt if you just re-use the same jvm for multiple things. This can happen in tez, spark and if we decide to run stuff on the client/in hs2 afaik. More specifically though: This config should take effect at compile time. There is no reason to evaluate the condition for each value in each row at run time. We should be able to just install the appropriate udf when we compile the query, no? > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347540#comment-14347540 ] Sergey Shelukhin commented on HIVE-3454: It just seems like a bad design pattern irrespective of LLAP. [~hagleitn] did some work in operator pipeline to clean up statics and such, he can comment. What is the real scope of this variable? It should be scoped accordingly. Even a singleton map would be better... > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347042#comment-14347042 ] Aihua Xu commented on HIVE-3454: Was not aware of LLAP. Seems reasonable to defer the issue since I guess LLAP needs to handle many other similar thread-safe problems like this. Correct me if I'm wrong with the hadoop execution: currently (without LLAP) each ExecMapper and ExecReducer runs in its own JVM and it could spin off new threads. Each ExecMapper and ExecReducer handles one job from a query and we want those threads to share that same static variable value. With LLAP, if ExecMapper runs in a thread and doesn't spin off new threads, then thread-local would work, but we still need to handle current existing scenario. If new threads are spun off, thread-local actually won't work. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346454#comment-14346454 ] Brock Noland commented on HIVE-3454: bq. an HOS have multiple threads doing work for more than one query at the same time? no it cannot. HOS does do concurrent tasks but not across queries. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346318#comment-14346318 ] Jason Dere commented on HIVE-3454: -- I'm willing to defer that work to a different Jira. +1 But I'm surprised this would not be an issue with HOS as well .. can HOS have multiple threads doing work for more than one query at the same time? If so then HOS would get hit by this too since this setting is relying on a static variable. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346302#comment-14346302 ] Brock Noland commented on HIVE-3454: HOS won't have this issue as the executors are tied to a user session. I think we should commit the patch as is and create a follow-up to address the issue with LLAP. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346184#comment-14346184
]
Jason Dere commented on HIVE-3454:
--
{quote}
Actually we don't want it to be thread-local since we want all threads, e.g,
within Mapper to have the same settings. If we set to be thread-local, the
threads within the same mapper which don't set intToTimeStampInSeconds will
have incorrect behaviors. Of course, the original problem was that we can't
access hive properties from TimestampWritable class and we have to set from
outside.
{quote}
Asking around a bit about HIVE-7926, it sounds like LLAP will potentially have
multiple queries running at the same time in different threads, which would
make a single static variable problematic. It also sounds like each thread
should in theory have the equivalent of something like ExecMapper.configure()
where thread-local context is initialized, so I think that any threads running
map/reduce work should have an opportunity to set intToTimeStampInSeconds. I
would think that things would be similar for Spark as well, though someone with
more familiarity with Hive-Spark execution would be better suited to answer
that question.
> Problem with CAST(BIGINT as TIMESTAMP)
> --
>
> Key: HIVE-3454
> URL: https://issues.apache.org/jira/browse/HIVE-3454
> Project: Hive
> Issue Type: Bug
> Components: Types, UDF
>Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0,
> 0.13.1
>Reporter: Ryan Harris
>Assignee: Aihua Xu
> Labels: newbie, newdev, patch
> Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch,
> HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch
>
>
> Ran into an issue while working with timestamp conversion.
> CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current
> time from the BIGINT returned by unix_timestamp()
> Instead, however, a 1970-01-16 timestamp is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346156#comment-14346156 ] Sergey Shelukhin commented on HIVE-3454: Please don't use the static intToTimeStampInSeconds, it is not thread-safe > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346115#comment-14346115 ] Aihua Xu commented on HIVE-3454: #Actually we don't want it to be thread-local since we want all threads, e.g, within Mapper to have the same settings. If we set to be thread-local, the threads within the same mapper which don't set intToTimeStampInSeconds will have incorrect behaviors. Of course, the original problem was that we can't access hive properties from TimestampWritable class and we have to set from outside. #I will create a follow-up jira to do the rest (Tez and Spark). > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346047#comment-14346047 ] Jason Dere commented on HIVE-3454: -- - Can you make intToTimeStampInSeconds thread-local? - According to [~vikram.dixit], Tez uses separate classes, the equivalent to ExecMapper/ExecReducer are MapRecordProcessor/ReduceRecordProcessor. It looks like Spark also has its own equivalents (SparkMapRecordHandler/SparkMapReduceHandler). If you want we can defer those changes and open a follow-up Jira to do that work. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345766#comment-14345766 ] Aihua Xu commented on HIVE-3454: [~jdere] Can you review the latest patch? Right now it should be set once for each Mapper and Reducer. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[
https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14344253#comment-14344253
]
Hive QA commented on HIVE-3454:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12701998/HIVE-3454.4.patch
{color:green}SUCCESS:{color} +1 7580 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2921/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2921/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2921/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12701998 - PreCommit-HIVE-TRUNK-Build
> Problem with CAST(BIGINT as TIMESTAMP)
> --
>
> Key: HIVE-3454
> URL: https://issues.apache.org/jira/browse/HIVE-3454
> Project: Hive
> Issue Type: Bug
> Components: Types, UDF
>Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0,
> 0.13.1
>Reporter: Ryan Harris
>Assignee: Aihua Xu
> Labels: newbie, newdev, patch
> Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch,
> HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch
>
>
> Ran into an issue while working with timestamp conversion.
> CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current
> time from the BIGINT returned by unix_timestamp()
> Instead, however, a 1970-01-16 timestamp is returned.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343913#comment-14343913 ] Aihua Xu commented on HIVE-3454: Updated to initialize for local mode in Driver.runInternal() and configure() of ExecMapper and ExecReducer. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340221#comment-14340221 ] Aihua Xu commented on HIVE-3454: I see your points that multiple threads run in single mapper or reducer access that class and that may cause concurrency issue. I will try to call the initialize() from Mapper and Reducer's initialization function to make sure it initializes once. And also Driver.compile() (I will check what would be the good place for the local mode). > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337027#comment-14337027 ] Jason Dere commented on HIVE-3454: -- If concurrent compilation is possible, then there is still the possibility of multiple threads setting/accessing the same static variable and messing up the config setting for the other threads. You're adding this to TimestampWritable - there is even a member in that class that was changed to thread-local as a result of HIVE-4516. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14336537#comment-14336537 ] Aihua Xu commented on HIVE-3454: If the job runs locally in-process, like cast(0 as timestamp), SerDe is still used for intermediate output and thus the property still gets set and works fine. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335946#comment-14335946 ] Jason Dere commented on HIVE-3454: -- I guess the only possible issue here might be for expressions that get evaluated during compilation as opposed to during the tasks, something like cast(0 as timestamp) which would get evaluated by constant folding. Not sure if there is any other possible work/tasks that happen locally in-process. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14335874#comment-14335874 ] Aihua Xu commented on HIVE-3454: The unit test failure is unrelated to the change. [~jdere] More comments on my input above? > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-3454) Problem with CAST(BIGINT as TIMESTAMP)
[ https://issues.apache.org/jira/browse/HIVE-3454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14333554#comment-14333554 ] Aihua Xu commented on HIVE-3454: [~jdere] I spent some time looking into the parallel in Hive. Hive can handle queries in parallel or a complex query breaking into subtasks executed in parallel. In both cases, Hive hands over the multiple jobs to hadoop and those jobs won't affect each other. I have simulated the cases by debugging them from different sessions and it works as expected. > Problem with CAST(BIGINT as TIMESTAMP) > -- > > Key: HIVE-3454 > URL: https://issues.apache.org/jira/browse/HIVE-3454 > Project: Hive > Issue Type: Bug > Components: Types, UDF >Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.11.0, 0.12.0, 0.13.0, > 0.13.1 >Reporter: Ryan Harris >Assignee: Aihua Xu > Labels: newbie, newdev, patch > Attachments: HIVE-3454.1.patch.txt, HIVE-3454.2.patch, > HIVE-3454.3.patch, HIVE-3454.4.patch, HIVE-3454.patch > > > Ran into an issue while working with timestamp conversion. > CAST(unix_timestamp() as TIMESTAMP) should create a timestamp for the current > time from the BIGINT returned by unix_timestamp() > Instead, however, a 1970-01-16 timestamp is returned. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
