[jira] [Commented] (HIVE-20007) Hive should carry out timestamp computations in UTC
[
https://issues.apache.org/jira/browse/HIVE-20007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525992#comment-16525992
]
Hive QA commented on HIVE-20007:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929350/HIVE-20007.patch
{color:green}SUCCESS:{color} +1 due to 80 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14616 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12205/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12205/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12205/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929350 - PreCommit-HIVE-Build
> Hive should carry out timestamp computations in UTC
> ---
>
> Key: HIVE-20007
> URL: https://issues.apache.org/jira/browse/HIVE-20007
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Ryan Blue
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Labels: timestamp
> Attachments: HIVE-20007.patch
>
>
> Hive currently uses the "local" time of a java.sql.Timestamp to represent the
> SQL data type TIMESTAMP WITHOUT TIME ZONE. The purpose is to be able to use
> {{Timestamp#getYear()}} and similar methods to implement SQL functions like
> {{year}}.
> When the SQL session's time zone is a DST zone, such as America/Los_Angeles
> that alternates between PST and PDT, there are times that cannot be
> represented because the effective zone skips them.
> {code}
> hive> select TIMESTAMP '2015-03-08 02:10:00.101';
> 2015-03-08 03:10:00.101
> {code}
> Using UTC instead of the SQL session time zone as the underlying zone for a
> java.sql.Timestamp avoids this bug, while still returning correct values for
> {{getYear}} etc. Using UTC as the convenience representation (timestamp
> without time zone has no real zone) would make timestamp calculations more
> consistent and avoid similar problems in the future.
> Notably, this would break the {{unix_timestamp}} UDF that specifies the
> result is with respect to ["the default timezone and default
> locale"|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions].
> That function would need to be updated to use the
> {{System.getProperty("user.timezone")}} zone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20004) Wrong scale used by ConvertDecimal64ToDecimal results in incorrect results
[
https://issues.apache.org/jira/browse/HIVE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525942#comment-16525942
]
Hive QA commented on HIVE-20004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929346/HIVE-20004.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12204/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12204/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12204/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12929346/HIVE-20004.2.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929346 - PreCommit-HIVE-Build
> Wrong scale used by ConvertDecimal64ToDecimal results in incorrect results
> --
>
> Key: HIVE-20004
> URL: https://issues.apache.org/jira/browse/HIVE-20004
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-20004.1.patch, HIVE-20004.2.patch
>
>
> ConvertDecimal64ToDecimal uses scale from output column vector which results
> in incorrect results.
> Input: decimal(8,1) Output: decimal(9,2)
> Input value: 963.8 gets converted to 96.38 which is wrong. The scale should
> not change this case (value should be 963.8 even after the conversion).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19967) SMB Join : Need Optraits for PTFOperator ala GBY Op
[
https://issues.apache.org/jira/browse/HIVE-19967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525941#comment-16525941
]
Hive QA commented on HIVE-19967:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929453/HIVE-19967.7.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 14614 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_dynamic_partition]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_expressions]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_masking]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test1]
(batchId=190)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_test_insert]
(batchId=190)
org.apache.hadoop.hive.metastore.client.TestRuntimeStats.testCleanup[Embedded]
(batchId=216)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testParallelCompilation3 (batchId=250)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12203/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12203/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12203/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929453 - PreCommit-HIVE-Build
> SMB Join : Need Optraits for PTFOperator ala GBY Op
> ---
>
> Key: HIVE-19967
> URL: https://issues.apache.org/jira/browse/HIVE-19967
> Project: Hive
> Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-19967.1.patch, HIVE-19967.2.patch,
> HIVE-19967.3.patch, HIVE-19967.4.patch, HIVE-19967.5.patch,
> HIVE-19967.6.patch, HIVE-19967.7.patch
>
>
> The SMB join on one or more PTF Ops should reset the optraits keys just like
> GBY Op does.
> Currently there is no implementation of PTFOp optraits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19967) SMB Join : Need Optraits for PTFOperator ala GBY Op
[
https://issues.apache.org/jira/browse/HIVE-19967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525921#comment-16525921
]
Hive QA commented on HIVE-19967:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
55s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 1 new + 14 unchanged - 1 fixed
= 15 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12203/dev-support/hive-personality.sh
|
| git revision | master / 67b0a67 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12203/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12203/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SMB Join : Need Optraits for PTFOperator ala GBY Op
> ---
>
> Key: HIVE-19967
> URL: https://issues.apache.org/jira/browse/HIVE-19967
> Project: Hive
> Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-19967.1.patch, HIVE-19967.2.patch,
> HIVE-19967.3.patch, HIVE-19967.4.patch, HIVE-19967.5.patch,
> HIVE-19967.6.patch, HIVE-19967.7.patch
>
>
> The SMB join on one or more PTF Ops should reset the optraits keys just like
> GBY Op does.
> Currently there is no implementation of PTFOp optraits.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19812) Disable external table replication by default via a configuration property
[
https://issues.apache.org/jira/browse/HIVE-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525907#comment-16525907
]
Hive QA commented on HIVE-19812:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929331/HIVE-19812.10.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14614 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins]
(batchId=191)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12202/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12202/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12202/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929331 - PreCommit-HIVE-Build
> Disable external table replication by default via a configuration property
> --
>
> Key: HIVE-19812
> URL: https://issues.apache.org/jira/browse/HIVE-19812
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-19812.01.patch, HIVE-19812.02.patch,
> HIVE-19812.03.patch, HIVE-19812.04.patch, HIVE-19812.05.patch,
> HIVE-19812.06-branch-3.patch, HIVE-19812.06.patch, HIVE-19812.07.patch,
> HIVE-19812.08.patch, HIVE-19812.09.patch, HIVE-19812.10.patch
>
>
> use a hive config property to allow external table replication. set this
> property by default to prevent external table replication.
> for metadata only hive repl always export metadata for external tables.
>
> REPL_DUMP_EXTERNAL_TABLES("hive.repl.dump.include.external.tables", false,
> "Indicates if repl dump should include information about external tables. It
> should be \n"
> + "used in conjunction with 'hive.repl.dump.metadata.only' set to false. if
> 'hive.repl.dump.metadata.only' \n"
> + " is set to true then this config parameter has no effect as external table
> meta data is flushed \n"
> + " always by default.")
> This should be done for only replication dump and not for export
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19829) Incremental replication load should create tasks in execution phase rather than semantic phase
[ https://issues.apache.org/jira/browse/HIVE-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525895#comment-16525895 ] Sankar Hariappan commented on HIVE-19829: - 10.patch is committed to master. Thanks [~maheshk114] for the contribution! > Incremental replication load should create tasks in execution phase rather > than semantic phase > -- > > Key: HIVE-19829 > URL: https://issues.apache.org/jira/browse/HIVE-19829 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19829.01.patch, HIVE-19829.02.patch, > HIVE-19829.03.patch, HIVE-19829.04.patch, HIVE-19829.06.patch, > HIVE-19829.07.patch, HIVE-19829.07.patch, HIVE-19829.08-branch-3.patch, > HIVE-19829.08.patch, HIVE-19829.09.patch, HIVE-19829.10.patch > > > Split the incremental load into multiple iterations. In each iteration create > number of tasks equal to the configured value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19829) Incremental replication load should create tasks in execution phase rather than semantic phase
[ https://issues.apache.org/jira/browse/HIVE-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525891#comment-16525891 ] Sankar Hariappan commented on HIVE-19829: - +1 > Incremental replication load should create tasks in execution phase rather > than semantic phase > -- > > Key: HIVE-19829 > URL: https://issues.apache.org/jira/browse/HIVE-19829 > Project: Hive > Issue Type: Task > Components: repl >Affects Versions: 3.1.0, 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-19829.01.patch, HIVE-19829.02.patch, > HIVE-19829.03.patch, HIVE-19829.04.patch, HIVE-19829.06.patch, > HIVE-19829.07.patch, HIVE-19829.07.patch, HIVE-19829.08-branch-3.patch, > HIVE-19829.08.patch, HIVE-19829.09.patch, HIVE-19829.10.patch > > > Split the incremental load into multiple iterations. In each iteration create > number of tasks equal to the configured value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19812) Disable external table replication by default via a configuration property
[
https://issues.apache.org/jira/browse/HIVE-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525892#comment-16525892
]
Hive QA commented on HIVE-19812:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 110
unchanged - 0 fixed = 112 total (was 110) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12202/dev-support/hive-personality.sh
|
| git revision | master / 6a8f4cb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12202/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: common ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12202/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Disable external table replication by default via a configuration property
> --
>
> Key: HIVE-19812
> URL: https://issues.apache.org/jira/browse/HIVE-19812
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 3.2.0
>
> Attachments: HIVE-19812.01.patch, HIVE-19812.02.patch,
> HIVE-19812.03.patch, HIVE-19812.04.patch, HIVE-19812.05.patch,
> HIVE-19812.06-branch-3.patch, HIVE-19812.06.patch, HIVE-19812.07.patch,
> HIVE-19812.08.patch, HIVE-19812.09.patch, HIVE-19812.10.patch
>
>
> use a hive config p
[jira] [Updated] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19916: Resolution: Not A Problem Status: Resolved (was: Patch Available) After some email discussion, looks like this patch is not needed. The existing code always uses the same txn list so it can never get different write IDs. > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525870#comment-16525870 ] Sergey Shelukhin commented on HIVE-19975: - Committed the patch to the branch that adds acid_stats test; this tests two scenarios I was looking at, the first part is the broken part, after the 2nd partition, explain doesn't use stats anymore. > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table > --- > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19829) Incremental replication load should create tasks in execution phase rather than semantic phase
[
https://issues.apache.org/jira/browse/HIVE-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525869#comment-16525869
]
Hive QA commented on HIVE-19829:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929332/HIVE-19829.10.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14612 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12201/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12201/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12201/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929332 - PreCommit-HIVE-Build
> Incremental replication load should create tasks in execution phase rather
> than semantic phase
> --
>
> Key: HIVE-19829
> URL: https://issues.apache.org/jira/browse/HIVE-19829
> Project: Hive
> Issue Type: Task
> Components: repl
>Affects Versions: 3.1.0, 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-19829.01.patch, HIVE-19829.02.patch,
> HIVE-19829.03.patch, HIVE-19829.04.patch, HIVE-19829.06.patch,
> HIVE-19829.07.patch, HIVE-19829.07.patch, HIVE-19829.08-branch-3.patch,
> HIVE-19829.08.patch, HIVE-19829.09.patch, HIVE-19829.10.patch
>
>
> Split the incremental load into multiple iterations. In each iteration create
> number of tasks equal to the configured value.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19975: Summary: Checking writeIdList per table may not check the commit level of a partition on a partitioned table (was: Checking writeIdList per table may not check the commit level of a partition on a partitioned table.) > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table > --- > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Attachment: HIVE-19850.patch
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesha Shreedhara updated HIVE-19850:
--
Attachment: (was: HIVE-19850.patch)
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2`;
> CREATE VIEW `view2` AS select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_boolean`,`t2`.`dt`
> from `t2` union all select
> `t3`.`key`,`t3`.`value`,`t3`.`c_int`,`t3`.`c_float`,`t3`.`c_boolean`,`t3`.`dt`
> from `t3`;
> create table t4 as select key,value,c_int,c_float,c_boolean,dt from t1 union
> all select v1.key,v1.value,v1.c_int,v1.c_float,v1.c_boolean,v1.dt from view1
> v1 join view2 v2 on v1.dt=v2.dt;
> CREATE VIEW `view3` AS select
> `t4`.`key`,`t4`.`value`,`t4`.`c_int`,`t4`.`c_float`,`t4`.`c_boolean`,`t4`.`dt`
> from `t4` union all select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1`;
>
> select count(0) from view2 v2 join view3 v3 on v2.dt=v3.dt; // Throws No work
> found for tablescan error
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table.
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525857#comment-16525857 ] Sergey Shelukhin commented on HIVE-19975: - I found out why the tests work. in HiveMetaStore, HiveMetaStore::get_partitions_statistics_req method does not propagate write ID to the next class (they are never extracted from the request); so, it calls the overload without the stats check. Also one of my test fixes introduced another issue downstream that also disables the check silently after that. I will fix both and commit a test that shows this problem. > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table. > > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table.
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525857#comment-16525857 ] Sergey Shelukhin edited comment on HIVE-19975 at 6/28/18 3:12 AM: -- I found out why the tests work. in HiveMetaStore, HiveMetaStore::get_partitions_statistics_req method does not propagate write ID to the next class (they are never extracted from the request); so, it calls the overload without the stats check. Also one of my test fixes introduced another issue downstream that also disables the check silently after that is fixed. I will fix both and commit a test that shows this problem. was (Author: sershe): I found out why the tests work. in HiveMetaStore, HiveMetaStore::get_partitions_statistics_req method does not propagate write ID to the next class (they are never extracted from the request); so, it calls the overload without the stats check. Also one of my test fixes introduced another issue downstream that also disables the check silently after that. I will fix both and commit a test that shows this problem. > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table. > > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19829) Incremental replication load should create tasks in execution phase rather than semantic phase
[
https://issues.apache.org/jira/browse/HIVE-19829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525854#comment-16525854
]
Hive QA commented on HIVE-19829:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 8 new + 178 unchanged - 8
fixed = 186 total (was 186) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 122
unchanged - 4 fixed = 126 total (was 126) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
15s{color} | {color:red} ql generated 3 new + 2277 unchanged - 3 fixed = 2280
total (was 2280) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | org.apache.hadoop.hive.ql.exec.repl.ReplLoadWork is Serializable;
consider declaring a serialVersionUID At ReplLoadWork.java:a serialVersionUID
At ReplLoadWork.java:[lines 42-111] |
| | Write to static field
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.numIteration
from instance method new
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder(String,
String, String, IncrementalLoadEventsIterator, HiveConf) At
IncrementalLoadTasksBuilder.java:from instance method new
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder(String,
String, String, IncrementalLoadEventsIterator, HiveConf) At
IncrementalLoadTasksBuilder.java:[line 83] |
| | Write to static field
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.numIteration
from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger) At IncrementalLoadTasksBuilder.java:from instance method
org.apache.hadoop.hive.ql.exec.repl.incremental.IncrementalLoadTasksBuilder.build(DriverContext,
Hive, Logger) At IncrementalLoadTasksBuilder.java:[line 92] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12201/dev-support/hive-persona
[jira] [Commented] (HIVE-19027) Make materializations invalidation cache work with multiple active remote metastores
[
https://issues.apache.org/jira/browse/HIVE-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525842#comment-16525842
]
Hive QA commented on HIVE-19027:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
6s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
13s{color} | {color:blue} standalone-metastore in master has 228 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} common: The patch generated 1 new + 422 unchanged - 5
fixed = 423 total (was 427) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} standalone-metastore: The patch generated 9 new + 1945
unchanged - 13 fixed = 1954 total (was 1958) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
44s{color} | {color:red} ql: The patch generated 4 new + 380 unchanged - 4
fixed = 384 total (was 384) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
35s{color} | {color:red} root: The patch generated 14 new + 2747 unchanged - 22
fixed = 2761 total (was 2769) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m
20s{color} | {color:red} standalone-metastore generated 4 new + 226 unchanged -
2 fixed = 230 total (was 228) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
18s{color} | {color:red} ql generated 1 new + 2279 unchanged - 1 fixed = 2280
total (was 2280) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} standalone-metastore generated 0 new + 53 unchanged
- 1 fixed = 53 total (was 54) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
27s{color} | {color:green} root generated 0 new + 368 unchanged - 1 fixed = 368
total (was 369) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:standalone-metastore |
| | Dead store to numCompsWritten in
org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(CommitTxnRequest) At
TxnHandler.java:org.apache.hadoop.hive.metastore.txn.Txn
[jira] [Commented] (HIVE-19027) Make materializations invalidation cache work with multiple active remote metastores
[
https://issues.apache.org/jira/browse/HIVE-19027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525839#comment-16525839
]
Hive QA commented on HIVE-19027:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929326/HIVE-19027.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14587 tests
executed
*Failed tests:*
{noformat}
TestReplicationScenariosAcrossInstances - did not produce a TEST-*.xml file
(likely timed out) (batchId=238)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.tablesInOtherCatalogs[Embedded]
(batchId=216)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.tablesInOtherCatalogs[Remote]
(batchId=216)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12200/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12200/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12200/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929326 - PreCommit-HIVE-Build
> Make materializations invalidation cache work with multiple active remote
> metastores
>
>
> Key: HIVE-19027
> URL: https://issues.apache.org/jira/browse/HIVE-19027
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: HIVE-19027.01.patch, HIVE-19027.02.patch,
> HIVE-19027.03.patch, HIVE-19027.04.patch
>
>
> The main points:
> - Only MVs stored in transactional tables can have a time window value of 0.
> Those are the only MVs that can be guaranteed to not be outdated when a query
> is executed, if we use custom storage handlers to store the materialized
> view, we cannot make any promises.
> - For MVs that +cannot be outdated+, we do not check the metastore. Instead,
> comparison is based on valid write id lists.
> - For MVs that +can be outdated+, we still rely on the invalidation cache.
> ** The window for valid outdated MVs can be specified in intervals of 1
> minute (less than that, it is difficult to have any guarantees about whether
> the MV is actually outdated by less than a minute or not).
> ** The async loading is done every interval / 2 (or probably better, we can
> make it configurable).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20021) LLAP: Fall back to Synthetic File-ids when getting a HdfsConstants.GRANDFATHER_INODE_ID
[ https://issues.apache.org/jira/browse/HIVE-20021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20021: --- Status: Patch Available (was: Open) > LLAP: Fall back to Synthetic File-ids when getting a > HdfsConstants.GRANDFATHER_INODE_ID > --- > > Key: HIVE-20021 > URL: https://issues.apache.org/jira/browse/HIVE-20021 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20021.1.patch > > > HDFS client implementations have multiple server implementations, which do > not all support the inodes for file locations. > If the client returns a 0 InodeId, fall back to the synthetic ones. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20021) LLAP: Fall back to Synthetic File-ids when getting a HdfsConstants.GRANDFATHER_INODE_ID
[ https://issues.apache.org/jira/browse/HIVE-20021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525838#comment-16525838 ] Gopal V commented on HIVE-20021: Oddly, there is no way to actually test this with a mini-hdfs cluster, because the Apache implementation will not return this case. > LLAP: Fall back to Synthetic File-ids when getting a > HdfsConstants.GRANDFATHER_INODE_ID > --- > > Key: HIVE-20021 > URL: https://issues.apache.org/jira/browse/HIVE-20021 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20021.1.patch > > > HDFS client implementations have multiple server implementations, which do > not all support the inodes for file locations. > If the client returns a 0 InodeId, fall back to the synthetic ones. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20021) LLAP: Fall back to Synthetic File-ids when getting a HdfsConstants.GRANDFATHER_INODE_ID
[ https://issues.apache.org/jira/browse/HIVE-20021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-20021: -- Assignee: Gopal V > LLAP: Fall back to Synthetic File-ids when getting a > HdfsConstants.GRANDFATHER_INODE_ID > --- > > Key: HIVE-20021 > URL: https://issues.apache.org/jira/browse/HIVE-20021 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-20021.1.patch > > > HDFS client implementations have multiple server implementations, which do > not all support the inodes for file locations. > If the client returns a 0 InodeId, fall back to the synthetic ones. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20021) LLAP: Fall back to Synthetic File-ids when getting a HdfsConstants.GRANDFATHER_INODE_ID
[ https://issues.apache.org/jira/browse/HIVE-20021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-20021: --- Attachment: HIVE-20021.1.patch > LLAP: Fall back to Synthetic File-ids when getting a > HdfsConstants.GRANDFATHER_INODE_ID > --- > > Key: HIVE-20021 > URL: https://issues.apache.org/jira/browse/HIVE-20021 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Major > Attachments: HIVE-20021.1.patch > > > HDFS client implementations have multiple server implementations, which do > not all support the inodes for file locations. > If the client returns a 0 InodeId, fall back to the synthetic ones. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19404) Revise DDL Task Result Logging
[
https://issues.apache.org/jira/browse/HIVE-19404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525833#comment-16525833
]
Yongzhi Chen commented on HIVE-19404:
-
The change looks fine. +1
> Revise DDL Task Result Logging
> --
>
> Key: HIVE-19404
> URL: https://issues.apache.org/jira/browse/HIVE-19404
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Andrew Sherman
>Priority: Trivial
> Labels: noob
> Attachments: HIVE-19404.1.patch
>
>
> There is some logging in {{DDLTask}} that can be made better:
> {code}
> 2018-05-03 03:08:32,524 INFO hive.ql.exec.DDLTask:
> [HiveServer2-Background-Pool: Thread-101980]: results : 706
> {code}
> This logging should either be demoted to _debug_ level logging and/or
> requires additional context.
> {code}
> 2018-05-03 03:08:32,524 INFO hive.ql.exec.DDLTask:
> [HiveServer2-Background-Pool: Thread-101980]: Found 706 tables that match the
> SHOW DATABASE statement
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19532) fix tests for master-txnstats branch
[ https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19532: Attachment: HIVE-19532.13.patch > fix tests for master-txnstats branch > > > Key: HIVE-19532 > URL: https://issues.apache.org/jira/browse/HIVE-19532 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch, > HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch, > HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.06.patch, > HIVE-19532.07.patch, HIVE-19532.08.patch, HIVE-19532.09.patch, > HIVE-19532.10.patch, HIVE-19532.11.patch, HIVE-19532.12.patch, > HIVE-19532.13.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[ https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525831#comment-16525831 ] Sergey Shelukhin commented on HIVE-19532: - Fixed the conflict w/master > fix tests for master-txnstats branch > > > Key: HIVE-19532 > URL: https://issues.apache.org/jira/browse/HIVE-19532 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch, > HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch, > HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.06.patch, > HIVE-19532.07.patch, HIVE-19532.08.patch, HIVE-19532.09.patch, > HIVE-19532.10.patch, HIVE-19532.11.patch, HIVE-19532.12.patch, > HIVE-19532.13.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525791#comment-16525791
]
Hive QA commented on HIVE-19532:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929484/HIVE-19532.12.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12199/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12199/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12199/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12929484/HIVE-19532.12.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929484 - PreCommit-HIVE-Build
> fix tests for master-txnstats branch
>
>
> Key: HIVE-19532
> URL: https://issues.apache.org/jira/browse/HIVE-19532
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch,
> HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch,
> HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.06.patch,
> HIVE-19532.07.patch, HIVE-19532.08.patch, HIVE-19532.09.patch,
> HIVE-19532.10.patch, HIVE-19532.11.patch, HIVE-19532.12.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525789#comment-16525789
]
Hive QA commented on HIVE-19532:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929484/HIVE-19532.12.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12198/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12198/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12198/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-06-28 01:07:27.854
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12198/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-06-28 01:07:27.859
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
c7fe4ef..6a8f4cb master -> origin/master
cf0..648ab64 master-txnstats -> origin/master-txnstats
+ git reset --hard HEAD
HEAD is now at c7fe4ef HIVE-19969: Dependency order (dirlist) assessment fails
in yetus run (Adam Szita, reviewed by Peter Vary)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 6a8f4cb HIVE-19981: Managed tables converted to external tables
by the HiveStrictManagedMigration utility should be set to delete data when the
table is dropped (Jason Dere, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-06-28 01:07:29.249
+ rm -rf ../yetus_PreCommit-HIVE-Build-12198
+ mkdir ../yetus_PreCommit-HIVE-Build-12198
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12198
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12198/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:4092
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
with conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:1161: trailing whitespace.
explain insert into table stats_nonpartitioned select * from mysource where p
== 100;
/data/hiveptest/working/scratch/build.patch:1162: trailing whitespace.
insert into table stats_nonpartitioned select * from mysource where p == 100;
/data/hiveptest/working/scratch/build.patch:1265: trailing whitespace.
--delete from stats_part where key in (20, 41);
/data/hiveptest/working/scratch/build.patch:1369: trailing whitespace.
delete from stats_part where key in (20, 41);
/data/hiveptest/working/scratch/build.patch:1424: trailing whitespace.
explain insert into table stats_nonpartitioned select * from mysource where p
== 100;
error: patch failed:
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:4092
Falling back to three-way merge...
Applied patch to
'standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
with conflicts.
U
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
warning: squelched 865 whitespace errors
warning: 870 lines add whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12198
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929484 - PreCommit-HIVE-Build
> fix tests
[jira] [Commented] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525788#comment-16525788
]
Hive QA commented on HIVE-19850:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929304/HIVE-19850.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14609 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druid_timestamptz]
(batchId=191)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_joins]
(batchId=191)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12197/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12197/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12197/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929304 - PreCommit-HIVE-Build
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1140)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1130)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:204)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:433)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:894)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:825)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:726)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:223)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136){code}
>
> *Steps to reproduce:*
> set hive.execution.engine=tez;
> set hive.tez.dynamic.partition.pruning=true;
> CREATE TABLE t1(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t2(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
> CREATE TABLE t3(key string, value string, c_int int, c_float float, c_boolean
> boolean) partitioned by (dt string);
>
> insert into table t1 partition(dt='2018') values ('k1','v1',1,1.0,true);
> insert into table t2 partition(dt='2018') values ('k2','v2',2,2.0,true);
> insert into table t3 partition(dt='2018') values ('k3','v3',3,3.0,true);
>
> CREATE VIEW `view1` AS select
> `t1`.`key`,`t1`.`value`,`t1`.`c_int`,`t1`.`c_float`,`t1`.`c_boolean`,`t1`.`dt`
> from `t1` union all select
> `t2`.`key`,`t2`.`value`,`t2`.`c_int`,`t2`.`c_float`,`t2`.`c_
[jira] [Resolved] (HIVE-19934) Verify ACID table UPDATE/DELETE commands on transactional stats.
[ https://issues.apache.org/jira/browse/HIVE-19934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-19934. - Resolution: Fixed Fix Version/s: (was: 4.0.0) txnstats Added a simple test for variations of insert and delete. Partially covered by other new tests, this just focuses on various combinations of rows affected and all the operations.. seems like it works correctly for all cases. > Verify ACID table UPDATE/DELETE commands on transactional stats. > > > Key: HIVE-19934 > URL: https://issues.apache.org/jira/browse/HIVE-19934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: txnstats > > > ACID table UPDATE/DELETE > 1. should accurately compute table stats (if started from > accurate stats status and with hive.stats.autogather or > hive.starts.column.autogather > being true) and keep "true" for "BASIC_STATS" of COLUMN_STATS_ACCURATE. > 2. should turn off COLUMN_STATS (or turn to "false"). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20020) Hive contrib jar should not be in lib
[ https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johndee Burks updated HIVE-20020: - Description: Currently the way hive is packaged it includes hive-contrib-.jar in lib, we should not include it here because it is picked up by services like HS2. This creates a situation in which experimental features such as the [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] are accessible without understanding how to really install and use it. For example you can create a table using HS2 via beeline with the aforementioned SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not work because the SerDe is not in aux to get shipped into distcache. I propose we do not package it this way and if someone would like to leverage an experimental feature they can add it manually to their environment. (was: Currently the way hive is packaged it includes hive-contrib-.jar in lib, we should not include it here because it is picked up by services like HS2. This creates a situation in which experimental features such as the [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] is accessible without understanding how to really install and use it. For example you can create a table using HS2 via beeline with the aforementioned SerDe and it will work as long you do not do M/R jobs. The M/R jobs to do not work because the SerDe is not in aux to get shipped into distcache. I propose we do not package it this way and if someone would like to leverage an experimental feature they can add it manually to their environment. ) > Hive contrib jar should not be in lib > - > > Key: HIVE-20020 > URL: https://issues.apache.org/jira/browse/HIVE-20020 > Project: Hive > Issue Type: Improvement > Components: Contrib >Reporter: Johndee Burks >Priority: Trivial > > Currently the way hive is packaged it includes hive-contrib-.jar in > lib, we should not include it here because it is picked up by services like > HS2. This creates a situation in which experimental features such as the > [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] > are accessible without understanding how to really install and use it. For > example you can create a table using HS2 via beeline with the aforementioned > SerDe and it will work as long you do not do M/R jobs. The M/R jobs do not > work because the SerDe is not in aux to get shipped into distcache. I propose > we do not package it this way and if someone would like to leverage an > experimental feature they can add it manually to their environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19850) Dynamic partition pruning in Tez is leading to 'No work found for tablescan' error
[
https://issues.apache.org/jira/browse/HIVE-19850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525768#comment-16525768
]
Hive QA commented on HIVE-19850:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12197/dev-support/hive-personality.sh
|
| git revision | master / c7fe4ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12197/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dynamic partition pruning in Tez is leading to 'No work found for tablescan'
> error
> --
>
> Key: HIVE-19850
> URL: https://issues.apache.org/jira/browse/HIVE-19850
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.0.0
>Reporter: Ganesha Shreedhara
>Assignee: Ganesha Shreedhara
>Priority: Major
> Attachments: HIVE-19850.patch
>
>
>
> When multiple views are used along with union all, it is resulting in the
> following error when dynamic partition pruning is enabled in tez.
>
> {code:java}
> Exception in thread "main" java.lang.AssertionError: No work found for
> tablescan TS[8]
> at
> org.apache.hadoop.hive.ql.parse.GenTezUtils.processAppMasterEvent(GenTezUtils.java:408)
> at
> org.apache.hadoop.hive.ql.parse.TezCompiler.generateTaskTree(TezCompiler.java:383)
> at
> org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:205)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10371)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:208)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:239)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:479)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:347)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1203)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1257)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java
[jira] [Updated] (HIVE-20020) Hive contrib jar should not be in lib
[ https://issues.apache.org/jira/browse/HIVE-20020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johndee Burks updated HIVE-20020: - Description: Currently the way hive is packaged it includes hive-contrib-.jar in lib, we should not include it here because it is picked up by services like HS2. This creates a situation in which experimental features such as the [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] is accessible without understanding how to really install and use it. For example you can create a table using HS2 via beeline with the aforementioned SerDe and it will work as long you do not do M/R jobs. The M/R jobs to do not work because the SerDe is not in aux to get shipped into distcache. I propose we do not package it this way and if someone would like to leverage an experimental feature they can add it manually to their environment. (was: Currently the way hive is packaged it includes hive-contrib-.jar in lib, we should not include it here because it is picked up by services like HS2. This creates a situation in which experimental features such as the [*MultiDelimitSerDe|[https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java]]* is accessible without understanding how to really install and use it. For example you can create a table using HS2 via beeline with the aforementioned SerDe and it will work as long you do not do M/R jobs. The M/R jobs to do not work because the SerDe is not in aux to get shipped into distcache. I propose we do not package it this way and if someone would like to leverage an experimental feature they can add it manually to their environment. ) > Hive contrib jar should not be in lib > - > > Key: HIVE-20020 > URL: https://issues.apache.org/jira/browse/HIVE-20020 > Project: Hive > Issue Type: Improvement > Components: Contrib >Reporter: Johndee Burks >Priority: Trivial > > Currently the way hive is packaged it includes hive-contrib-.jar in > lib, we should not include it here because it is picked up by services like > HS2. This creates a situation in which experimental features such as the > [MultiDelimitSerDe|https://github.com/apache/hive/blob/master/contrib/src/java/org/apache/hadoop/hive/contrib/serde2/MultiDelimitSerDe.java] > is accessible without understanding how to really install and use it. For > example you can create a table using HS2 via beeline with the aforementioned > SerDe and it will work as long you do not do M/R jobs. The M/R jobs to do not > work because the SerDe is not in aux to get shipped into distcache. I propose > we do not package it this way and if someone would like to leverage an > experimental feature they can add it manually to their environment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20002) Shipping jdbd-storage-handler dependency jars in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525743#comment-16525743 ] Sergey Shelukhin commented on HIVE-20002: - +1 pending tests > Shipping jdbd-storage-handler dependency jars in LLAP > - > > Key: HIVE-20002 > URL: https://issues.apache.org/jira/browse/HIVE-20002 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20002.1.patch, HIVE-20002.2.patch > > > Shipping the following jars to LLAP to make jdbc storage-handler work: > commons-dbcp, commons-pool, db specific jdbc jar whichever exists in > classpath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20004) Wrong scale used by ConvertDecimal64ToDecimal results in incorrect results
[
https://issues.apache.org/jira/browse/HIVE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525740#comment-16525740
]
Hive QA commented on HIVE-20004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929346/HIVE-20004.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14611 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_5]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=59)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=169)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=133)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12196/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12196/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12196/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929346 - PreCommit-HIVE-Build
> Wrong scale used by ConvertDecimal64ToDecimal results in incorrect results
> --
>
> Key: HIVE-20004
> URL: https://issues.apache.org/jira/browse/HIVE-20004
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-20004.1.patch, HIVE-20004.2.patch
>
>
> ConvertDecimal64ToDecimal uses scale from output column vector which results
> in incorrect results.
> Input: decimal(8,1) Output: decimal(9,2)
> Input value: 963.8 gets converted to 96.38 which is wrong. The scale should
> not change this case (value should be 963.8 even after the conversion).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20013) Add an Implicit cast to date type for to_date function
[
https://issues.apache.org/jira/browse/HIVE-20013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525738#comment-16525738
]
Ashutosh Chauhan commented on HIVE-20013:
-
+1
> Add an Implicit cast to date type for to_date function
> --
>
> Key: HIVE-20013
> URL: https://issues.apache.org/jira/browse/HIVE-20013
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20013.patch
>
>
> Issue -
> SELECT TO_DATE(date1), TO_DATE(datetime1) FROM druid_table_n1;
> Running this query on Druid returns null values when date1 and datetime1 are
> of type String.
> {code}
> INFO : Executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac):
> EXPLAIN SELECT TO_DATE(datetime0) ,TO_DATE(date0) FROM calcs
> INFO : Starting task [Stage-1:EXPLAIN] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac);
> Time taken: 0.003 seconds
> INFO : OK
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1]|
> | Output:["_col0","_col1"] |
> | TableScan [TS_0] |
> |
> Output:["vc","vc0"],properties:{"druid.fieldNames":"vc,vc0","druid.fieldTypes":"date,date","druid.query.json":"{\"queryType\":\"scan\",\"dataSource\":\"druid_tableau.calcs\",\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"],\"virtualColumns\":[{\"type\":\"expression\",\"name\":\"vc\",\"expression\":\"timestamp_floor(\\\"datetime0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"},{\"type\":\"expression\",\"name\":\"vc0\",\"expression\":\"timestamp_floor(\\\"date0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"}],\"columns\":[\"vc\",\"vc0\"],\"resultFormat\":\"compactedList\"}","druid.query.type":"scan"}
> |
> ||
> ++
> 10 rows selected (0.606 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20002) Shipping jdbd-storage-handler dependency jars in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525736#comment-16525736 ] Daniel Dai commented on HIVE-20002: --- Addressing Sergey's review comments. > Shipping jdbd-storage-handler dependency jars in LLAP > - > > Key: HIVE-20002 > URL: https://issues.apache.org/jira/browse/HIVE-20002 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20002.1.patch, HIVE-20002.2.patch > > > Shipping the following jars to LLAP to make jdbc storage-handler work: > commons-dbcp, commons-pool, db specific jdbc jar whichever exists in > classpath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20002) Shipping jdbd-storage-handler dependency jars in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20002: -- Attachment: HIVE-20002.2.patch > Shipping jdbd-storage-handler dependency jars in LLAP > - > > Key: HIVE-20002 > URL: https://issues.apache.org/jira/browse/HIVE-20002 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20002.1.patch, HIVE-20002.2.patch > > > Shipping the following jars to LLAP to make jdbc storage-handler work: > commons-dbcp, commons-pool, db specific jdbc jar whichever exists in > classpath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525734#comment-16525734 ] Steve Yeom commented on HIVE-19916: --- Hi [~sershe] As we discussed during the meeting, do you think you can send an email with a "test case" for this jira issue? To [~jcamachorodriguez] and I? Thank you. Steve. > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20004) Wrong scale used by ConvertDecimal64ToDecimal results in incorrect results
[
https://issues.apache.org/jira/browse/HIVE-20004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525732#comment-16525732
]
Hive QA commented on HIVE-20004:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12196/dev-support/hive-personality.sh
|
| git revision | master / c7fe4ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12196/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Wrong scale used by ConvertDecimal64ToDecimal results in incorrect results
> --
>
> Key: HIVE-20004
> URL: https://issues.apache.org/jira/browse/HIVE-20004
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.0.1, 4.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-20004.1.patch, HIVE-20004.2.patch
>
>
> ConvertDecimal64ToDecimal uses scale from output column vector which results
> in incorrect results.
> Input: decimal(8,1) Output: decimal(9,2)
> Input value: 963.8 gets converted to 96.38 which is wrong. The scale should
> not change this case (value should be 963.8 even after the conversion).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19971) TestRuntimeStats.testCleanup() is flaky
[ https://issues.apache.org/jira/browse/HIVE-19971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-19971: -- Attachment: HIVE-19971.4.patch > TestRuntimeStats.testCleanup() is flaky > --- > > Key: HIVE-19971 > URL: https://issues.apache.org/jira/browse/HIVE-19971 > Project: Hive > Issue Type: Task >Reporter: Andrew Sherman >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-19971.1.patch, HIVE-19971.2.patch, > HIVE-19971.3.patch, HIVE-19971.4.patch > > > This test is timing dependent and sometimes fails. [You can see that it > sometimes fails in otherwise clean > runs|https://issues.apache.org/jira/issues/?jql=text%20~%20%22TestRuntimeStats%22]. > The test inserts a stat, sleeps for 2 seconds, inserts another stat, then > deletes stats that are older than 1 second. The test asserts that exactly one > stat is deleted. If the deletion is slow for some reason (perhaps a GC?) then > 2 stats will be deleted and the test will fail. The trouble is that the 1 > second window is too small to work consistently. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19939) Verify any other aggregation functions other than COUNT
[ https://issues.apache.org/jira/browse/HIVE-19939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gunther Hagleitner resolved HIVE-19939. --- Resolution: Duplicate > Verify any other aggregation functions other than COUNT > --- > > Key: HIVE-19939 > URL: https://issues.apache.org/jira/browse/HIVE-19939 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Priority: Major > > 1. More on MAX > I have added MAX query into stats_part.q and stats_nonpart.q but showed a > slightly different > explain.out which might be a bug. > 2. Other functions than MAX and COUNT. > Also I think we need to check other possible aggregation functions than MAX > and COUNT. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20010) Fix create view over literals
[
https://issues.apache.org/jira/browse/HIVE-20010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525722#comment-16525722
]
Ashutosh Chauhan commented on HIVE-20010:
-
+1
> Fix create view over literals
> -
>
> Key: HIVE-20010
> URL: https://issues.apache.org/jira/browse/HIVE-20010
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-20010.01.patch
>
>
> {code}
> create or replace view v1 as select 1 as q
> {code}
> results in an error:
> {code}
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Table already exists:
> default.v1 (state=08S01,code=1)
> {code}
> however the following works (thank you [~mgergely])
> {code}
> create or replace view v1 as select 1 as q union all select 1 as qq where
> false
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19967) SMB Join : Need Optraits for PTFOperator ala GBY Op
[ https://issues.apache.org/jira/browse/HIVE-19967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525718#comment-16525718 ] Jason Dere commented on HIVE-19967: --- +1 > SMB Join : Need Optraits for PTFOperator ala GBY Op > --- > > Key: HIVE-19967 > URL: https://issues.apache.org/jira/browse/HIVE-19967 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-19967.1.patch, HIVE-19967.2.patch, > HIVE-19967.3.patch, HIVE-19967.4.patch, HIVE-19967.5.patch, > HIVE-19967.6.patch, HIVE-19967.7.patch > > > The SMB join on one or more PTF Ops should reset the optraits keys just like > GBY Op does. > Currently there is no implementation of PTFOp optraits. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19951) Vectorization: Need to disable encoded LLAP I/O for ORC when there is data type conversion (Schema Evolution)
[ https://issues.apache.org/jira/browse/HIVE-19951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19951: Attachment: HIVE-19951.08.patch > Vectorization: Need to disable encoded LLAP I/O for ORC when there is data > type conversion (Schema Evolution) > -- > > Key: HIVE-19951 > URL: https://issues.apache.org/jira/browse/HIVE-19951 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19951.01.patch, HIVE-19951.02.patch, > HIVE-19951.03.patch, HIVE-19951.04.patch, HIVE-19951.05.patch, > HIVE-19951.06.patch, HIVE-19951.07.patch, HIVE-19951.08.patch > > > Currently, reading encoded ORC data does not support data type conversion. > So, encoded reading and cache populating needs to be disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19951) Vectorization: Need to disable encoded LLAP I/O for ORC when there is data type conversion (Schema Evolution)
[ https://issues.apache.org/jira/browse/HIVE-19951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19951: Status: Patch Available (was: In Progress) Again... > Vectorization: Need to disable encoded LLAP I/O for ORC when there is data > type conversion (Schema Evolution) > -- > > Key: HIVE-19951 > URL: https://issues.apache.org/jira/browse/HIVE-19951 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19951.01.patch, HIVE-19951.02.patch, > HIVE-19951.03.patch, HIVE-19951.04.patch, HIVE-19951.05.patch, > HIVE-19951.06.patch, HIVE-19951.07.patch, HIVE-19951.08.patch > > > Currently, reading encoded ORC data does not support data type conversion. > So, encoded reading and cache populating needs to be disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19951) Vectorization: Need to disable encoded LLAP I/O for ORC when there is data type conversion (Schema Evolution)
[ https://issues.apache.org/jira/browse/HIVE-19951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19951: Status: In Progress (was: Patch Available) > Vectorization: Need to disable encoded LLAP I/O for ORC when there is data > type conversion (Schema Evolution) > -- > > Key: HIVE-19951 > URL: https://issues.apache.org/jira/browse/HIVE-19951 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-19951.01.patch, HIVE-19951.02.patch, > HIVE-19951.03.patch, HIVE-19951.04.patch, HIVE-19951.05.patch, > HIVE-19951.06.patch, HIVE-19951.07.patch > > > Currently, reading encoded ORC data does not support data type conversion. > So, encoded reading and cache populating needs to be disabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20002) Shipping jdbd-storage-handler dependency jars in LLAP
[ https://issues.apache.org/jira/browse/HIVE-20002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525703#comment-16525703 ] Sergey Shelukhin commented on HIVE-20002: - Hmm.. this was intended to be a constant list, and non-constant things like HBase are handled separately. Perhaps instead of modifying the constant list it should add the classes in the actual code? Also nit: the Class.for-add-catch block could be refactored into a method, it's repeated a few times. > Shipping jdbd-storage-handler dependency jars in LLAP > - > > Key: HIVE-20002 > URL: https://issues.apache.org/jira/browse/HIVE-20002 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20002.1.patch > > > Shipping the following jars to LLAP to make jdbc storage-handler work: > commons-dbcp, commons-pool, db specific jdbc jar whichever exists in > classpath. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19951) Vectorization: Need to disable encoded LLAP I/O for ORC when there is data type conversion (Schema Evolution)
[
https://issues.apache.org/jira/browse/HIVE-19951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525700#comment-16525700
]
Hive QA commented on HIVE-19951:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929298/HIVE-19951.07.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14610 tests
executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=247)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=247)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=247)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=247)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12195/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12195/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12195/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929298 - PreCommit-HIVE-Build
> Vectorization: Need to disable encoded LLAP I/O for ORC when there is data
> type conversion (Schema Evolution)
> --
>
> Key: HIVE-19951
> URL: https://issues.apache.org/jira/browse/HIVE-19951
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19951.01.patch, HIVE-19951.02.patch,
> HIVE-19951.03.patch, HIVE-19951.04.patch, HIVE-19951.05.patch,
> HIVE-19951.06.patch, HIVE-19951.07.patch
>
>
> Currently, reading encoded ORC data does not support data type conversion.
> So, encoded reading and cache populating needs to be disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19932) INSERT INTO SELECT case test may be needed.
[ https://issues.apache.org/jira/browse/HIVE-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-19932. - Resolution: Fixed Newly added tests already cover insert into ... select. There's a separate jira to cover other aggregate fns > INSERT INTO SELECT case test may be needed. > --- > > Key: HIVE-19932 > URL: https://issues.apache.org/jira/browse/HIVE-19932 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Priority: Major > > INSERT INTO SELECT case test may be needed to make sure > this command is executed correctly in that next SELECT COUNT/MAX/MIN works > only with stats. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-19932) INSERT INTO SELECT case test may be needed.
[ https://issues.apache.org/jira/browse/HIVE-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-19932. - Resolution: Done > INSERT INTO SELECT case test may be needed. > --- > > Key: HIVE-19932 > URL: https://issues.apache.org/jira/browse/HIVE-19932 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Priority: Major > > INSERT INTO SELECT case test may be needed to make sure > this command is executed correctly in that next SELECT COUNT/MAX/MIN works > only with stats. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-19932) INSERT INTO SELECT case test may be needed.
[ https://issues.apache.org/jira/browse/HIVE-19932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reopened HIVE-19932: - > INSERT INTO SELECT case test may be needed. > --- > > Key: HIVE-19932 > URL: https://issues.apache.org/jira/browse/HIVE-19932 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Priority: Major > > INSERT INTO SELECT case test may be needed to make sure > this command is executed correctly in that next SELECT COUNT/MAX/MIN works > only with stats. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19934) Verify ACID table UPDATE/DELETE commands on transactional stats.
[ https://issues.apache.org/jira/browse/HIVE-19934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19934: --- Assignee: Sergey Shelukhin > Verify ACID table UPDATE/DELETE commands on transactional stats. > > > Key: HIVE-19934 > URL: https://issues.apache.org/jira/browse/HIVE-19934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > ACID table UPDATE/DELETE > 1. should accurately compute table stats (if started from > accurate stats status and with hive.stats.autogather or > hive.starts.column.autogather > being true) and keep "true" for "BASIC_STATS" of COLUMN_STATS_ACCURATE. > 2. should turn off COLUMN_STATS (or turn to "false"). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19981) Managed tables converted to external tables by the HiveStrictManagedMigration utility should be set to delete data when the table is dropped
[ https://issues.apache.org/jira/browse/HIVE-19981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525675#comment-16525675 ] Daniel Dai commented on HIVE-19981: --- +1 > Managed tables converted to external tables by the HiveStrictManagedMigration > utility should be set to delete data when the table is dropped > > > Key: HIVE-19981 > URL: https://issues.apache.org/jira/browse/HIVE-19981 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-19981.1.patch > > > Using the HiveStrictManagedMigration utility, tables can be converted to > conform to the Hive strict managed tables mode. > For managed tables that are converted to external tables by the utility, > these tables should keep the "drop data on delete" semantics they had when > they were managed tables. > One way to do this is to introduce a table property "external.table.purge", > which if true (and if the table is an external table), will let Hive know to > delete the table data when the table is dropped. This property will be set by > the HiveStrictManagedMigration utility when managed tables are converted to > external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19951) Vectorization: Need to disable encoded LLAP I/O for ORC when there is data type conversion (Schema Evolution)
[
https://issues.apache.org/jira/browse/HIVE-19951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525667#comment-16525667
]
Hive QA commented on HIVE-19951:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} llap-server in master has 83 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} llap-server: The patch generated 12 new + 30 unchanged
- 0 fixed = 42 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12195/dev-support/hive-personality.sh
|
| git revision | master / c7fe4ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12195/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: ql llap-server itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12195/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Need to disable encoded LLAP I/O for ORC when there is data
> type conversion (Schema Evolution)
> --
>
> Key: HIVE-19951
> URL: https://issues.apache.org/jira/browse/HIVE-19951
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-19951.01.patch, HIVE-19951.02.patch,
> HIVE-19951.03.patch, HIVE-19951.04.patch, HIVE-19951.05.patch,
> HIVE-19951.06.patch, HIVE-19951.07.patch
>
>
> Currently, reading encoded ORC data does not support data type conversion.
> So, encoded reading and cache populating needs to be disabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19975) Checking writeIdList per table may not check the commit level of a partition on a partitioned table.
[ https://issues.apache.org/jira/browse/HIVE-19975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525659#comment-16525659 ] Steve Yeom commented on HIVE-19975: --- [~sershe] has added test case for partition here: " Consider two scenarios: 1) Serially, with no parallel txns: Write ID 1 inserts into partition k=1, partition stats table in metastore has valid write ID list is (1). Reader tries to read, table’s valid write ID list is (1), partition k=1 list in stats table is (1), they are equivalent, all good. Write ID 2 inserts into partition k=2, partition stats table in metastore has valid write ID list is (1,2). Reader tries to read, table’s valid write ID list is (1,2). For partition k=2, the list is (1,2), equivalent returns true, all good. But for partition k=1, the list is still (1), because writer 2 doesn’t touch it. Equivalent returns false, stats cannot be used. " My answer to this test case scenarios is For #1, I have already simulated all the possible cases regarding reader’s starting point in time. It should work with the current patch. > Checking writeIdList per table may not check the commit level of a partition > on a partitioned table. > > > Key: HIVE-19975 > URL: https://issues.apache.org/jira/browse/HIVE-19975 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > writeIdList is per table entity but stats for a partitioned table are per > partition. > I.e., each record in PARTITIONS has an independent stats. > So if we check the validity of a partition's stats, we need to check in the > context of > a partiton. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[ https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525655#comment-16525655 ] Sergey Shelukhin commented on HIVE-19532: - Couple more out file updates; and rerun for TestListPartitions which I also cannot repro locally > fix tests for master-txnstats branch > > > Key: HIVE-19532 > URL: https://issues.apache.org/jira/browse/HIVE-19532 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch, > HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch, > HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.06.patch, > HIVE-19532.07.patch, HIVE-19532.08.patch, HIVE-19532.09.patch, > HIVE-19532.10.patch, HIVE-19532.11.patch, HIVE-19532.12.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525653#comment-16525653 ] Sergey Shelukhin commented on HIVE-19916: - The patch is attached as branch-... patch. Don't worry about conflicts, both patches may change in parallel, we'll resolve when it's time to commit. > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19532) fix tests for master-txnstats branch
[ https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19532: Attachment: HIVE-19532.12.patch > fix tests for master-txnstats branch > > > Key: HIVE-19532 > URL: https://issues.apache.org/jira/browse/HIVE-19532 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.2.0 > > Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch, > HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch, > HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.06.patch, > HIVE-19532.07.patch, HIVE-19532.08.patch, HIVE-19532.09.patch, > HIVE-19532.10.patch, HIVE-19532.11.patch, HIVE-19532.12.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19326) stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525649#comment-16525649
]
Hive QA commented on HIVE-19326:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929293/HIVE-19326.10.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 14611 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert]
(batchId=253)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=168)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12194/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12194/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12194/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929293 - PreCommit-HIVE-Build
> stats auto gather: incorrect aggregation during UNION queries (may lead to
> incorrect results)
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19326.01wip01.patch, HIVE-19326.02.patch,
> HIVE-19326.03.patch, HIVE-19326.04.patch, HIVE-19326.05.patch,
> HIVE-19326.06.patch, HIVE-19326.06wip01.patch, HIVE-19326.06wip02.patch,
> HIVE-19326.06wip03.patch, HIVE-19326.06wip04.patch, HIVE-19326.06wip05.patch,
> HIVE-19326.07.patch, HIVE-19326.08.patch, HIVE-19326.09.patch,
> HIVE-19326.10.patch
>
>
> Found when investigating the results change after converting tables to MM,
> turns out the MM result is correct but the current one is not.
> The test ends like so:
> {noformat}
> desc formatted small_alltypesorc_a;
> ANALYZE TABLE small_alltypesorc_a COMPUTE STATISTICS;
> desc formatted small_alltypesorc_a;
> insert into table small_alltypesorc_a select * from small_alltypesorc1a;
> desc formatted small_alltypesorc_a;
> {noformat}
> The results from the descs in the golden file are:
> {noformat}
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 5
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles1
> numRows 15
> ...
> COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\"}
> numFiles2
> numRows 20
> {noformat}
> Note the result change after analyze - the original nomRows is inaccurate,
> but BASIC_STATS is set to true.
> I am assuming with metadata only optimization this can produce incorrect
> results.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525647#comment-16525647 ] Steve Yeom commented on HIVE-19916: --- Hey [~sershe] do you think you can send me the patch you created for this jira. Let me check the changes you made. > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19934) Verify ACID table UPDATE/DELETE commands on transactional stats.
[ https://issues.apache.org/jira/browse/HIVE-19934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19934: --- Assignee: (was: Sergey Shelukhin) > Verify ACID table UPDATE/DELETE commands on transactional stats. > > > Key: HIVE-19934 > URL: https://issues.apache.org/jira/browse/HIVE-19934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Priority: Major > Fix For: 4.0.0 > > > ACID table UPDATE/DELETE > 1. should accurately compute table stats (if started from > accurate stats status and with hive.stats.autogather or > hive.starts.column.autogather > being true) and keep "true" for "BASIC_STATS" of COLUMN_STATS_ACCURATE. > 2. should turn off COLUMN_STATS (or turn to "false"). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19934) Verify ACID table UPDATE/DELETE commands on transactional stats.
[ https://issues.apache.org/jira/browse/HIVE-19934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19934: --- Assignee: Sergey Shelukhin > Verify ACID table UPDATE/DELETE commands on transactional stats. > > > Key: HIVE-19934 > URL: https://issues.apache.org/jira/browse/HIVE-19934 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 4.0.0 > > > ACID table UPDATE/DELETE > 1. should accurately compute table stats (if started from > accurate stats status and with hive.stats.autogather or > hive.starts.column.autogather > being true) and keep "true" for "BASIC_STATS" of COLUMN_STATS_ACCURATE. > 2. should turn off COLUMN_STATS (or turn to "false"). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19865) Full ACID table stats has wrong rawDataSize - phase 2
[ https://issues.apache.org/jira/browse/HIVE-19865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19865: Issue Type: Bug (was: Sub-task) Parent: (was: HIVE-19416) > Full ACID table stats has wrong rawDataSize - phase 2 > - > > Key: HIVE-19865 > URL: https://issues.apache.org/jira/browse/HIVE-19865 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 4.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525631#comment-16525631 ] Sergey Shelukhin commented on HIVE-19916: - Attached the master patch (whole branch + this patch) for HiveQA. If we see actual failures in optimizer, then those would need to be fixed (because getting writeIds twice with the current double compile scheme where optimizers run outside of lock could mean incorrect results). If some DDL paths fail then we'd postpone... if none fail (compared to the branch itself) we'd commit. > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525629#comment-16525629 ] Sergey Shelukhin commented on HIVE-19916: - Unfortunately it's a pain to remove this call because the methods like alterTable, etc are called from many places. It would have been better to never get it from metastore, but to require all the callers to make sure write IDs are available, so compile paths wouldn't need to do anything (they already get writeIDs), and the other paths would need to initialize config, that would have been easy to detect in early testing. In the interest of time now I only removed metastore call from some obvious paths. If this breaks too many tests we might postpone this. > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19916: Status: Patch Available (was: Open) > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19916: Summary: master-txnstats branch - don't get write IDs from metastore when it's not safe (was: master-txnstats branch - integrate with HIVE-19382) > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19916: Attachment: branch-19916.patch > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19916) master-txnstats branch - don't get write IDs from metastore when it's not safe
[ https://issues.apache.org/jira/browse/HIVE-19916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19916: Attachment: HIVE-19916.patch > master-txnstats branch - don't get write IDs from metastore when it's not safe > -- > > Key: HIVE-19916 > URL: https://issues.apache.org/jira/browse/HIVE-19916 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19916.patch, branch-19916.patch > > > There's some code in original txn stats patch that may go to metastore to get > write Ids. This code should not go to metastore, it should fail instead. > HIVE-19382 should ensure that we have correct IDs already present during > optimizer - they are using by e.g. materialized view optimizer, so they > should be there; if they are not present, some integration might be needed so > that txn stats optimizations also have access to those write Ids. > cc [~jcamachorodriguez] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-20019: Assignee: Prasanth Jayachandran > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19326) stats auto gather: incorrect aggregation during UNION queries (may lead to incorrect results)
[
https://issues.apache.org/jira/browse/HIVE-19326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525608#comment-16525608
]
Hive QA commented on HIVE-19326:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} itests/util in master has 52 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} ql: The patch generated 0 new + 259 unchanged - 2
fixed = 259 total (was 261) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch util passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-12194/dev-support/hive-personality.sh
|
| git revision | master / c7fe4ef |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12194/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12194/yetus/whitespace-tabs.txt
|
| modules | C: ql itests itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-12194/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> stats auto gather: incorrect aggregation during UNION queries (may lead to
> incorrect results)
> -
>
> Key: HIVE-19326
> URL: https://issues.apache.org/jira/browse/HIVE-19326
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Sergey Shelukhin
>Assignee: Zoltan Haindrich
>Priority: Critical
> Attachments: HIVE-19326.01wip01.patch, HIVE-19326.02.patch,
> HIVE-19326.03.patch,
[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings
[
https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HIVE-19668:
--
Status: Patch Available (was: In Progress)
> Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and
> duplicate strings
> --
>
> Key: HIVE-19668
> URL: https://issues.apache.org/jira/browse/HIVE-19668
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-19668.01.patch, HIVE-19668.02.patch,
> image-2018-05-22-17-41-39-572.png
>
>
> I've recently analyzed a HS2 heap dump, obtained when there was a huge memory
> spike during compilation of some big query. The analysis was done with jxray
> ([www.jxray.com).|http://www.jxray.com)./] It turns out that more than 90% of
> the 20G heap was used by data structures associated with query parsing
> ({{org.apache.hadoop.hive.ql.parse.QBExpr}}). There are probably multiple
> opportunities for optimizations here. One of them is to stop the code from
> creating duplicate instances of {{org.antlr.runtime.CommonToken}} class. See
> a sample of these objects in the attached image:
> !image-2018-05-22-17-41-39-572.png|width=879,height=399!
> Looks like these particular {{CommonToken}} objects are constants, that don't
> change once created. I see some code, e.g. in
> {{org.apache.hadoop.hive.ql.parse.CalcitePlanner}}, where such objects are
> apparently repeatedly created with e.g. {{new
> CommonToken(HiveParser.TOK_INSERT, "TOK_INSERT")}} If these 33 token kinds
> are instead created once and reused, we will save more than 1/10th of the
> heap in this scenario. Plus, since these objects are small but very numerous,
> getting rid of them will remove a gread deal of pressure from the GC.
> Another source of waste are duplicate strings, that collectively waste 26.1%
> of memory. Some of them come from CommonToken objects that have the same text
> (i.e. for multiple CommonToken objects the contents of their 'text' Strings
> are the same, but each has its own copy of that String). Other duplicate
> strings come from other sources, that are easy enough to fix by adding
> String.intern() calls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings
[
https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HIVE-19668:
--
Attachment: HIVE-19668.02.patch
> Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and
> duplicate strings
> --
>
> Key: HIVE-19668
> URL: https://issues.apache.org/jira/browse/HIVE-19668
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-19668.01.patch, HIVE-19668.02.patch,
> image-2018-05-22-17-41-39-572.png
>
>
> I've recently analyzed a HS2 heap dump, obtained when there was a huge memory
> spike during compilation of some big query. The analysis was done with jxray
> ([www.jxray.com).|http://www.jxray.com)./] It turns out that more than 90% of
> the 20G heap was used by data structures associated with query parsing
> ({{org.apache.hadoop.hive.ql.parse.QBExpr}}). There are probably multiple
> opportunities for optimizations here. One of them is to stop the code from
> creating duplicate instances of {{org.antlr.runtime.CommonToken}} class. See
> a sample of these objects in the attached image:
> !image-2018-05-22-17-41-39-572.png|width=879,height=399!
> Looks like these particular {{CommonToken}} objects are constants, that don't
> change once created. I see some code, e.g. in
> {{org.apache.hadoop.hive.ql.parse.CalcitePlanner}}, where such objects are
> apparently repeatedly created with e.g. {{new
> CommonToken(HiveParser.TOK_INSERT, "TOK_INSERT")}} If these 33 token kinds
> are instead created once and reused, we will save more than 1/10th of the
> heap in this scenario. Plus, since these objects are small but very numerous,
> getting rid of them will remove a gread deal of pressure from the GC.
> Another source of waste are duplicate strings, that collectively waste 26.1%
> of memory. Some of them come from CommonToken objects that have the same text
> (i.e. for multiple CommonToken objects the contents of their 'text' Strings
> are the same, but each has its own copy of that String). Other duplicate
> strings come from other sources, that are easy enough to fix by adding
> String.intern() calls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings
[
https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misha Dmitriev updated HIVE-19668:
--
Status: In Progress (was: Patch Available)
> Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and
> duplicate strings
> --
>
> Key: HIVE-19668
> URL: https://issues.apache.org/jira/browse/HIVE-19668
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-19668.01.patch, image-2018-05-22-17-41-39-572.png
>
>
> I've recently analyzed a HS2 heap dump, obtained when there was a huge memory
> spike during compilation of some big query. The analysis was done with jxray
> ([www.jxray.com).|http://www.jxray.com)./] It turns out that more than 90% of
> the 20G heap was used by data structures associated with query parsing
> ({{org.apache.hadoop.hive.ql.parse.QBExpr}}). There are probably multiple
> opportunities for optimizations here. One of them is to stop the code from
> creating duplicate instances of {{org.antlr.runtime.CommonToken}} class. See
> a sample of these objects in the attached image:
> !image-2018-05-22-17-41-39-572.png|width=879,height=399!
> Looks like these particular {{CommonToken}} objects are constants, that don't
> change once created. I see some code, e.g. in
> {{org.apache.hadoop.hive.ql.parse.CalcitePlanner}}, where such objects are
> apparently repeatedly created with e.g. {{new
> CommonToken(HiveParser.TOK_INSERT, "TOK_INSERT")}} If these 33 token kinds
> are instead created once and reused, we will save more than 1/10th of the
> heap in this scenario. Plus, since these objects are small but very numerous,
> getting rid of them will remove a gread deal of pressure from the GC.
> Another source of waste are duplicate strings, that collectively waste 26.1%
> of memory. Some of them come from CommonToken objects that have the same text
> (i.e. for multiple CommonToken objects the contents of their 'text' Strings
> are the same, but each has its own copy of that String). Other duplicate
> strings come from other sources, that are easy enough to fix by adding
> String.intern() calls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20019) Remove commons-logging and move to slf4j
[ https://issues.apache.org/jira/browse/HIVE-20019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-20019: - Issue Type: Sub-task (was: Improvement) Parent: HIVE-19640 > Remove commons-logging and move to slf4j > > > Key: HIVE-20019 > URL: https://issues.apache.org/jira/browse/HIVE-20019 > Project: Hive > Issue Type: Sub-task > Components: Logging >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Priority: Major > > Still seeing several references to commons-logging. We should move all > classes to slf4j instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19990) Query with interval literal in join condition fails
[
https://issues.apache.org/jira/browse/HIVE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19990:
---
Status: Open (was: Patch Available)
> Query with interval literal in join condition fails
> ---
>
> Key: HIVE-19990
> URL: https://issues.apache.org/jira/browse/HIVE-19990
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19990.1.patch, HIVE-19990.2.patch
>
>
> *Reproducer*
> {code:sql}
> > create table date_dim_d1(
> d_week_seqint,
> d_datestring);
> > SELECT
>d1.d_week_seq
> FROM
>date_dim_d1 d1
>JOIN date_dim_d1 d3
> WHERE
>Cast(d3.d_date AS date) > Cast(d1.d_date AS date) + INTERVAL '5' day ;
> {code}
> *Exception*
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: '5 00:00:00.0'
> encountered with 0 children
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2780)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2775)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:3060)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2959)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:9633)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11380)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11285)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:593)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12150)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:658)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1829)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1776)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1771)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:832)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:770)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:694)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19990) Query with interval literal in join condition fails
[
https://issues.apache.org/jira/browse/HIVE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19990:
---
Attachment: HIVE-19990.2.patch
> Query with interval literal in join condition fails
> ---
>
> Key: HIVE-19990
> URL: https://issues.apache.org/jira/browse/HIVE-19990
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19990.1.patch, HIVE-19990.2.patch
>
>
> *Reproducer*
> {code:sql}
> > create table date_dim_d1(
> d_week_seqint,
> d_datestring);
> > SELECT
>d1.d_week_seq
> FROM
>date_dim_d1 d1
>JOIN date_dim_d1 d3
> WHERE
>Cast(d3.d_date AS date) > Cast(d1.d_date AS date) + INTERVAL '5' day ;
> {code}
> *Exception*
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: '5 00:00:00.0'
> encountered with 0 children
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2780)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2775)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:3060)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2959)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:9633)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11380)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11285)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:593)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12150)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:658)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1829)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1776)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1771)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:832)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:770)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:694)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19990) Query with interval literal in join condition fails
[
https://issues.apache.org/jira/browse/HIVE-19990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19990:
---
Status: Patch Available (was: Open)
Test failures are not reproducible (looks flaky). Uploading rebased patch.
> Query with interval literal in join condition fails
> ---
>
> Key: HIVE-19990
> URL: https://issues.apache.org/jira/browse/HIVE-19990
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19990.1.patch, HIVE-19990.2.patch
>
>
> *Reproducer*
> {code:sql}
> > create table date_dim_d1(
> d_week_seqint,
> d_datestring);
> > SELECT
>d1.d_week_seq
> FROM
>date_dim_d1 d1
>JOIN date_dim_d1 d3
> WHERE
>Cast(d3.d_date AS date) > Cast(d1.d_date AS date) + INTERVAL '5' day ;
> {code}
> *Exception*
> {code}
> org.apache.hadoop.hive.ql.parse.SemanticException: '5 00:00:00.0'
> encountered with 0 children
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2780)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondPopulateAlias(SemanticAnalyzer.java:2775)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:3060)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.parseJoinCondition(SemanticAnalyzer.java:2959)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genJoinTree(SemanticAnalyzer.java:9633)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11380)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11285)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:12071)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:593)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12150)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:330)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:288)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:658)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1829)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1776)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1771)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:832)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:770)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:694)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19989) Metastore uses wrong application name for HADOOP2 metrics
[ https://issues.apache.org/jira/browse/HIVE-19989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19989: --- Status: Open (was: Patch Available) > Metastore uses wrong application name for HADOOP2 metrics > - > > Key: HIVE-19989 > URL: https://issues.apache.org/jira/browse/HIVE-19989 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-19989.1.patch, HIVE-19989.2.patch > > > Right now it is hardcoded as 'metastore'. It should instead be fetched from > config like it was previously. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19989) Metastore uses wrong application name for HADOOP2 metrics
[ https://issues.apache.org/jira/browse/HIVE-19989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19989: --- Status: Patch Available (was: Open) Druid failures are flaky and not related. Rebased patch. > Metastore uses wrong application name for HADOOP2 metrics > - > > Key: HIVE-19989 > URL: https://issues.apache.org/jira/browse/HIVE-19989 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-19989.1.patch, HIVE-19989.2.patch > > > Right now it is hardcoded as 'metastore'. It should instead be fetched from > config like it was previously. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19989) Metastore uses wrong application name for HADOOP2 metrics
[ https://issues.apache.org/jira/browse/HIVE-19989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19989: --- Attachment: HIVE-19989.2.patch > Metastore uses wrong application name for HADOOP2 metrics > - > > Key: HIVE-19989 > URL: https://issues.apache.org/jira/browse/HIVE-19989 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-19989.1.patch, HIVE-19989.2.patch > > > Right now it is hardcoded as 'metastore'. It should instead be fetched from > config like it was previously. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18545) Add UDF to parse complex types from json
[
https://issues.apache.org/jira/browse/HIVE-18545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525558#comment-16525558
]
Hive QA commented on HIVE-18545:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929292/HIVE-18545.06.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14624 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12193/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12193/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12193/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929292 - PreCommit-HIVE-Build
> Add UDF to parse complex types from json
>
>
> Key: HIVE-18545
> URL: https://issues.apache.org/jira/browse/HIVE-18545
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18545.02.patch, HIVE-18545.03.patch,
> HIVE-18545.04.patch, HIVE-18545.05.patch, HIVE-18545.06.patch,
> HIVE-18545.06.patch, HIVE-18545.06.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18545) Add UDF to parse complex types from json
[
https://issues.apache.org/jira/browse/HIVE-18545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525559#comment-16525559
]
Hive QA commented on HIVE-18545:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 190 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2280 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} hcatalog/core in master has 33 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} streaming in master has 2 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 13m
4s{color} | {color:red} branch/itests/hive-jmh cannot run convertXmlToText from
findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} serde: The patch generated 23 new + 6 unchanged - 385
fixed = 29 total (was 391) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 11 new + 71 unchanged - 0
fixed = 82 total (was 71) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} hcatalog/core: The patch generated 0 new + 10
unchanged - 31 fixed = 10 total (was 41) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch streaming passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} itests/hive-jmh: The patch generated 7 new + 0
unchanged - 0 fixed = 7 total (was 0) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 3 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
50s{color} | {color:red} serde generated 4 new + 187 unchanged - 3 fixed = 191
total (was 190) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} ql in the patch passed. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} hcatalog/core generated 0 new + 31 unchanged - 2
fixed = 31 total (was 33) {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} streaming in the patch passed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 14m
15s{color} | {color:red} patch/itests/hive-jmh cannot run convertXmlToText from
findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color
[jira] [Updated] (HIVE-19820) add ACID stats support to background stats updater
[ https://issues.apache.org/jira/browse/HIVE-19820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19820: Attachment: HIVE-19820.03-master-txnstats.patch > add ACID stats support to background stats updater > -- > > Key: HIVE-19820 > URL: https://issues.apache.org/jira/browse/HIVE-19820 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19820.01-master-txnstats.patch, > HIVE-19820.02-master-txnstats.patch, HIVE-19820.03-master-txnstats.patch > > > Follow-up from HIVE-19418. > Right now it checks whether stats are valid in an old-fashioned way... and > also gets ACID state, and discards it without using. > When ACID stats are implemented, ACID state needs to be used to do > version-aware valid stats checks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20014) Druid SECOND/HOUR/MINUTE does not return correct values when applied to String Columns
[
https://issues.apache.org/jira/browse/HIVE-20014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa resolved HIVE-20014.
-
Resolution: Invalid
Verified that this is the same behavior with orc tables.
This is an expected behavior change in 3.0
> Druid SECOND/HOUR/MINUTE does not return correct values when applied to
> String Columns
> --
>
> Key: HIVE-20014
> URL: https://issues.apache.org/jira/browse/HIVE-20014
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
>
> Query SELECT MINUTE(`time1`) FROM calcs; returns null when the String column
> only contains timestamp and does not contain any date information in the
> column. The Druid parser fails to parse the time string values and returns
> null.
> {code}
> 1: jdbc:hive2://ctr-e138-1518143905142-379982> SELECT MINUTE(`time1`) FROM
> calcs;
> INFO : Compiling
> command(queryId=hive_20180627145215_05147329-b8d8-491c-9bab-6fd5045542db):
> SELECT MINUTE(`time1`) FROM calcs
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:vc,
> type:int, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20180627145215_05147329-b8d8-491c-9bab-6fd5045542db);
> Time taken: 0.134 seconds
> INFO : Executing
> command(queryId=hive_20180627145215_05147329-b8d8-491c-9bab-6fd5045542db):
> SELECT MINUTE(`time1`) FROM calcs
> INFO : Completed executing
> command(queryId=hive_20180627145215_05147329-b8d8-491c-9bab-6fd5045542db);
> Time taken: 0.002 seconds
> INFO : OK
> +---+
> | vc |
> +---+
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> | NULL |
> +---+
> 17 rows selected (0.266 seconds)
> 1: jdbc:hive2://ctr-e138-1518143905142-379982> SELECT time1 from calcs;
> INFO : Compiling
> command(queryId=hive_20180627145225_93b872de-a698-4859-9730-983eede6935d):
> SELECT time1 from calcs
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:time1,
> type:string, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20180627145225_93b872de-a698-4859-9730-983eede6935d);
> Time taken: 0.116 seconds
> INFO : Executing
> command(queryId=hive_20180627145225_93b872de-a698-4859-9730-983eede6935d):
> SELECT time1 from calcs
> INFO : Completed executing
> command(queryId=hive_20180627145225_93b872de-a698-4859-9730-983eede6935d);
> Time taken: 0.003 seconds
> INFO : OK
> +---+
> | time1 |
> +---+
> | 22:20:14 |
> | 22:50:16 |
> | 19:36:22 |
> | 19:48:23 |
> | 00:05:57 |
> | NULL |
> | 04:48:07 |
> | NULL |
> | 19:57:33 |
> | NULL |
> | 04:40:49 |
> | 02:05:25 |
> | NULL |
> | NULL |
> | 12:33:57 |
> | 18:58:41 |
> | 09:33:31 |
> +---+
> 17 rows selected (0.202 seconds)
> 1: jdbc:hive2://ctr-e138-1518143905142-379982> EXPLAIN SELECT
> MINUTE(`time1`) FROM calcs;
> INFO : Compiling
> command(queryId=hive_20180627145237_39e53a7e-35cb-4e17-8ccb-884c6f6358cd):
> EXPLAIN SELECT MINUTE(`time1`) FROM calcs
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:Explain,
> type:string, comment:null)], properties:null)
> INFO : Completed compiling
> command(queryId=hive_20180627145237_39e53a7e-35cb-4e17-8ccb-884c6f6358cd);
> Time taken: 0.107 seconds
> INFO : Executing
> command(queryId=hive_20180627145237_39e53a7e-35cb-4e17-8ccb-884c6f6358cd):
> EXPLAIN SELECT MINUTE(`time1`) FROM calcs
> INFO : Starting task [Stage-1:EXPLAIN] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180627145237_39e53a7e-35cb-4e17-8ccb-884c6f6358cd);
> Time taken: 0.003 seconds
> INFO : OK
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1]|
> | Output:["_col0"] |
> | TableScan [TS_0] |
> |
> Output:["vc"],properties:{"druid.fieldNames":"vc","druid.fieldTypes":"int","druid.query.json":"{\"queryType\":\"scan\",\"dataSo
[jira] [Commented] (HIVE-20007) Hive should carry out timestamp computations in UTC
[
https://issues.apache.org/jira/browse/HIVE-20007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525488#comment-16525488
]
Xuefu Zhang commented on HIVE-20007:
cc: [~lirui]
> Hive should carry out timestamp computations in UTC
> ---
>
> Key: HIVE-20007
> URL: https://issues.apache.org/jira/browse/HIVE-20007
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Reporter: Ryan Blue
>Assignee: Jesus Camacho Rodriguez
>Priority: Blocker
> Labels: timestamp
> Attachments: HIVE-20007.patch
>
>
> Hive currently uses the "local" time of a java.sql.Timestamp to represent the
> SQL data type TIMESTAMP WITHOUT TIME ZONE. The purpose is to be able to use
> {{Timestamp#getYear()}} and similar methods to implement SQL functions like
> {{year}}.
> When the SQL session's time zone is a DST zone, such as America/Los_Angeles
> that alternates between PST and PDT, there are times that cannot be
> represented because the effective zone skips them.
> {code}
> hive> select TIMESTAMP '2015-03-08 02:10:00.101';
> 2015-03-08 03:10:00.101
> {code}
> Using UTC instead of the SQL session time zone as the underlying zone for a
> java.sql.Timestamp avoids this bug, while still returning correct values for
> {{getYear}} etc. Using UTC as the convenience representation (timestamp
> without time zone has no real zone) would make timestamp calculations more
> consistent and avoid similar problems in the future.
> Notably, this would break the {{unix_timestamp}} UDF that specifies the
> result is with respect to ["the default timezone and default
> locale"|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-DateFunctions].
> That function would need to be updated to use the
> {{System.getProperty("user.timezone")}} zone.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19668) Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and duplicate strings
[
https://issues.apache.org/jira/browse/HIVE-19668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525477#comment-16525477
]
Aihua Xu commented on HIVE-19668:
-
[~mi...@cloudera.com] The patch looks good to me. There are some style issues
like missing apache header not from your change. Can you fix those issues?
> Over 30% of the heap wasted by duplicate org.antlr.runtime.CommonToken's and
> duplicate strings
> --
>
> Key: HIVE-19668
> URL: https://issues.apache.org/jira/browse/HIVE-19668
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Affects Versions: 3.0.0
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>Priority: Major
> Attachments: HIVE-19668.01.patch, image-2018-05-22-17-41-39-572.png
>
>
> I've recently analyzed a HS2 heap dump, obtained when there was a huge memory
> spike during compilation of some big query. The analysis was done with jxray
> ([www.jxray.com).|http://www.jxray.com)./] It turns out that more than 90% of
> the 20G heap was used by data structures associated with query parsing
> ({{org.apache.hadoop.hive.ql.parse.QBExpr}}). There are probably multiple
> opportunities for optimizations here. One of them is to stop the code from
> creating duplicate instances of {{org.antlr.runtime.CommonToken}} class. See
> a sample of these objects in the attached image:
> !image-2018-05-22-17-41-39-572.png|width=879,height=399!
> Looks like these particular {{CommonToken}} objects are constants, that don't
> change once created. I see some code, e.g. in
> {{org.apache.hadoop.hive.ql.parse.CalcitePlanner}}, where such objects are
> apparently repeatedly created with e.g. {{new
> CommonToken(HiveParser.TOK_INSERT, "TOK_INSERT")}} If these 33 token kinds
> are instead created once and reused, we will save more than 1/10th of the
> heap in this scenario. Plus, since these objects are small but very numerous,
> getting rid of them will remove a gread deal of pressure from the GC.
> Another source of waste are duplicate strings, that collectively waste 26.1%
> of memory. Some of them come from CommonToken objects that have the same text
> (i.e. for multiple CommonToken objects the contents of their 'text' Strings
> are the same, but each has its own copy of that String). Other duplicate
> strings come from other sources, that are easy enough to fix by adding
> String.intern() calls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19995) Aggregate row traffic for acid tables
[
https://issues.apache.org/jira/browse/HIVE-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525474#comment-16525474
]
Hive QA commented on HIVE-19995:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12929436/HIVE-19995.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 14610 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=81)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv]
(batchId=189)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_into_default_keyword]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_3]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_transactional]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12192/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12192/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12192/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12929436 - PreCommit-HIVE-Build
> Aggregate row traffic for acid tables
> -
>
> Key: HIVE-19995
> URL: https://issues.apache.org/jira/browse/HIVE-19995
> Project: Hive
> Issue Type: Sub-task
> Components: Statistics, Transactions
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-19995.01.patch, HIVE-19995.01wip01.patch,
> HIVE-19995.01wip01.patch, HIVE-19995.01wip02.patch
>
>
> for transactional tables we store basic stats in case of explicit
> analyze/rewrite; but doesn't do anything in other caseswhich may even
> lead to plans which oom...
> It would be better to aggregate the total row traffic...because that is
> already available; so that operator tree estimations could work with a real
> upper bound of the row numbers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19988) Precommit jobs erroring out
[
https://issues.apache.org/jira/browse/HIVE-19988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar resolved HIVE-19988.
Resolution: Duplicate
> Precommit jobs erroring out
> ---
>
> Key: HIVE-19988
> URL: https://issues.apache.org/jira/browse/HIVE-19988
> Project: Hive
> Issue Type: Bug
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Blocker
>
> {code}
> + mvn clean package -B -DskipTests -Drat.numUnapprovedLicenses=1000
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/.m2/repository
> [INFO] Scanning for projects...
> [INFO]
> [INFO]
>
> [INFO] Building hive-ptest 3.0
> [INFO]
>
> [INFO] Downloading from central:
> https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-clean-plugin/2.5/maven-clean-plugin-2.5.pom
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 0.925 s
> [INFO] Finished at: 2018-06-25T20:46:27Z
> [INFO] Final Memory: 24M/1447M
> [INFO]
>
> [ERROR] Plugin org.apache.maven.plugins:maven-clean-plugin:2.5 or one of its
> dependencies could not be resolved: Failed to read artifact descriptor for
> org.apache.maven.plugins:maven-clean-plugin:jar:2.5: Could not transfer
> artifact org.apache.maven.plugins:maven-clean-plugin:pom:2.5 from/to central
> (https://repo.maven.apache.org/maven2): Received fatal alert:
> protocol_version -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/PluginResolutionException
> + return 1
> + ret=1
> + unpack_test_results
> + '[' -z
> /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build ']'
> + cd
> /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target
> jenkins-execute-build.sh: line 61: cd:
> /home/jenkins/jenkins-slave/workspace/PreCommit-HIVE-Build/hive/build/hive/testutils/ptest2/target:
> No such file or directory
> + [[ -f test-results.tar.gz ]]
> + exit 1
> + rm -f /tmp/tmp.LFKzzyYwIt
> Build step 'Execute shell' marked build as failure
> Recording test results
> ERROR: Step ?Publish JUnit test result report? failed: No test report files
> were found. Configuration error?
> [description-setter] Description set: HIVE-19980 / master-mr2
> Finished: FAILURE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[ https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19860: --- Assignee: Sergey Shelukhin (was: Rajkumar Singh) > HiveServer2 ObjectInspectorFactory memory leak with > cachedUnionStructObjectInspector > > > Key: HIVE-19860 > URL: https://issues.apache.org/jira/browse/HIVE-19860 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 > Environment: hiveserver2 Interactive with LLAP. >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19860.01.patch, HIVE-19860.patch, Screen Shot > 2018-06-11 at 2.01.00 PM.png > > > hiveserver2 is start seeing the memory pressure once the > cachedUnionStructObjectInspector start going > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345] > I did not see any eviction policy for cachedUnionStructObjectInspector, so we > should implement some size or time-based eviction policy. > !Screen Shot 2018-06-11 at 2.01.00 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[ https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525470#comment-16525470 ] Sergey Shelukhin commented on HIVE-19860: - Rebased the patch > HiveServer2 ObjectInspectorFactory memory leak with > cachedUnionStructObjectInspector > > > Key: HIVE-19860 > URL: https://issues.apache.org/jira/browse/HIVE-19860 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 > Environment: hiveserver2 Interactive with LLAP. >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-19860.01.patch, HIVE-19860.patch, Screen Shot > 2018-06-11 at 2.01.00 PM.png > > > hiveserver2 is start seeing the memory pressure once the > cachedUnionStructObjectInspector start going > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345] > I did not see any eviction policy for cachedUnionStructObjectInspector, so we > should implement some size or time-based eviction policy. > !Screen Shot 2018-06-11 at 2.01.00 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[ https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19860: --- Assignee: Rajkumar Singh (was: Sergey Shelukhin) > HiveServer2 ObjectInspectorFactory memory leak with > cachedUnionStructObjectInspector > > > Key: HIVE-19860 > URL: https://issues.apache.org/jira/browse/HIVE-19860 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 > Environment: hiveserver2 Interactive with LLAP. >Reporter: Rajkumar Singh >Assignee: Rajkumar Singh >Priority: Major > Attachments: HIVE-19860.01.patch, HIVE-19860.patch, Screen Shot > 2018-06-11 at 2.01.00 PM.png > > > hiveserver2 is start seeing the memory pressure once the > cachedUnionStructObjectInspector start going > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345] > I did not see any eviction policy for cachedUnionStructObjectInspector, so we > should implement some size or time-based eviction policy. > !Screen Shot 2018-06-11 at 2.01.00 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19860) HiveServer2 ObjectInspectorFactory memory leak with cachedUnionStructObjectInspector
[ https://issues.apache.org/jira/browse/HIVE-19860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19860: Attachment: HIVE-19860.01.patch > HiveServer2 ObjectInspectorFactory memory leak with > cachedUnionStructObjectInspector > > > Key: HIVE-19860 > URL: https://issues.apache.org/jira/browse/HIVE-19860 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.0 > Environment: hiveserver2 Interactive with LLAP. >Reporter: Rajkumar Singh >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19860.01.patch, HIVE-19860.patch, Screen Shot > 2018-06-11 at 2.01.00 PM.png > > > hiveserver2 is start seeing the memory pressure once the > cachedUnionStructObjectInspector start going > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L345] > I did not see any eviction policy for cachedUnionStructObjectInspector, so we > should implement some size or time-based eviction policy. > !Screen Shot 2018-06-11 at 2.01.00 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20013) Add an Implicit cast to date type for to_date function
[
https://issues.apache.org/jira/browse/HIVE-20013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20013:
Status: Patch Available (was: Open)
> Add an Implicit cast to date type for to_date function
> --
>
> Key: HIVE-20013
> URL: https://issues.apache.org/jira/browse/HIVE-20013
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20013.patch
>
>
> Issue -
> SELECT TO_DATE(date1), TO_DATE(datetime1) FROM druid_table_n1;
> Running this query on Druid returns null values when date1 and datetime1 are
> of type String.
> {code}
> INFO : Executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac):
> EXPLAIN SELECT TO_DATE(datetime0) ,TO_DATE(date0) FROM calcs
> INFO : Starting task [Stage-1:EXPLAIN] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac);
> Time taken: 0.003 seconds
> INFO : OK
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1]|
> | Output:["_col0","_col1"] |
> | TableScan [TS_0] |
> |
> Output:["vc","vc0"],properties:{"druid.fieldNames":"vc,vc0","druid.fieldTypes":"date,date","druid.query.json":"{\"queryType\":\"scan\",\"dataSource\":\"druid_tableau.calcs\",\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"],\"virtualColumns\":[{\"type\":\"expression\",\"name\":\"vc\",\"expression\":\"timestamp_floor(\\\"datetime0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"},{\"type\":\"expression\",\"name\":\"vc0\",\"expression\":\"timestamp_floor(\\\"date0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"}],\"columns\":[\"vc\",\"vc0\"],\"resultFormat\":\"compactedList\"}","druid.query.type":"scan"}
> |
> ||
> ++
> 10 rows selected (0.606 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-20013) Add an Implicit cast to date type for to_date function
[
https://issues.apache.org/jira/browse/HIVE-20013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525464#comment-16525464
]
Nishant Bangarwa commented on HIVE-20013:
-
+cc [~ashutoshc] please review.
> Add an Implicit cast to date type for to_date function
> --
>
> Key: HIVE-20013
> URL: https://issues.apache.org/jira/browse/HIVE-20013
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20013.patch
>
>
> Issue -
> SELECT TO_DATE(date1), TO_DATE(datetime1) FROM druid_table_n1;
> Running this query on Druid returns null values when date1 and datetime1 are
> of type String.
> {code}
> INFO : Executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac):
> EXPLAIN SELECT TO_DATE(datetime0) ,TO_DATE(date0) FROM calcs
> INFO : Starting task [Stage-1:EXPLAIN] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac);
> Time taken: 0.003 seconds
> INFO : OK
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1]|
> | Output:["_col0","_col1"] |
> | TableScan [TS_0] |
> |
> Output:["vc","vc0"],properties:{"druid.fieldNames":"vc,vc0","druid.fieldTypes":"date,date","druid.query.json":"{\"queryType\":\"scan\",\"dataSource\":\"druid_tableau.calcs\",\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"],\"virtualColumns\":[{\"type\":\"expression\",\"name\":\"vc\",\"expression\":\"timestamp_floor(\\\"datetime0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"},{\"type\":\"expression\",\"name\":\"vc0\",\"expression\":\"timestamp_floor(\\\"date0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"}],\"columns\":[\"vc\",\"vc0\"],\"resultFormat\":\"compactedList\"}","druid.query.type":"scan"}
> |
> ||
> ++
> 10 rows selected (0.606 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20013) Add an Implicit cast to date type for to_date function
[
https://issues.apache.org/jira/browse/HIVE-20013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-20013:
Attachment: HIVE-20013.patch
> Add an Implicit cast to date type for to_date function
> --
>
> Key: HIVE-20013
> URL: https://issues.apache.org/jira/browse/HIVE-20013
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-20013.patch
>
>
> Issue -
> SELECT TO_DATE(date1), TO_DATE(datetime1) FROM druid_table_n1;
> Running this query on Druid returns null values when date1 and datetime1 are
> of type String.
> {code}
> INFO : Executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac):
> EXPLAIN SELECT TO_DATE(datetime0) ,TO_DATE(date0) FROM calcs
> INFO : Starting task [Stage-1:EXPLAIN] in serial mode
> INFO : Completed executing
> command(queryId=hive_20180627144822_d4395567-e3cb-4b20-b53b-4e5eba2d7dac);
> Time taken: 0.003 seconds
> INFO : OK
> ++
> | Explain |
> ++
> | Plan optimized by CBO. |
> ||
> | Stage-0|
> | Fetch Operator |
> | limit:-1 |
> | Select Operator [SEL_1]|
> | Output:["_col0","_col1"] |
> | TableScan [TS_0] |
> |
> Output:["vc","vc0"],properties:{"druid.fieldNames":"vc,vc0","druid.fieldTypes":"date,date","druid.query.json":"{\"queryType\":\"scan\",\"dataSource\":\"druid_tableau.calcs\",\"intervals\":[\"1900-01-01T00:00:00.000Z/3000-01-01T00:00:00.000Z\"],\"virtualColumns\":[{\"type\":\"expression\",\"name\":\"vc\",\"expression\":\"timestamp_floor(\\\"datetime0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"},{\"type\":\"expression\",\"name\":\"vc0\",\"expression\":\"timestamp_floor(\\\"date0\\\",'P1D','','UTC')\",\"outputType\":\"LONG\"}],\"columns\":[\"vc\",\"vc0\"],\"resultFormat\":\"compactedList\"}","druid.query.type":"scan"}
> |
> ||
> ++
> 10 rows selected (0.606 seconds)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-20018:
Issue Type: Sub-task (was: Bug)
Parent: HIVE-19416
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-20018) fix TestReplicationScenarios on the branch
[
https://issues.apache.org/jira/browse/HIVE-20018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-20018:
---
> fix TestReplicationScenarios on the branch
> --
>
> Key: HIVE-20018
> URL: https://issues.apache.org/jira/browse/HIVE-20018
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
>Priority: Major
>
> {noformat}
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropPartition
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithDropPartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadFailAndRetry
> (batchId=238)
> org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testStatus
> (batchId=238)
> {noformat}
> Most tests have errors, not being able to find a table/partition in
> getPartition. Might be related to catalog handling, or something else.
> Some have NPEs, some no obvious errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19532) fix tests for master-txnstats branch
[
https://issues.apache.org/jira/browse/HIVE-19532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16525452#comment-16525452
]
Sergey Shelukhin commented on HIVE-19532:
-
acid_nullscan and druidmini_mv look like they just need an update. For 2 files
with result changes I filed a bug yesterday.
Replication tests have the following errors that may be related:
{noformat}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
NoSuchObjectException(message:hive.concatenatepartitionedtable.ptned table not
found)
at org.apache.hadoop.hive.ql.metadata.Hive.getPartitions(Hive.java:3043)
at
org.apache.hadoop.hive.ql.exec.ArchiveUtils.conflictingArchiveNameOrNull(ArchiveUtils.java:299)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.validate(SemanticAnalyzer.java:12778)
... 41 more
Caused by: NoSuchObjectException(message:hive.concatenatepartitionedtable.ptned
table not found)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_ps_with_auth_result$get_partitions_ps_with_auth_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_ps_with_auth_result$get_partitions_ps_with_auth_resultStandardScheme.read(ThriftHiveMetastore.java)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_ps_with_auth_result.read(ThriftHiveMetastore.java)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partitions_ps_with_auth(ThriftHiveMetastore.java:3148)
at
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partitions_ps_with_auth(ThriftHiveMetastore.java:3130)
{noformat}
some bug in catalog handling?
{noformat}
2018-06-27T02:17:56,911 ERROR [main] ql.Driver: FAILED: Execution Error, return
code 1 from org.apache.hadoop.hive.ql.exec.MoveTask.
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:494)
at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:399)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
{noformat}
> fix tests for master-txnstats branch
>
>
> Key: HIVE-19532
> URL: https://issues.apache.org/jira/browse/HIVE-19532
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Steve Yeom
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: HIVE-19532.01.patch, HIVE-19532.01.prepatch,
> HIVE-19532.02.patch, HIVE-19532.02.prepatch, HIVE-19532.03.patch,
> HIVE-19532.04.patch, HIVE-19532.05.patch, HIVE-19532.06.patch,
> HIVE-19532.07.patch, HIVE-19532.08.patch, HIVE-19532.09.patch,
> HIVE-19532.10.patch, HIVE-19532.11.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
