[jira] [Commented] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
[ https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967186#comment-15967186 ] Colin Ma commented on HIVE-16311: - All tests are passed, and the failed test is not patch related. [Review board|https://reviews.apache.org/r/58377/] is also updated with the latest patch. > Improve the performance for FastHiveDecimalImpl.fastDivide > -- > > Key: HIVE-16311 > URL: https://issues.apache.org/jira/browse/HIVE-16311 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Colin Ma >Assignee: Colin Ma > Fix For: 3.0.0 > > Attachments: HIVE-16311.001.patch, HIVE-16311.002.patch, > HIVE-16311.003.patch, HIVE-16311.004.patch, HIVE-16311.005.patch, > HIVE-16311.006.patch, HIVE-16311.007.patch, HIVE-16311.008.patch, > HIVE-16311.withTrailingZero.patch > > > FastHiveDecimalImpl.fastDivide is poor performance when evaluate the > expression as 12345.67/123.45 > There are 2 points can be improved: > 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the > BigDecimal.divide. > 2. Get the precision for BigInteger in a fast way if possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp
[
https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967185#comment-15967185
]
Hive QA commented on HIVE-14412:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12830014/HIVE-14412.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4671/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4671/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4671/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-04-13 06:21:23.845
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-4671/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-04-13 06:21:23.848
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 3f681c2 HIVE-16390 : LLAP IO should take job config into
account; also LLAP config should load defaults (Sergey Shelukhin, reviewed by
Siddharth Seth)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 3f681c2 HIVE-16390 : LLAP IO should take job config into
account; also LLAP config should load defaults (Sergey Shelukhin, reviewed by
Siddharth Seth)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-04-13 06:21:25.789
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/contrib/src/test/queries/clientnegative/serde_regex.q: No such file or
directory
error: a/contrib/src/test/queries/clientpositive/serde_regex.q: No such file or
directory
error: a/contrib/src/test/results/clientnegative/serde_regex.q.out: No such
file or directory
error: a/contrib/src/test/results/clientpositive/serde_regex.q.out: No such
file or directory
error: a/hbase-handler/src/test/queries/positive/hbase_timestamp.q: No such
file or directory
error: a/hbase-handler/src/test/results/positive/hbase_timestamp.q.out: No such
file or directory
error: a/jdbc/src/java/org/apache/hive/jdbc/HiveBaseResultSet.java: No such
file or directory
error: a/jdbc/src/java/org/apache/hive/jdbc/JdbcColumn.java: No such file or
directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java: No
such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java: No
such file or directory
error:
a/ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java: No
such file or directory
error:
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g: No such file
or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g: No such file
or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g: No
such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java:
No such file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java: No such
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHour.java: No such file
or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java: No such
file or directory
error: a/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java: No such file
or directory
error: a/ql/src/java/org/apache/hadoop/hive/
[jira] [Commented] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
[
https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967180#comment-15967180
]
Hive QA commented on HIVE-16311:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863182/HIVE-16311.008.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4670/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4670/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4670/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863182 - PreCommit-HIVE-Build
> Improve the performance for FastHiveDecimalImpl.fastDivide
> --
>
> Key: HIVE-16311
> URL: https://issues.apache.org/jira/browse/HIVE-16311
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.2.0
>Reporter: Colin Ma
>Assignee: Colin Ma
> Fix For: 3.0.0
>
> Attachments: HIVE-16311.001.patch, HIVE-16311.002.patch,
> HIVE-16311.003.patch, HIVE-16311.004.patch, HIVE-16311.005.patch,
> HIVE-16311.006.patch, HIVE-16311.007.patch, HIVE-16311.008.patch,
> HIVE-16311.withTrailingZero.patch
>
>
> FastHiveDecimalImpl.fastDivide is poor performance when evaluate the
> expression as 12345.67/123.45
> There are 2 points can be improved:
> 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the
> BigDecimal.divide.
> 2. Get the precision for BigInteger in a fast way if possible.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16433) Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HIVE-16433:
-
Description:
Not nullify variable {{rj}} to avoid NPE due to race condition in ExecDriver.
currently {{rj}} is set to null in ExecDriver.shutdown which is called from
other thread for query cancellation. It can happen at any time. There is a
potential race condition, the rj is still accessed after shutdown is called.
For example: if the following code is executed right after ExecDriver.shutdown
is called.
{code}
this.jobID = rj.getJobID();
updateStatusInQueryDisplay();
returnVal = jobExecHelper.progress(rj, jc, ctx);
{code}
Currently the purpose of nullifying rj is mainly to make sure {{rj.killJob()}}
is only called once.
I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
once.
was:
Not nullify rj to avoid NPE due to race condition in ExecDriver. currently
{{rj}} is set to null in ExecDriver.shutdown which is called from other thread
for query cancellation. It can happen at any time. There is a potential race
condition, the rj is still accessed after shutdown is called. For example: if
the following code is executed right after ExecDriver.shutdown is called.
{code}
this.jobID = rj.getJobID();
updateStatusInQueryDisplay();
returnVal = jobExecHelper.progress(rj, jc, ctx);
{code}
Currently the purpose of nullifying rj is mainly to make sure {{rj.killJob()}}
is only called once.
I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
once.
Summary: Not nullify variable "rj" to avoid NPE due to race condition
in ExecDriver. (was: Not nullify rj to avoid NPE due to race condition in
ExecDriver.)
> Not nullify variable "rj" to avoid NPE due to race condition in ExecDriver.
> ---
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify variable {{rj}} to avoid NPE due to race condition in ExecDriver.
> currently {{rj}} is set to null in ExecDriver.shutdown which is called from
> other thread for query cancellation. It can happen at any time. There is a
> potential race condition, the rj is still accessed after shutdown is called.
> For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying rj is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
> once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16433) Not nullify rj to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967153#comment-15967153
]
Hive QA commented on HIVE-16433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863175/HIVE-16433.000.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=155)
org.apache.hive.jdbc.TestJdbcDriver2.testSelectExecAsync2 (batchId=221)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4669/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4669/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4669/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863175 - PreCommit-HIVE-Build
> Not nullify rj to avoid NPE due to race condition in ExecDriver.
>
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify rj to avoid NPE due to race condition in ExecDriver. currently
> {{rj}} is set to null in ExecDriver.shutdown which is called from other
> thread for query cancellation. It can happen at any time. There is a
> potential race condition, the rj is still accessed after shutdown is called.
> For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying rj is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
> once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[ https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967140#comment-15967140 ] Xuefu Zhang commented on HIVE-16430: +1 > Add log to show the cancelled query id when cancelOperation is called. > -- > > Key: HIVE-16430 > URL: https://issues.apache.org/jira/browse/HIVE-16430 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > Attachments: HIVE-16430.000.patch, HIVE-16430.001.patch > > > Add log to show the cancelled query id when cancelOperation is called. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[
https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967135#comment-15967135
]
zhihai xu edited comment on HIVE-16430 at 4/13/17 4:55 AM:
---
Thanks for the review [~xuefuz]! good suggestion, I add log when successfully
cancelled a query in the new patch HIVE-16430.001.patch. Looks like for
cancellation failure, currently it is already logged at the following code:
{code}
try {
cliService.cancelOperation(new OperationHandle(req.getOperationHandle()));
resp.setStatus(OK_STATUS);
} catch (Exception e) {
LOG.warn("Error cancelling operation: ", e);
resp.setStatus(HiveSQLException.toTStatus(e));
}
{code}
So I don't need to add the failure log.
was (Author: zxu):
Thanks for the review [~xuefuz]! good suggestion, I add log when successfully
cancelled a query. Looks like for cancellation failure, currently it is already
logged at the following code:
{code}
try {
cliService.cancelOperation(new OperationHandle(req.getOperationHandle()));
resp.setStatus(OK_STATUS);
} catch (Exception e) {
LOG.warn("Error cancelling operation: ", e);
resp.setStatus(HiveSQLException.toTStatus(e));
}
{code}
So I don't need to add the failure log.
> Add log to show the cancelled query id when cancelOperation is called.
> --
>
> Key: HIVE-16430
> URL: https://issues.apache.org/jira/browse/HIVE-16430
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Trivial
> Attachments: HIVE-16430.000.patch, HIVE-16430.001.patch
>
>
> Add log to show the cancelled query id when cancelOperation is called.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[
https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967135#comment-15967135
]
zhihai xu commented on HIVE-16430:
--
Thanks for the review [~xuefuz]! good suggestion, I add log when successfully
cancelled a query. Looks like for cancellation failure, currently it is already
logged at the following code:
{code}
try {
cliService.cancelOperation(new OperationHandle(req.getOperationHandle()));
resp.setStatus(OK_STATUS);
} catch (Exception e) {
LOG.warn("Error cancelling operation: ", e);
resp.setStatus(HiveSQLException.toTStatus(e));
}
{code}
So I don't need to add the failure log.
> Add log to show the cancelled query id when cancelOperation is called.
> --
>
> Key: HIVE-16430
> URL: https://issues.apache.org/jira/browse/HIVE-16430
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Trivial
> Attachments: HIVE-16430.000.patch, HIVE-16430.001.patch
>
>
> Add log to show the cancelled query id when cancelOperation is called.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[ https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-16430: - Attachment: HIVE-16430.001.patch > Add log to show the cancelled query id when cancelOperation is called. > -- > > Key: HIVE-16430 > URL: https://issues.apache.org/jira/browse/HIVE-16430 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > Attachments: HIVE-16430.000.patch, HIVE-16430.001.patch > > > Add log to show the cancelled query id when cancelOperation is called. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[
https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967104#comment-15967104
]
Hive QA commented on HIVE-16431:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863165/HIVE-16431.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4668/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4668/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4668/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863165 - PreCommit-HIVE-Build
> Support Parquet StatsNoJobTask for Spark & Tez engine
> -
>
> Key: HIVE-16431
> URL: https://issues.apache.org/jira/browse/HIVE-16431
> Project: Hive
> Issue Type: Improvement
>Reporter: Chao Sun
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HIVE-16431.1.patch
>
>
> It seems only MR uses StatsNoJobTask for Parquet input format when computing
> stats. We should add it to Tez & Spark as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967060#comment-15967060
]
Hive QA commented on HIVE-16287:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863163/HIVE-16287.02.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_merge8] (batchId=78)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testRenamePartition
(batchId=200)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testRenamePartition
(batchId=203)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testRenamePartition
(batchId=199)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testRenamePartition
(batchId=197)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testRenamePartition
(batchId=208)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4667/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4667/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4667/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863163 - PreCommit-HIVE-Build
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16435) Update "Understanding Hive Branches" in Hive Wiki
[ https://issues.apache.org/jira/browse/HIVE-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967032#comment-15967032 ] Pengcheng Xiong commented on HIVE-16435: Thanks [~sladymon], i guess the best person would be our president [~ashutoshc] :). But I would encourage you to put an initial version here and we can help you review. > Update "Understanding Hive Branches" in Hive Wiki > - > > Key: HIVE-16435 > URL: https://issues.apache.org/jira/browse/HIVE-16435 > Project: Hive > Issue Type: Bug > Components: Documentation >Reporter: Shannon Ladymon > > The "Understanding Hive Branches" section of the Hive Wiki was written before > the Hive 2.x and Hive 3.x versions were introduced and is currently outdated. > It needs to be fixed to include a description of the purpose of each branch > and what patches should be applied to which branches. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15986) Support "is [not] distinct from"
[
https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967024#comment-15967024
]
Hive QA commented on HIVE-15986:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863161/HIVE-15986.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10572 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[sample_islocalmode_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4666/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4666/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4666/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863161 - PreCommit-HIVE-Build
> Support "is [not] distinct from"
>
>
> Key: HIVE-15986
> URL: https://issues.apache.org/jira/browse/HIVE-15986
> Project: Hive
> Issue Type: Sub-task
> Components: SQL
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
> Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch,
> HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, HIVE-15986.5.patch
>
>
> Support standard "is [not] distinct from" syntax. For example this gives a
> standard way to do a comparison to null safe join: select * from t1 join t2
> on t1.x is not distinct from t2.y. SQL standard reference Section 8.15
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16435) Update "Understanding Hive Branches" in Hive Wiki
[ https://issues.apache.org/jira/browse/HIVE-16435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15967008#comment-15967008 ] Shannon Ladymon commented on HIVE-16435: [~pxiong], after seeing your [email | http://mail-archives.apache.org/mod_mbox/hive-dev/201703.mbox/%3ccams4yue4wn3zabeet-acn6smhup0k8ffgeu3w6m4y9wgjfs...@mail.gmail.com%3E] about the Hive 2.2, 2.3, and 3.0 releases, I went to update the Hive Wiki with some of that information, but realized that I don't have enough context to explain the purpose of the various branches in Hive. Can you suggest the best person to update the "Understanding Hive Branches" section of the Hive wiki page? * [HowToContribute - Understanding Hive Branches | https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-UnderstandingHiveBranches] > Update "Understanding Hive Branches" in Hive Wiki > - > > Key: HIVE-16435 > URL: https://issues.apache.org/jira/browse/HIVE-16435 > Project: Hive > Issue Type: Bug > Components: Documentation >Reporter: Shannon Ladymon > > The "Understanding Hive Branches" section of the Hive Wiki was written before > the Hive 2.x and Hive 3.x versions were introduced and is currently outdated. > It needs to be fixed to include a description of the purpose of each branch > and what patches should be applied to which branches. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[
https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966979#comment-15966979
]
Hive QA commented on HIVE-16430:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863158/HIVE-16430.000.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 10547 tests
executed
*Failed tests:*
{noformat}
TestWorker2 - did not produce a TEST-*.xml file (likely timed out) (batchId=252)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4665/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4665/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4665/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863158 - PreCommit-HIVE-Build
> Add log to show the cancelled query id when cancelOperation is called.
> --
>
> Key: HIVE-16430
> URL: https://issues.apache.org/jira/browse/HIVE-16430
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Trivial
> Attachments: HIVE-16430.000.patch
>
>
> Add log to show the cancelled query id when cancelOperation is called.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16311) Improve the performance for FastHiveDecimalImpl.fastDivide
[ https://issues.apache.org/jira/browse/HIVE-16311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-16311: Attachment: HIVE-16311.008.patch > Improve the performance for FastHiveDecimalImpl.fastDivide > -- > > Key: HIVE-16311 > URL: https://issues.apache.org/jira/browse/HIVE-16311 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.2.0 >Reporter: Colin Ma >Assignee: Colin Ma > Fix For: 3.0.0 > > Attachments: HIVE-16311.001.patch, HIVE-16311.002.patch, > HIVE-16311.003.patch, HIVE-16311.004.patch, HIVE-16311.005.patch, > HIVE-16311.006.patch, HIVE-16311.007.patch, HIVE-16311.008.patch, > HIVE-16311.withTrailingZero.patch > > > FastHiveDecimalImpl.fastDivide is poor performance when evaluate the > expression as 12345.67/123.45 > There are 2 points can be improved: > 1. Don't always use HiveDecimal.MAX_SCALE as scale when do the > BigDecimal.divide. > 2. Get the precision for BigInteger in a fast way if possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16434) Add support for parsing additional timestamp formats
[ https://issues.apache.org/jira/browse/HIVE-16434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966949#comment-15966949 ] Ashutosh Chauhan commented on HIVE-16434: - This should be supported both as constant literal in query text as well in Text File Format (via LazySimpleSerde). > Add support for parsing additional timestamp formats > > > Key: HIVE-16434 > URL: https://issues.apache.org/jira/browse/HIVE-16434 > Project: Hive > Issue Type: Bug > Components: File Formats, Query Planning >Reporter: Ashutosh Chauhan > > Will be useful to handle additional timestamp formats. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16434) Add support for parsing additional timestamp formats
[ https://issues.apache.org/jira/browse/HIVE-16434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966948#comment-15966948 ] Ashutosh Chauhan commented on HIVE-16434: - This should be supported both as constant literal in query text as well in Text File Format (via LazySimpleSerde). > Add support for parsing additional timestamp formats > > > Key: HIVE-16434 > URL: https://issues.apache.org/jira/browse/HIVE-16434 > Project: Hive > Issue Type: Bug > Components: File Formats, Query Planning >Reporter: Ashutosh Chauhan > > Will be useful to handle additional timestamp formats. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16434) Add support for parsing additional timestamp formats
[
https://issues.apache.org/jira/browse/HIVE-16434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966946#comment-15966946
]
Ashutosh Chauhan commented on HIVE-16434:
-
Currently, following returns NULL, whereas postgres and sql server (via its
datetime type) can successfully parse these:
{code}
select cast ('2011/01/01' as timestamp);
select cast ('2011.01.01' as timestamp);
select cast ('March 8, 1999' as timestamp);
select cast ('1/8/1999' as timestamp);
select cast ('01/02/03' as timestamp);
select cast ('1999-Jan-08' as timestamp);
select cast ('08-Jan-1999' as timestamp);
select cast ('08-Jan-99' as timestamp);
select cast ('19990108' as timestamp);
select cast ('990108' as timestamp);
select cast ('04 Mar 2013' as timestamp);
select cast('2012-02-03 04:0' as timestamp);
select cast('2012-02-03 04:05' as timestamp);
{code}
> Add support for parsing additional timestamp formats
>
>
> Key: HIVE-16434
> URL: https://issues.apache.org/jira/browse/HIVE-16434
> Project: Hive
> Issue Type: Bug
> Components: File Formats, Query Planning
>Reporter: Ashutosh Chauhan
>
> Will be useful to handle additional timestamp formats.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-15442) Driver.java has a redundancy code
[ https://issues.apache.org/jira/browse/HIVE-15442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966943#comment-15966943 ] Saijin Huang commented on HIVE-15442: - hi,[~aihuaxu] can you plz take a commit? > Driver.java has a redundancy code > -- > > Key: HIVE-15442 > URL: https://issues.apache.org/jira/browse/HIVE-15442 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > Attachments: HIVE-15442.1.patch > > > Driver.java has a redundancy code about "explain output", i think the if > statement " if (conf.getBoolVar(ConfVars.HIVE_LOG_EXPLAIN_OUTPUT))" has a > repeat judge with the above statement. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16409) TestEventHandlerFactory has lacked the ASF header
[ https://issues.apache.org/jira/browse/HIVE-16409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966942#comment-15966942 ] Saijin Huang commented on HIVE-16409: - hi,[~aihuaxu] can you plz take a commit? > TestEventHandlerFactory has lacked the ASF header > -- > > Key: HIVE-16409 > URL: https://issues.apache.org/jira/browse/HIVE-16409 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > Attachments: HIVE-16409.1.patch > > > when i test the issue HIVE-16061, i found TestEventHandlerFactory.java lack > the ASF header. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-8838) Support Parquet through HCatalog
[ https://issues.apache.org/jira/browse/HIVE-8838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966923#comment-15966923 ] Ferdinand Xu commented on HIVE-8838: Hi [~szita], I don't have circle to do this. Feel free to take it over. Thank you! > Support Parquet through HCatalog > > > Key: HIVE-8838 > URL: https://issues.apache.org/jira/browse/HIVE-8838 > Project: Hive > Issue Type: Bug >Reporter: Brock Noland >Assignee: Ferdinand Xu > > Similar to HIVE-8687 for Avro we need to fix Parquet with HCatalog. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966915#comment-15966915 ] Wei Zheng commented on HIVE-16399: -- [~ekoifman] Can you review the patches? Thanks. > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16399) create an index for tc_txnid in TXN_COMPONENTS
[ https://issues.apache.org/jira/browse/HIVE-16399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16399: - Attachment: HIVE-16399.master.patch HIVE-16399.branch-2.patch HIVE-16399.branch-2.3.patch > create an index for tc_txnid in TXN_COMPONENTS > -- > > Key: HIVE-16399 > URL: https://issues.apache.org/jira/browse/HIVE-16399 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Wei Zheng > Attachments: HIVE-16399.branch-2.3.patch, HIVE-16399.branch-2.patch, > HIVE-16399.master.patch > > > w/o this TxnStore.cleanEmptyAbortedTxns() can be very slow -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16433) Not nullify rj to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HIVE-16433:
-
Status: Patch Available (was: Open)
> Not nullify rj to avoid NPE due to race condition in ExecDriver.
>
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify rj to avoid NPE due to race condition in ExecDriver. currently
> {{rj}} is set to null in ExecDriver.shutdown which is called from other
> thread for query cancellation. It can happen at any time. There is a
> potential race condition, the rj is still accessed after shutdown is called.
> For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying rj is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
> once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16433) Not nullify rj to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HIVE-16433:
-
Attachment: HIVE-16433.000.patch
> Not nullify rj to avoid NPE due to race condition in ExecDriver.
>
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify rj to avoid NPE due to race condition in ExecDriver. currently
> {{rj}} is set to null in ExecDriver.shutdown which is called from other
> thread for query cancellation. It can happen at any time. There is a
> potential race condition, the rj is still accessed after shutdown is called.
> For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying rj is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
> once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16433) Not nullify rj to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated HIVE-16433:
-
Description:
Not nullify rj to avoid NPE due to race condition in ExecDriver. currently
{{rj}} is set to null in ExecDriver.shutdown which is called from other thread
for query cancellation. It can happen at any time. There is a potential race
condition, the rj is still accessed after shutdown is called. For example: if
the following code is executed right after ExecDriver.shutdown is called.
{code}
this.jobID = rj.getJobID();
updateStatusInQueryDisplay();
returnVal = jobExecHelper.progress(rj, jc, ctx);
{code}
Currently the purpose of nullifying rj is mainly to make sure {{rj.killJob()}}
is only called once.
I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
once.
was:
Not nullify rj to avoid NPE due to race condition in ExecDriver. currently
{{rj}} is set to null in ExecDriver.shutdown which is called from other thread
for query cancellation. It can happen at any time. There is a potential race
condition, the rj is still accessed after shutdown is called. For example: if
the following is called right after ExecDriver.shutdown is called.
{code}
this.jobID = rj.getJobID();
updateStatusInQueryDisplay();
returnVal = jobExecHelper.progress(rj, jc, ctx);
{code}
Currently the purpose of nullifying rj is mainly to make sure {{rj.killJob()}}
is only called once.
I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
once.
> Not nullify rj to avoid NPE due to race condition in ExecDriver.
>
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16433.000.patch
>
>
> Not nullify rj to avoid NPE due to race condition in ExecDriver. currently
> {{rj}} is set to null in ExecDriver.shutdown which is called from other
> thread for query cancellation. It can happen at any time. There is a
> potential race condition, the rj is still accessed after shutdown is called.
> For example: if the following code is executed right after
> ExecDriver.shutdown is called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying rj is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
> once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[
https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966911#comment-15966911
]
Hive QA commented on HIVE-16429:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863151/HIVE-16429.000.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 10571 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4664/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4664/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4664/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863151 - PreCommit-HIVE-Build
> Should call invokeFailureHooks in handleInterruption to track failed query
> execution due to interrupted command.
>
>
> Key: HIVE-16429
> URL: https://issues.apache.org/jira/browse/HIVE-16429
> Project: Hive
> Issue Type: Improvement
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
> Attachments: HIVE-16429.000.patch
>
>
> Should call invokeFailureHooks in handleInterruption to track failed query
> execution due to interrupted command.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16433) Not nullify rj to avoid NPE due to race condition in ExecDriver.
[
https://issues.apache.org/jira/browse/HIVE-16433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu reassigned HIVE-16433:
> Not nullify rj to avoid NPE due to race condition in ExecDriver.
>
>
> Key: HIVE-16433
> URL: https://issues.apache.org/jira/browse/HIVE-16433
> Project: Hive
> Issue Type: Bug
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Minor
>
> Not nullify rj to avoid NPE due to race condition in ExecDriver. currently
> {{rj}} is set to null in ExecDriver.shutdown which is called from other
> thread for query cancellation. It can happen at any time. There is a
> potential race condition, the rj is still accessed after shutdown is called.
> For example: if the following is called right after ExecDriver.shutdown is
> called.
> {code}
> this.jobID = rj.getJobID();
> updateStatusInQueryDisplay();
> returnVal = jobExecHelper.progress(rj, jc, ctx);
> {code}
> Currently the purpose of nullifying rj is mainly to make sure
> {{rj.killJob()}} is only called once.
> I will add a flag jobKilled to make sure {{rj.killJob()}} will be only called
> once.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16321) Possible deadlock in metastore with Acid enabled
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16321:
--
Description:
TxnStore.MutexAPI is a mechanism how different Metastore instances can
coordinate their operations. It uses a JDBCConnection to achieve it.
In some cases this may lead to deadlock. TxnHandler uses a connection pool of
fixed size. Suppose you have X simultaneous calls to TxnHandler.lock(), where
X is >= size of the pool. This take all connections form the pool, so when
{noformat}
handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
{noformat}
is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
pool is empty and the system is deadlocked.
MutexAPI can't use the same connection as the operation it's protecting.
(TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
We could make MutexAPI use a separate connection pool (size > 'primary' conn
pool).
Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
enqueueing the lock with the expectation that the caller will always follow up
with a call to checkLock(CheckLockRequest rqst).
cc [~f1sherox]
was:
TxnStore.MutexAPI is a mechanism how different Metastore instances can
coordinate their operations. It uses a JDBCConnection to achieve it.
In some cases this may lead to deadlock. TxnHandler uses a connection pool of
fixed size. Suppose you X simultaneous calls to TxnHandler.lock(), where X is
>= size of the pool. This take all connections form the pool, so when
{noformat}
handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
{noformat}
is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
pool is empty and the system is deadlocked.
MutexAPI can't use the same connection as the operation it's protecting.
(TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
We could make MutexAPI use a separate connection pool (size > 'primary' conn
pool).
Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
enqueueing the lock with the expectation that the caller will always follow up
with a call to checkLock(CheckLockRequest rqst).
cc [~f1sherox]
> Possible deadlock in metastore with Acid enabled
>
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you have X simultaneous calls to TxnHandler.lock(),
> where X is >= size of the pool. This take all connections form the pool, so
> when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16321) Possible deadlock in metastore with Acid enabled
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-16321:
--
Summary: Possible deadlock in metastore with Acid enabled (was: Possible
deadlock in metastore with Acid workload)
> Possible deadlock in metastore with Acid enabled
>
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you X simultaneous calls to TxnHandler.lock(), where
> X is >= size of the pool. This take all connections form the pool, so when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966896#comment-15966896 ] Eugene Koifman commented on HIVE-16193: --- LGTM > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Fix For: 2.2.0, 2.3.0, 3.0.0 > > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch, > HIVE-16193.addendum.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16193: - Attachment: HIVE-16193.addendum.patch [~ekoifman] Can you take a look at the addendum patch to see if the error message is good enough? > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Fix For: 2.2.0, 2.3.0, 3.0.0 > > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch, > HIVE-16193.addendum.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[ https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966879#comment-15966879 ] Xuefu Zhang commented on HIVE-16430: Thanks for working on this, [~zxu]. The patch looks good. It would be great if we can also add a msg after the cancellation operation, such as "Query xxx is successfully cancelled" or "Failed to cancel query xxx". > Add log to show the cancelled query id when cancelOperation is called. > -- > > Key: HIVE-16430 > URL: https://issues.apache.org/jira/browse/HIVE-16430 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > Attachments: HIVE-16430.000.patch > > > Add log to show the cancelled query id when cancelOperation is called. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16432) Improve plan for subquery with EXISTS
[ https://issues.apache.org/jira/browse/HIVE-16432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-16432: -- > Improve plan for subquery with EXISTS > - > > Key: HIVE-16432 > URL: https://issues.apache.org/jira/browse/HIVE-16432 > Project: Hive > Issue Type: Sub-task >Reporter: Vineet Garg >Assignee: Vineet Garg > > If an EXISTS/NOT EXISTS subquery contains an aggregate (with no group by or > windowing) it is guaranteed to produce at least one row. Since such subquery > in WHERE clause tests for existence of row it could be replaced by true > during compile time. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-10299) Enable new cost model for Tez execution engine
[ https://issues.apache.org/jira/browse/HIVE-10299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-10299: --- Status: Open (was: Patch Available) > Enable new cost model for Tez execution engine > -- > > Key: HIVE-10299 > URL: https://issues.apache.org/jira/browse/HIVE-10299 > Project: Hive > Issue Type: Task >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg > Attachments: HIVE-10299.2.patch, HIVE-10299.3.patch, HIVE-10299.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-10299) Enable new cost model for Tez execution engine
[ https://issues.apache.org/jira/browse/HIVE-10299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-10299: --- Status: Patch Available (was: Open) Submitting the same patch again to get test report > Enable new cost model for Tez execution engine > -- > > Key: HIVE-10299 > URL: https://issues.apache.org/jira/browse/HIVE-10299 > Project: Hive > Issue Type: Task >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg > Attachments: HIVE-10299.2.patch, HIVE-10299.3.patch, HIVE-10299.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-16431: Attachment: HIVE-16431.1.patch > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-16431.1.patch > > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-16431: Status: Patch Available (was: Open) > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > Attachments: HIVE-16431.1.patch > > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16287) Alter table partition rename with location - moves partition back to hive warehouse
[
https://issues.apache.org/jira/browse/HIVE-16287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-16287:
---
Attachment: HIVE-16287.02.patch
Thanks for the review [~spena] It turns out that there was a similar method
already in Warehouse.java. When I looked more I found that there are multiple
methods doing slightly different things in returning the partition path. I
found it very confusing and hence I renamed a method and added javadoc for all
the methods so that it is easier to understand when to use which one. Updating
the patch with my changes.
> Alter table partition rename with location - moves partition back to hive
> warehouse
> ---
>
> Key: HIVE-16287
> URL: https://issues.apache.org/jira/browse/HIVE-16287
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 2.1.0
> Environment: RHEL 6.8
>Reporter: Ying Chen
>Assignee: Vihang Karajgaonkar
>Priority: Minor
> Attachments: HIVE-16287.01.patch, HIVE-16287.02.patch
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> I was renaming my partition in a table that I've created using the location
> clause, and noticed that when after rename is completed, my partition is
> moved to the hive warehouse (hive.metastore.warehouse.dir).
> {quote}
> create table test_local_part (col1 int) partitioned by (col2 int) location
> '/tmp/testtable/test_local_part';
> insert into test_local_part partition (col2=1) values (1),(3);
> insert into test_local_part partition (col2=2) values (3);
> alter table test_local_part partition (col2='1') rename to partition
> (col2='4');
> {quote}
> Running:
>describe formatted test_local_part partition (col2='2')
> # Detailed Partition Information
> Partition Value: [2]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:25:28 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/tmp/testtable/test_local_part/col2=2*
> Running:
>describe formatted test_local_part partition (col2='4')
> # Detailed Partition Information
> Partition Value: [4]
> Database: default
> Table:test_local_part
> CreateTime: Mon Mar 20 13:24:53 PDT 2017
> LastAccessTime: UNKNOWN
> Protect Mode: None
> Location:
> *hdfs://my.server.com:8020/apps/hive/warehouse/test_local_part/col2=4*
> ---
> Per Sergio's comment - "The rename should create the new partition name in
> the same location of the table. "
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-15986) Support "is [not] distinct from"
[ https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15986: --- Status: Open (was: Patch Available) > Support "is [not] distinct from" > > > Key: HIVE-15986 > URL: https://issues.apache.org/jira/browse/HIVE-15986 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Vineet Garg > Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch, > HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, HIVE-15986.5.patch > > > Support standard "is [not] distinct from" syntax. For example this gives a > standard way to do a comparison to null safe join: select * from t1 join t2 > on t1.x is not distinct from t2.y. SQL standard reference Section 8.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15986) Support "is [not] distinct from"
[ https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15986: --- Attachment: HIVE-15986.5.patch HIVE-15986.5.patch has added more tests. > Support "is [not] distinct from" > > > Key: HIVE-15986 > URL: https://issues.apache.org/jira/browse/HIVE-15986 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Vineet Garg > Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch, > HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, HIVE-15986.5.patch > > > Support standard "is [not] distinct from" syntax. For example this gives a > standard way to do a comparison to null safe join: select * from t1 join t2 > on t1.x is not distinct from t2.y. SQL standard reference Section 8.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-15986) Support "is [not] distinct from"
[ https://issues.apache.org/jira/browse/HIVE-15986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15986: --- Status: Patch Available (was: Open) > Support "is [not] distinct from" > > > Key: HIVE-15986 > URL: https://issues.apache.org/jira/browse/HIVE-15986 > Project: Hive > Issue Type: Sub-task > Components: SQL >Reporter: Carter Shanklin >Assignee: Vineet Garg > Attachments: HIVE-15986.1.patch, HIVE-15986.2.patch, > HIVE-15986.3.patch, HIVE-15986.4.patch, HIVE-15986.5.patch, HIVE-15986.5.patch > > > Support standard "is [not] distinct from" syntax. For example this gives a > standard way to do a comparison to null safe join: select * from t1 join t2 > on t1.x is not distinct from t2.y. SQL standard reference Section 8.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16431) Support Parquet StatsNoJobTask for Spark & Tez engine
[ https://issues.apache.org/jira/browse/HIVE-16431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun reassigned HIVE-16431: --- > Support Parquet StatsNoJobTask for Spark & Tez engine > - > > Key: HIVE-16431 > URL: https://issues.apache.org/jira/browse/HIVE-16431 > Project: Hive > Issue Type: Improvement >Reporter: Chao Sun >Assignee: Chao Sun >Priority: Minor > > It seems only MR uses StatsNoJobTask for Parquet input format when computing > stats. We should add it to Tez & Spark as well. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16356) Table#validateColumns should avoid checking exhaustively for matches in a list
[
https://issues.apache.org/jira/browse/HIVE-16356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966799#comment-15966799
]
Hive QA commented on HIVE-16356:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12862820/HIVE-16356.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_non_partitioned]
(batchId=7)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteTimestamp
(batchId=178)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteTinyint
(batchId=178)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4663/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4663/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4663/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12862820 - PreCommit-HIVE-Build
> Table#validateColumns should avoid checking exhaustively for matches in a list
> --
>
> Key: HIVE-16356
> URL: https://issues.apache.org/jira/browse/HIVE-16356
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Janos Gub
> Attachments: HIVE-16356.patch
>
>
> Creating a table with lots of columns..seems like Hive is taking a break at
> validating the column names uniqueness :)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (HIVE-16321) Possible deadlock in metastore with Acid workload
[
https://issues.apache.org/jira/browse/HIVE-16321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-16321:
-
Assignee: Eugene Koifman
> Possible deadlock in metastore with Acid workload
> -
>
> Key: HIVE-16321
> URL: https://issues.apache.org/jira/browse/HIVE-16321
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.3.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> TxnStore.MutexAPI is a mechanism how different Metastore instances can
> coordinate their operations. It uses a JDBCConnection to achieve it.
> In some cases this may lead to deadlock. TxnHandler uses a connection pool
> of fixed size. Suppose you X simultaneous calls to TxnHandler.lock(), where
> X is >= size of the pool. This take all connections form the pool, so when
> {noformat}
> handle = getMutexAPI().acquireLock(MUTEX_KEY.CheckLock.name());
> {noformat}
> is executed in _TxnHandler.checkLock(Connection dbConn, long extLockId)_ the
> pool is empty and the system is deadlocked.
> MutexAPI can't use the same connection as the operation it's protecting.
> (TxnHandler.checkLock(Connection dbConn, long extLockId) is an example).
> We could make MutexAPI use a separate connection pool (size > 'primary' conn
> pool).
> Or we could make TxnHandler.lock(LockRequest rqst) return immediately after
> enqueueing the lock with the expectation that the caller will always follow
> up with a call to checkLock(CheckLockRequest rqst).
> cc [~f1sherox]
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[ https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-16430: - Status: Patch Available (was: Open) > Add log to show the cancelled query id when cancelOperation is called. > -- > > Key: HIVE-16430 > URL: https://issues.apache.org/jira/browse/HIVE-16430 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > Attachments: HIVE-16430.000.patch > > > Add log to show the cancelled query id when cancelOperation is called. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[ https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-16430: - Attachment: HIVE-16430.000.patch > Add log to show the cancelled query id when cancelOperation is called. > -- > > Key: HIVE-16430 > URL: https://issues.apache.org/jira/browse/HIVE-16430 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > Attachments: HIVE-16430.000.patch > > > Add log to show the cancelled query id when cancelOperation is called. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16430) Add log to show the cancelled query id when cancelOperation is called.
[ https://issues.apache.org/jira/browse/HIVE-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu reassigned HIVE-16430: > Add log to show the cancelled query id when cancelOperation is called. > -- > > Key: HIVE-16430 > URL: https://issues.apache.org/jira/browse/HIVE-16430 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > > Add log to show the cancelled query id when cancelOperation is called. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[ https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-16429: - Status: Patch Available (was: Open) > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. > > > Key: HIVE-16429 > URL: https://issues.apache.org/jira/browse/HIVE-16429 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16429.000.patch > > > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[ https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-16429: - Attachment: HIVE-16429.000.patch > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. > > > Key: HIVE-16429 > URL: https://issues.apache.org/jira/browse/HIVE-16429 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: HIVE-16429.000.patch > > > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16429) Should call invokeFailureHooks in handleInterruption to track failed query execution due to interrupted command.
[ https://issues.apache.org/jira/browse/HIVE-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu reassigned HIVE-16429: > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. > > > Key: HIVE-16429 > URL: https://issues.apache.org/jira/browse/HIVE-16429 > Project: Hive > Issue Type: Improvement >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > > Should call invokeFailureHooks in handleInterruption to track failed query > execution due to interrupted command. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966748#comment-15966748
]
Hive QA commented on HIVE-11133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863139/HIVE-11133.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10572 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_semijoin]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=144)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4662/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4662/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4662/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863139 - PreCommit-HIVE-Build
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-13842) Expose ability to set number of connections in the pool in TxnHandler
[
https://issues.apache.org/jira/browse/HIVE-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-13842:
--
Description:
Current defaults are hardcoded 8/10 for dbcp/bonecp
Various Connection pool managers support reading configs from a config file.
The implementation would ideally allow end user to point Hive at the manager
specific config file to let the user have full control
HikariCP supports
HikariConfig config = new HikariConfig("/some/path/hikari.properties");
see https://github.com/brettwooldridge/HikariCP
BoneCP supports
bonecp-config.xml
see
https://github.com/wwadge/bonecp/blob/master/bonecp/src/main/java/com/jolbox/bonecp/BoneCPConfig.java
was:
Current defaults are hardcoded 8/10 for dbcp/bonecp
Various Connection pool managers support reading configs from a config file.
The implementation would ideally allow end user to point Hive at the manager
specific config file to let the user have full control
> Expose ability to set number of connections in the pool in TxnHandler
> -
>
> Key: HIVE-13842
> URL: https://issues.apache.org/jira/browse/HIVE-13842
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Priority: Critical
>
> Current defaults are hardcoded 8/10 for dbcp/bonecp
> Various Connection pool managers support reading configs from a config file.
> The implementation would ideally allow end user to point Hive at the manager
> specific config file to let the user have full control
> HikariCP supports
> HikariConfig config = new HikariConfig("/some/path/hikari.properties");
> see https://github.com/brettwooldridge/HikariCP
> BoneCP supports
> bonecp-config.xml
> see
> https://github.com/wwadge/bonecp/blob/master/bonecp/src/main/java/com/jolbox/bonecp/BoneCPConfig.java
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966700#comment-15966700 ] Wei Zheng edited comment on HIVE-16193 at 4/12/17 10:08 PM: Thanks Eugene for the review. Committed to master, branch-2, branch-2.3 and branch-2.2. was (Author: wzheng): Thanks Eugene for the review. Committed to master, branch-2 and branch-2.3. > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Fix For: 2.2.0, 2.3.0, 3.0.0 > > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16193: - Fix Version/s: 2.2.0 > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Fix For: 2.2.0, 2.3.0, 3.0.0 > > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15851) CompactorMR.launchCompactionJob() should use JobClient.submitJob() not runJob
[ https://issues.apache.org/jira/browse/HIVE-15851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966724#comment-15966724 ] Wei Zheng commented on HIVE-15851: -- Previous commit by Eugene went into master, then branch-2, then branch-2.3. I just backported it to branch-2.2 as well. > CompactorMR.launchCompactionJob() should use JobClient.submitJob() not runJob > - > > Key: HIVE-15851 > URL: https://issues.apache.org/jira/browse/HIVE-15851 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Fix For: 2.2.0 > > Attachments: HIVE-15851.01.patch > > > otherwise hadoop JobId of the job is not visible in SHOW COMPACTIONS until > the job completes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12614) RESET command does not close spark session
[ https://issues.apache.org/jira/browse/HIVE-12614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966720#comment-15966720 ] Xuefu Zhang commented on HIVE-12614: Patch looks good. Minor comment on RB. > RESET command does not close spark session > -- > > Key: HIVE-12614 > URL: https://issues.apache.org/jira/browse/HIVE-12614 > Project: Hive > Issue Type: Bug > Components: Spark >Affects Versions: 1.3.0, 2.1.0 >Reporter: Nemon Lou >Assignee: Sahil Takiar >Priority: Minor > Attachments: HIVE-12614.1.patch, HIVE-12614.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-16193: - Resolution: Fixed Fix Version/s: 3.0.0 2.3.0 Status: Resolved (was: Patch Available) > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Fix For: 2.3.0, 3.0.0 > > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16390) LLAP IO should take job config into account; also LLAP config should load defaults
[ https://issues.apache.org/jira/browse/HIVE-16390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-16390: Resolution: Fixed Fix Version/s: 3.0.0 2.3.0 2.2.0 Status: Resolved (was: Patch Available) Committed to million branches. > LLAP IO should take job config into account; also LLAP config should load > defaults > -- > > Key: HIVE-16390 > URL: https://issues.apache.org/jira/browse/HIVE-16390 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Fix For: 2.2.0, 2.3.0, 3.0.0 > > Attachments: HIVE-16390.patch > > > Ensure the config is used consistently with task-based execution by default; > the exceptions should be specific (settings we don't want overridden, like > zero-copy). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966700#comment-15966700 ] Wei Zheng commented on HIVE-16193: -- Thanks Eugene for the review. Committed to master, branch-2 and branch-2.3. > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16146) If possible find a better way to filter the TestBeeLineDriver output
[ https://issues.apache.org/jira/browse/HIVE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966701#comment-15966701 ] Zoltan Haindrich commented on HIVE-16146: - [~pvary] It seems like the patch needs some rebasing now...since HIVE-16345 is in Thank you for your comments...It's good to know that you also see these things similarily! :) > If possible find a better way to filter the TestBeeLineDriver output > > > Key: HIVE-16146 > URL: https://issues.apache.org/jira/browse/HIVE-16146 > Project: Hive > Issue Type: Improvement > Components: Testing Infrastructure >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Attachments: HIVE-16146.02.patch, HIVE-16146.03.patch, > HIVE-16146.04.patch, HIVE-16146.05.patch, HIVE-16146.patch > > > Currently we apply a blacklist to filter the output of the BeeLine Qtest runs. > It might be a good idea to go thorough of the possibilities and find a better > way, if possible. > I think our main goal could be for the TestBeeLineDriver test output to match > the TestCliDriver output of the came query file. Or if it is not possible, > then at least a similar one > CC: [~vihangk1] -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16214) Explore the possibillity of introducing a service-client module
[ https://issues.apache.org/jira/browse/HIVE-16214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16214: Resolution: Feedback Received Status: Resolved (was: Patch Available) > Explore the possibillity of introducing a service-client module > --- > > Key: HIVE-16214 > URL: https://issues.apache.org/jira/browse/HIVE-16214 > Project: Hive > Issue Type: Improvement > Components: JDBC, Metastore >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-16214.1.patch, HIVE-16214.2.patch > > > The jdbc driver pulls in a lot of things from hive...and that may affect the > jdbc driver user. > In this ticket I experiment with the extraction of the relevant parts of > service(wrt to the jdbc driver) into a service-client module. > I've opened a PR...to enable commit by commit view: > https://github.com/apache/hive/pull/158 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16214) Explore the possibillity of introducing a service-client module
[ https://issues.apache.org/jira/browse/HIVE-16214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966685#comment-15966685 ] Zoltan Haindrich commented on HIVE-16214: - I prefer to do refactoring; and moving classes around separately :) so... I'm closing this; I will do the real work in smaller steps in HIVE-16354 > Explore the possibillity of introducing a service-client module > --- > > Key: HIVE-16214 > URL: https://issues.apache.org/jira/browse/HIVE-16214 > Project: Hive > Issue Type: Improvement > Components: JDBC, Metastore >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Attachments: HIVE-16214.1.patch, HIVE-16214.2.patch > > > The jdbc driver pulls in a lot of things from hive...and that may affect the > jdbc driver user. > In this ticket I experiment with the extraction of the relevant parts of > service(wrt to the jdbc driver) into a service-client module. > I've opened a PR...to enable commit by commit view: > https://github.com/apache/hive/pull/158 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16356) Table#validateColumns should avoid checking exhaustively for matches in a list
[ https://issues.apache.org/jira/browse/HIVE-16356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16356: Status: Patch Available (was: Open) +1 pending tests > Table#validateColumns should avoid checking exhaustively for matches in a list > -- > > Key: HIVE-16356 > URL: https://issues.apache.org/jira/browse/HIVE-16356 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Janos Gub > Attachments: HIVE-16356.patch > > > Creating a table with lots of columns..seems like Hive is taking a break at > validating the column names uniqueness :) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966675#comment-15966675
]
Sahil Takiar commented on HIVE-11133:
-
RB: https://reviews.apache.org/r/58402/
* Added a {{DagJsonParser}} which is an abstract class and has two
implementations: {{TezJsonParser}} and {{SparkJsonParser}}
** Almost all of the logic in {{TezJsonParser}} was moved to {{DagJsonParser}}
* Had to disable the {{CorrelationOptimizer}} for Spark, seems that it doesn't
work - should be ok since the {{CorrelationOptimizer}} is disabled by default
anyways
* Had to add a few explicit checks for JSON objects to avoid some
NullPointerExceptions
* Most of the other changes are just re-factoring
* Turned {{hive.explain.user}} off for all the Spark qtests so that we don't
have to updated all the qtests
* {{hive.explain.user}} is on by default, but we could create a separate config
for Spark and turn it off by default?
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16388) LLAP: Log rotation for daemon, history and gc files
[ https://issues.apache.org/jira/browse/HIVE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16388: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Test failures are unrelated and are already happening in master. Committed to master. Thanks for the reviews! > LLAP: Log rotation for daemon, history and gc files > --- > > Key: HIVE-16388 > URL: https://issues.apache.org/jira/browse/HIVE-16388 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Siddharth Seth >Assignee: Prasanth Jayachandran > Fix For: 3.0.0 > > Attachments: HIVE-16388.1.patch, HIVE-16388.2.patch > > > GC logs need to be rotated by date. > LLAP daemon history logs as well > Ideally, the daemon.out file needs the same > Need to be able to download relevant logfiles for a time window. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-13567) Auto-gather column stats - phase 2
[
https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1597#comment-1597
]
Hive QA commented on HIVE-13567:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863105/HIVE-13567.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 77 failed/errored test(s), 10572 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_11]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_12]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_13]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_16]
(batchId=234)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=234)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark1]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark2]
(batchId=2)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_spark3]
(batchId=42)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin5]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative2]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_3]
(batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_4]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cast1] (batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_rdd_cache]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby10] (batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby11] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby12] (batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby1] (batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby1_noskew]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_noskew]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_noskew_multi_distinct]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4] (batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4_noskew]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby6] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby6_noskew]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby7_map] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby7_map_multi_single_reducer]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby7_map_skew]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby8_map]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby8_map_skew]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby9] (batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_cube_multi_gby]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_position]
(batchId=37)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr] (batchId=28)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr_multi_distinct]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort_grouping_operators]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input12] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input13] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input8] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input9] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input_part10] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part1]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[load_dyn_part8]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_mixed]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_coltype_literals]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_constant_expr]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_join_partition_key]
(batchId
[jira] [Updated] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-16193: -- Target Version/s: 2.2.0, 3.0.0 > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966653#comment-15966653 ] Eugene Koifman commented on HIVE-16193: --- I think this needs to be back ported to 2.x line +1 > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-12614) RESET command does not close spark session
[
https://issues.apache.org/jira/browse/HIVE-12614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966654#comment-15966654
]
Sahil Takiar commented on HIVE-12614:
-
Approach is pretty simple, when reseting the configs in {{ResetProcessor}}, use
the {{SetProcessor#setConf}} method which already handles closing the spark
session if the execution engine is changed to something besides {{spark}}.
> RESET command does not close spark session
> --
>
> Key: HIVE-12614
> URL: https://issues.apache.org/jira/browse/HIVE-12614
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Sahil Takiar
>Priority: Minor
> Attachments: HIVE-12614.1.patch, HIVE-12614.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-11133) Support hive.explain.user for Spark [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-11133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-11133:
Attachment: HIVE-11133.4.patch
> Support hive.explain.user for Spark [Spark Branch]
> --
>
> Key: HIVE-11133
> URL: https://issues.apache.org/jira/browse/HIVE-11133
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Mohit Sabharwal
>Assignee: Sahil Takiar
> Attachments: HIVE-11133.1.patch, HIVE-11133.2.patch,
> HIVE-11133.3.patch, HIVE-11133.4.patch
>
>
> User friendly explain output ({{set hive.explain.user=true}}) should support
> Spark as well.
> Once supported, we should also enable related q-tests like {{explainuser_1.q}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16193) Hive show compactions not reflecting the status of the application
[ https://issues.apache.org/jira/browse/HIVE-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966647#comment-15966647 ] Wei Zheng commented on HIVE-16193: -- Couldn't find an easy way to fail the MR job from unit test. But the fix should restore the logic as before - if the job fails, an exception will be thrown from CompactorMR, which will cause Worker to mark that specific compaction entry as failed. > Hive show compactions not reflecting the status of the application > -- > > Key: HIVE-16193 > URL: https://issues.apache.org/jira/browse/HIVE-16193 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 2.2.0 >Reporter: Kavan Suresh >Assignee: Wei Zheng > Attachments: HIVE-16193.1.patch, HIVE-16193.2.patch > > > In a test for [HIVE-13354|https://issues.apache.org/jira/browse/HIVE-13354], > we set properties to make the compaction fail. Recently show compactions > indicates that compactions have been succeeding on the tables though the > corresponding application gets killed as expected. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HIVE-16428) Refactor & fix the logic in HoS mapjoin optimization
[
https://issues.apache.org/jira/browse/HIVE-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun reassigned HIVE-16428:
---
> Refactor & fix the logic in HoS mapjoin optimization
>
>
> Key: HIVE-16428
> URL: https://issues.apache.org/jira/browse/HIVE-16428
> Project: Hive
> Issue Type: Improvement
>Reporter: Chao Sun
>Assignee: Chao Sun
>
> [The logic for mapjoin optimization in
> HoS|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java#L276]
> seems unnecessarily complex and (perhaps) buggy. For instance, If
> {{bigInputStat}} is null AND the position is not in {{bigTableCandidateSet}},
> then the size for the current position will not be counted into the
> {{totalSize}}. This seems wrong.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16388) LLAP: Log rotation for daemon, history and gc files
[ https://issues.apache.org/jira/browse/HIVE-16388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966594#comment-15966594 ] Siddharth Seth commented on HIVE-16388: --- +1 > LLAP: Log rotation for daemon, history and gc files > --- > > Key: HIVE-16388 > URL: https://issues.apache.org/jira/browse/HIVE-16388 > Project: Hive > Issue Type: Bug > Components: llap >Affects Versions: 3.0.0 >Reporter: Siddharth Seth >Assignee: Prasanth Jayachandran > Attachments: HIVE-16388.1.patch, HIVE-16388.2.patch > > > GC logs need to be rotated by date. > LLAP daemon history logs as well > Ideally, the daemon.out file needs the same > Need to be able to download relevant logfiles for a time window. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16427) Fix multi-insert query and write qtests
[
https://issues.apache.org/jira/browse/HIVE-16427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Poepping updated HIVE-16427:
---
Description:
On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519 was
not actually fixed.
This task is to find the problem, fix it, and add qtests to verify no future
regression.
Specifically, the following query does not produce correct answers:
{code}
>From (select * from src) a
insert overwrite directory '/tmp/emp/dir1/'
select key, value
insert overwrite directory '/tmp/emp/dir2/'
select 'header'
limit 0
insert overwrite directory '/tmp/emp/dir3/'
select key, value
where key = 100;
{code}
was:
On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519 was
not actually fixed.
This task is to find the problem, fix it, and add qtests to verify no future
regression.
> Fix multi-insert query and write qtests
> ---
>
> Key: HIVE-16427
> URL: https://issues.apache.org/jira/browse/HIVE-16427
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Thomas Poepping
>
> On HIVE-16415, it was found that the bug reported to be fixed in HIVE-14519
> was not actually fixed.
> This task is to find the problem, fix it, and add qtests to verify no future
> regression.
> Specifically, the following query does not produce correct answers:
> {code}
> From (select * from src) a
> insert overwrite directory '/tmp/emp/dir1/'
> select key, value
> insert overwrite directory '/tmp/emp/dir2/'
> select 'header'
> limit 0
> insert overwrite directory '/tmp/emp/dir3/'
> select key, value
> where key = 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16316) Prepare master branch for 3.0.0 development.
[ https://issues.apache.org/jira/browse/HIVE-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966588#comment-15966588 ] Wei Zheng commented on HIVE-16316: -- [~ngangam] The addendum patch looks good. Thanks for the quick fix. Can you commit it? > Prepare master branch for 3.0.0 development. > > > Key: HIVE-16316 > URL: https://issues.apache.org/jira/browse/HIVE-16316 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 3.0.0 > > Attachments: HIVE-16316-addendum.patch, HIVE-16316.patch > > > master branch is now being used for 3.0.0 development. The build files will > need to reflect this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16171) Support replication of truncate table
[ https://issues.apache.org/jira/browse/HIVE-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966587#comment-15966587 ] Pengcheng Xiong commented on HIVE-16171: LGTM as well. > Support replication of truncate table > - > > Key: HIVE-16171 > URL: https://issues.apache.org/jira/browse/HIVE-16171 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16171.01.patch, HIVE-16171.02.patch, > HIVE-16171.03.patch, HIVE-16171.04.patch, HIVE-16171.05.patch > > > Need to support truncate table for replication. Key points to note. > 1. For non-partitioned table, truncate table will remove all the rows from > the table. > 2. For partitioned tables, need to consider how truncate behaves if truncate > a partition or the whole table. > 3. Bootstrap load with truncate table must work as it is just > loadTable/loadPartition with empty dataset. > 4. It is suggested to re-use the alter table/alter partition events to handle > truncate. > 5. Need to consider the case where insert event happens before truncate table > which needs to see their data files through change management. The data files > should be recycled to the cmroot path before trashing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add blobstore tests for insertion of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966580#comment-15966580 ] Thomas Poepping commented on HIVE-16415: Thanks [~spena], created HIVE-16427 as a follow-up to HIVE-14519. > Add blobstore tests for insertion of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16390) LLAP IO should take job config into account; also LLAP config should load defaults
[ https://issues.apache.org/jira/browse/HIVE-16390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966577#comment-15966577 ] Siddharth Seth commented on HIVE-16390: --- +1 > LLAP IO should take job config into account; also LLAP config should load > defaults > -- > > Key: HIVE-16390 > URL: https://issues.apache.org/jira/browse/HIVE-16390 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Attachments: HIVE-16390.patch > > > Ensure the config is used consistently with task-based execution by default; > the exceptions should be specific (settings we don't want overridden, like > zero-copy). -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16328) HoS: more aggressive mapjoin optimization when hive.spark.use.ts.stats.for.mapjoin is true
[
https://issues.apache.org/jira/browse/HIVE-16328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-16328:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks [~xuefuz] for the review.
> HoS: more aggressive mapjoin optimization when
> hive.spark.use.ts.stats.for.mapjoin is true
> --
>
> Key: HIVE-16328
> URL: https://issues.apache.org/jira/browse/HIVE-16328
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 3.0.0
>
> Attachments: HIVE-16328.1.patch, HIVE-16328.2.patch
>
>
> In HIVE-15489, when {{hive.spark.use.ts.stats.for.mapjoin}} is set to true,
> and if the JOIN op has any upstream RS operator, then we will stop converting
> the JOIN op to MAPJOIN op.
> However, this is overly conservative. A better solution is to treat the
> branch that has upstream RS as the big table and check if all other branches
> are map-only AND can fit in hash table size.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HIVE-16328) HoS: more aggressive mapjoin optimization when hive.spark.use.ts.stats.for.mapjoin is true
[
https://issues.apache.org/jira/browse/HIVE-16328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-16328:
Summary: HoS: more aggressive mapjoin optimization when
hive.spark.use.ts.stats.for.mapjoin is true (was: HoS: more aggressive mapjoin
optimization when hive.spark.use.file.size.for.mapjoin is true)
> HoS: more aggressive mapjoin optimization when
> hive.spark.use.ts.stats.for.mapjoin is true
> --
>
> Key: HIVE-16328
> URL: https://issues.apache.org/jira/browse/HIVE-16328
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-16328.1.patch, HIVE-16328.2.patch
>
>
> In HIVE-15489, when {{hive.spark.use.ts.stats.for.mapjoin}} is set to true,
> and if the JOIN op has any upstream RS operator, then we will stop converting
> the JOIN op to MAPJOIN op.
> However, this is overly conservative. A better solution is to treat the
> branch that has upstream RS as the big table and check if all other branches
> are map-only AND can fit in hash table size.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16197) Incremental insert into a partitioned table doesn't get replicated.
[
https://issues.apache.org/jira/browse/HIVE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966563#comment-15966563
]
Hive QA commented on HIVE-16197:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12863106/HIVE-16197.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10572 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_varchar_simple]
(batchId=71)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr]
(batchId=143)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/4660/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/4660/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-4660/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12863106 - PreCommit-HIVE-Build
> Incremental insert into a partitioned table doesn't get replicated.
> ---
>
> Key: HIVE-16197
> URL: https://issues.apache.org/jira/browse/HIVE-16197
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Affects Versions: 2.1.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
> Labels: DR
> Attachments: HIVE-16197.01.patch, HIVE-16197.02.patch,
> HIVE-16197.03.patch, HIVE-16197.04.patch
>
>
> Insert to a partitioned table doesn't replicate properly in case of
> incremental dump/load. Few key points to be noted.
> 1. If insert command itself created the new partition, then the inserted row
> is replicated. But the subsequent inserts into the same table doesn't get
> replicated.
> 2. If the partition is created using ALTER TABLE command, then none of the
> inserted rows to this partition is getting replicated. However, the partition
> metadata is getting replicated.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HIVE-16171) Support replication of truncate table
[ https://issues.apache.org/jira/browse/HIVE-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966416#comment-15966416 ] Thejas M Nair commented on HIVE-16171: -- The update for stats logic looks good to me. [~pxiong] can you please review ? cc [~ashutoshc] > Support replication of truncate table > - > > Key: HIVE-16171 > URL: https://issues.apache.org/jira/browse/HIVE-16171 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16171.01.patch, HIVE-16171.02.patch, > HIVE-16171.03.patch, HIVE-16171.04.patch, HIVE-16171.05.patch > > > Need to support truncate table for replication. Key points to note. > 1. For non-partitioned table, truncate table will remove all the rows from > the table. > 2. For partitioned tables, need to consider how truncate behaves if truncate > a partition or the whole table. > 3. Bootstrap load with truncate table must work as it is just > loadTable/loadPartition with empty dataset. > 4. It is suggested to re-use the alter table/alter partition events to handle > truncate. > 5. Need to consider the case where insert event happens before truncate table > which needs to see their data files through change management. The data files > should be recycled to the cmroot path before trashing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16171) Support replication of truncate table
[ https://issues.apache.org/jira/browse/HIVE-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966363#comment-15966363 ] Eugene Koifman commented on HIVE-16171: --- I think [~pxiong] is in a better position to comments on Stats > Support replication of truncate table > - > > Key: HIVE-16171 > URL: https://issues.apache.org/jira/browse/HIVE-16171 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16171.01.patch, HIVE-16171.02.patch, > HIVE-16171.03.patch, HIVE-16171.04.patch, HIVE-16171.05.patch > > > Need to support truncate table for replication. Key points to note. > 1. For non-partitioned table, truncate table will remove all the rows from > the table. > 2. For partitioned tables, need to consider how truncate behaves if truncate > a partition or the whole table. > 3. Bootstrap load with truncate table must work as it is just > loadTable/loadPartition with empty dataset. > 4. It is suggested to re-use the alter table/alter partition events to handle > truncate. > 5. Need to consider the case where insert event happens before truncate table > which needs to see their data files through change management. The data files > should be recycled to the cmroot path before trashing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16197) Incremental insert into a partitioned table doesn't get replicated.
[ https://issues.apache.org/jira/browse/HIVE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16197: Status: Patch Available (was: Open) > Incremental insert into a partitioned table doesn't get replicated. > --- > > Key: HIVE-16197 > URL: https://issues.apache.org/jira/browse/HIVE-16197 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16197.01.patch, HIVE-16197.02.patch, > HIVE-16197.03.patch, HIVE-16197.04.patch > > > Insert to a partitioned table doesn't replicate properly in case of > incremental dump/load. Few key points to be noted. > 1. If insert command itself created the new partition, then the inserted row > is replicated. But the subsequent inserts into the same table doesn't get > replicated. > 2. If the partition is created using ALTER TABLE command, then none of the > inserted rows to this partition is getting replicated. However, the partition > metadata is getting replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (HIVE-16197) Incremental insert into a partitioned table doesn't get replicated.
[ https://issues.apache.org/jira/browse/HIVE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966346#comment-15966346 ] Sankar Hariappan edited comment on HIVE-16197 at 4/12/17 6:26 PM: -- HIVE-16197.04.patch: Fixed the comments from [~sushanth]. - Mark the replace flag as optional in InsertEventData to avoid compatibility issue with older clients. - The comment for INSERT INTO partition was confusing as it was mentioned that insert into is supported on one partition. So, corrected the comment. Request [~sushanth] to review the new patch. was (Author: sankarh): HIVE-16197.04.patch: Fixed the comments from [~sushanth]. - Mark the replace flag as optional in InsertEventData to avoid compatibility issue with older clients. - The comment for INSERT INTO partition was confusing as it was mentioned that insert into is supported on one partition. So, corrected the comment. > Incremental insert into a partitioned table doesn't get replicated. > --- > > Key: HIVE-16197 > URL: https://issues.apache.org/jira/browse/HIVE-16197 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16197.01.patch, HIVE-16197.02.patch, > HIVE-16197.03.patch, HIVE-16197.04.patch > > > Insert to a partitioned table doesn't replicate properly in case of > incremental dump/load. Few key points to be noted. > 1. If insert command itself created the new partition, then the inserted row > is replicated. But the subsequent inserts into the same table doesn't get > replicated. > 2. If the partition is created using ALTER TABLE command, then none of the > inserted rows to this partition is getting replicated. However, the partition > metadata is getting replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16197) Incremental insert into a partitioned table doesn't get replicated.
[ https://issues.apache.org/jira/browse/HIVE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16197: Attachment: HIVE-16197.04.patch HIVE-16197.04.patch: Fixed the comments from [~sushanth]. - Mark the replace flag as optional in InsertEventData to avoid compatibility issue with older clients. - The comment for INSERT INTO partition was confusing as it was mentioned that insert into is supported on one partition. So, corrected the comment. > Incremental insert into a partitioned table doesn't get replicated. > --- > > Key: HIVE-16197 > URL: https://issues.apache.org/jira/browse/HIVE-16197 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16197.01.patch, HIVE-16197.02.patch, > HIVE-16197.03.patch, HIVE-16197.04.patch > > > Insert to a partitioned table doesn't replicate properly in case of > incremental dump/load. Few key points to be noted. > 1. If insert command itself created the new partition, then the inserted row > is replicated. But the subsequent inserts into the same table doesn't get > replicated. > 2. If the partition is created using ALTER TABLE command, then none of the > inserted rows to this partition is getting replicated. However, the partition > metadata is getting replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16197) Incremental insert into a partitioned table doesn't get replicated.
[ https://issues.apache.org/jira/browse/HIVE-16197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-16197: Status: Open (was: Patch Available) > Incremental insert into a partitioned table doesn't get replicated. > --- > > Key: HIVE-16197 > URL: https://issues.apache.org/jira/browse/HIVE-16197 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16197.01.patch, HIVE-16197.02.patch, > HIVE-16197.03.patch > > > Insert to a partitioned table doesn't replicate properly in case of > incremental dump/load. Few key points to be noted. > 1. If insert command itself created the new partition, then the inserted row > is replicated. But the subsequent inserts into the same table doesn't get > replicated. > 2. If the partition is created using ALTER TABLE command, then none of the > inserted rows to this partition is getting replicated. However, the partition > metadata is getting replicated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Status: Open (was: Patch Available) > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Attachment: HIVE-13567.04.patch > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-13567) Auto-gather column stats - phase 2
[ https://issues.apache.org/jira/browse/HIVE-13567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-13567: --- Status: Patch Available (was: Open) > Auto-gather column stats - phase 2 > -- > > Key: HIVE-13567 > URL: https://issues.apache.org/jira/browse/HIVE-13567 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-13567.01.patch, HIVE-13567.02.patch, > HIVE-13567.03.patch, HIVE-13567.04.patch > > > in phase 2, we are going to set auto-gather column on as default. This needs > to update golden files. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15037) Figure out whether the SCSL 3.0 license is acceptable
[ https://issues.apache.org/jira/browse/HIVE-15037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966338#comment-15966338 ] Alan Gates commented on HIVE-15037: --- I have filed LEGAL-298 to ask for advice on whether this jar is acceptable. > Figure out whether the SCSL 3.0 license is acceptable > - > > Key: HIVE-15037 > URL: https://issues.apache.org/jira/browse/HIVE-15037 > Project: Hive > Issue Type: Bug > Components: distribution >Affects Versions: 2.2.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > > The jar jta included in Hive's binary distribution is licensed under SCSL 3.0 > (http://jcp.org/aboutJava/communityprocess/SCSL3.0.rtf). This isn't listed > on the Apache page regarding licenses > (https://www.apache.org/legal/resolved.html). We need to figure out if this > is acceptable or not. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16171) Support replication of truncate table
[ https://issues.apache.org/jira/browse/HIVE-16171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966335#comment-15966335 ] Sankar Hariappan commented on HIVE-16171: - Hi [~ekoifman], For this truncate table support with change management, there are few changes done to update the table stats. Could you please have a look at it? I shall see couple of test failures due to this change where the result-logs have differences. But, I believe, the new code is fine and the result file should be updated for these 2 failed tests. Please give your opinion if I'm wrong. Brief description of my change: Base: 1. Update Stats (RowNum=0, NumOfFiles=0 etc). 2. AlterTable with new Stats. -- This effectively regenerate the stats from the table again and hence set to older values. 3. TruncateTable (Trash data files) New Code: 1. TruncateTable (Trash data files) 2. Update Stats (RowNum=0, NumOfFiles=0 etc). 3. AlterTable with new Stats. This was tested and is working fine. cc: [~thejas], [~sushanth] > Support replication of truncate table > - > > Key: HIVE-16171 > URL: https://issues.apache.org/jira/browse/HIVE-16171 > Project: Hive > Issue Type: Sub-task > Components: repl >Affects Versions: 2.1.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan > Labels: DR > Attachments: HIVE-16171.01.patch, HIVE-16171.02.patch, > HIVE-16171.03.patch, HIVE-16171.04.patch, HIVE-16171.05.patch > > > Need to support truncate table for replication. Key points to note. > 1. For non-partitioned table, truncate table will remove all the rows from > the table. > 2. For partitioned tables, need to consider how truncate behaves if truncate > a partition or the whole table. > 3. Bootstrap load with truncate table must work as it is just > loadTable/loadPartition with empty dataset. > 4. It is suggested to re-use the alter table/alter partition events to handle > truncate. > 5. Need to consider the case where insert event happens before truncate table > which needs to see their data files through change management. The data files > should be recycled to the cmroot path before trashing it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-7575) GetTables thrift call is very slow
[ https://issues.apache.org/jira/browse/HIVE-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966333#comment-15966333 ] Thejas M Nair commented on HIVE-7575: - FYI [~holman.l...@gmail.com] > GetTables thrift call is very slow > -- > > Key: HIVE-7575 > URL: https://issues.apache.org/jira/browse/HIVE-7575 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 0.12.0, 0.13.0 >Reporter: Ashu Pachauri >Assignee: Navis > Fix For: 2.0.0 > > Attachments: HIVE-7575.1.patch.txt, HIVE-7575.2.patch.txt, > HIVE-7575.3.patch.txt, HIVE-7575.4.patch.txt, HIVE-7575.5.patch.txt, > HIVE-7575.6.patch.txt, HIVE-7575.7.patch.txt > > > The GetTables thrift call takes a long time when the number of table is large. > With around 5000 tables, the call takes around 80 seconds compared to a "Show > Tables" query on the same HiveServer2 instance which takes 3-7 seconds. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16415) Add blobstore tests for insertion of zero rows
[ https://issues.apache.org/jira/browse/HIVE-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966308#comment-15966308 ] Sergio Peña commented on HIVE-16415: [~poeppt] Please create a follow-up jira that fixes the issue that Ashutosh pointed out. I will find someone who would like to fix it and add a test case in this same .q file. Let me know which jira you created btw. > Add blobstore tests for insertion of zero rows > -- > > Key: HIVE-16415 > URL: https://issues.apache.org/jira/browse/HIVE-16415 > Project: Hive > Issue Type: Test > Components: Tests >Affects Versions: 2.1.1 >Reporter: Thomas Poepping >Assignee: Thomas Poepping > Attachments: HIVE-16415.patch > > > This patch introduces two regression tests into the hive-blobstore qtest > module: zero_rows_hdfs.q and zero_rows_blobstore.q. These test doing INSERT > commands with a WHERE clause where the condition of the WHERE clause causes > zero rows to be considered. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16419) Exclude hadoop related classes for JDBC stabdalone jar
[ https://issues.apache.org/jira/browse/HIVE-16419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966297#comment-15966297 ] Tao Li commented on HIVE-16419: --- This change is basically reverting change from HIVE-14837. The hadoop jars do need to be added in classpath with this change. The reason that we are reverting this change is because we were running some issues with the shaded classes. For example, the un-relocated class name is specified in core-site.xml, while the standalone jar contains the relocated/renamed classes, which causes CNF issue. > Exclude hadoop related classes for JDBC stabdalone jar > -- > > Key: HIVE-16419 > URL: https://issues.apache.org/jira/browse/HIVE-16419 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Tao Li >Assignee: Tao Li >Priority: Blocker > Attachments: HIVE-16419.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-16316) Prepare master branch for 3.0.0 development.
[ https://issues.apache.org/jira/browse/HIVE-16316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966288#comment-15966288 ] Pengcheng Xiong commented on HIVE-16316: [~wzheng], can u apply this patch and see if it works in your scenario? thanks. > Prepare master branch for 3.0.0 development. > > > Key: HIVE-16316 > URL: https://issues.apache.org/jira/browse/HIVE-16316 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Fix For: 3.0.0 > > Attachments: HIVE-16316-addendum.patch, HIVE-16316.patch > > > master branch is now being used for 3.0.0 development. The build files will > need to reflect this change. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16387) Fix failing test org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
[ https://issues.apache.org/jira/browse/HIVE-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16387: --- Labels: 2.3.0 (was: ) > Fix failing test org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData > --- > > Key: HIVE-16387 > URL: https://issues.apache.org/jira/browse/HIVE-16387 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: 2.3.0 > Fix For: 2.3.0 > > Attachments: HIVE-16387.01.patch, HIVE-16387.02.patch, > HIVE-16387.03.patch, HIVE-16387.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HIVE-16387) Fix failing test org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData
[ https://issues.apache.org/jira/browse/HIVE-16387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-16387: --- Affects Version/s: 2.3.0 > Fix failing test org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData > --- > > Key: HIVE-16387 > URL: https://issues.apache.org/jira/browse/HIVE-16387 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.3.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Labels: 2.3.0 > Fix For: 2.3.0 > > Attachments: HIVE-16387.01.patch, HIVE-16387.02.patch, > HIVE-16387.03.patch, HIVE-16387.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
