[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273096&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273096
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:03
Start Date: 08/Jul/19 07:03
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300947209
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/load/DumpMetaData.java
##
@@ -85,48 +85,27 @@ private void readReplScope(String line) throws IOException
{
}
String[] lineContents = line.split("\t");
-if (lineContents.length < 1) {
- return;
-}
-
replScope = new ReplScope();
-
-LOG.info("Read ReplScope: Set Db Name: {}.", lineContents[0]);
-replScope.setDbName(lineContents[0]);
-
-// Read/set include and exclude tables list.
-int idx = readReplScopeTablesList(lineContents, 1, true);
-readReplScopeTablesList(lineContents, idx, false);
- }
-
- private int readReplScopeTablesList(String[] lineContents, int startIdx,
boolean includeList)
- throws IOException {
-// If the list doesn't exist, then return.
-if (startIdx >= lineContents.length) {
- return startIdx;
-}
-
-// Each tables list should start with "{" and ends with "}"
-if (!"{".equals(lineContents[startIdx])) {
- throw new IOException("Invalid repl tables list data in dump metadata
file. Missing \"{\".");
-}
-
-ListtableNames = new ArrayList<>();
-for (int i = (startIdx + 1); i < lineContents.length; i++) {
- String value = lineContents[i];
- if ("}".equals(value)) {
-if (includeList) {
- LOG.info("Read ReplScope: Set Include Table Names: {}.", tableNames);
- replScope.setIncludedTablePatterns(tableNames);
-} else {
- LOG.info("Read ReplScope: Set Exclude Table Names: {}.", tableNames);
- replScope.setExcludedTablePatterns(tableNames);
-}
-return (i + 1);
+for (int idx = 0; idx < lineContents.length; idx++) {
+ String value = lineContents[idx];
+ switch (idx) {
+case 0:
+ LOG.info("Read ReplScope: Set Db Name: {}.", value);
+ replScope.setDbName(value);
+ break;
+case 1:
+ LOG.info("Read ReplScope: Include table name list: {}.", value);
+ replScope.setIncludedTablePatterns(value);
+ break;
+case 2:
+ LOG.info("Read ReplScope: Exclude table name list: {}.", value);
+ replScope.setExcludedTablePatterns(value);
+ break;
+default:
+ LOG.warn("Found invalid info " + value + " in the repl dump file
while reading ReplScope");
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273096)
Time Spent: 1.5h (was: 1h 20m)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273097&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273097
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:04
Start Date: 08/Jul/19 07:04
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300947507
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplScope.java
##
@@ -31,10 +29,10 @@
private Pattern dbNamePattern;
// Include and exclude table names/patterns exist only for REPL DUMP.
- private List includedTableNames;
- private List excludedTableNames;
- private List includedTableNamePatterns;
- private List excludedTableNamePatterns;
+ private String includedTableNames;
+ private String excludedTableNames;
+ private Pattern includedTableNamePatterns;
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273097)
Time Spent: 1h 40m (was: 1.5h)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273099&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273099
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:05
Start Date: 08/Jul/19 07:05
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300947693
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -313,8 +313,8 @@ public void testIncorrectTablePolicyInReplDump() throws
Throwable {
}
Assert.assertTrue(failed);
-// Invalid pattern where we didn't enclose table pattern within single or
double quotes.
-replPolicy = primaryDbName + ".[t1].[t2]";
+// Invalid pattern where where table list is empty.
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273099)
Time Spent: 2h (was: 1h 50m)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273098
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:05
Start Date: 08/Jul/19 07:05
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300947660
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplScope.java
##
@@ -93,26 +91,13 @@ public boolean tableIncludedInReplScope(final String
tableName) {
return (inTableIncludedList(tableName) && !inTableExcludedList(tableName));
}
- private List compilePatterns(List patterns) {
-if (patterns == null || patterns.isEmpty()) {
+ private Pattern compilePatterns(String pattern) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273098)
Time Spent: 1h 50m (was: 1h 40m)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273100&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273100
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:07
Start Date: 08/Jul/19 07:07
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300948134
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -313,8 +313,8 @@ public void testIncorrectTablePolicyInReplDump() throws
Throwable {
}
Assert.assertTrue(failed);
-// Invalid pattern where we didn't enclose table pattern within single or
double quotes.
-replPolicy = primaryDbName + ".[t1].[t2]";
+// Invalid pattern where where table list is empty.
+replPolicy = primaryDbName + ".''.'t2'";
Review comment:
it's confusing ..if empty list ..that means include all ? if yes, then user
has to give empty exclude list or exclude list as * for dump only function ? i
think for simplicity we should not allow this special case ..if user wants then
they can use regex to exclude all tables
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273100)
Time Spent: 2h 10m (was: 2h)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880075#comment-16880075
]
Hive QA commented on HIVE-21958:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
30s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 15 new + 85 unchanged - 0
fixed = 100 total (was 85) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
15s{color} | {color:red} ql generated 9 new + 2243 unchanged - 9 fixed = 2252
total (was 2252) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA235 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
48087-48100] |
| | Dead store to LA29_128 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47827] |
| | Dead store to LA29_130 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47840] |
| | Dead store to LA29_132 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47853] |
| | Dead store to LA29_134 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47866] |
| | Dead store to LA29_136 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47879] |
| | Dead store to LA29_138 in
org.apache.hadoop.hive.ql.parse.HiveParser$DF
[jira] [Updated] (HIVE-21948) Implement parallell processing in Pre Upgrade Tool
[ https://issues.apache.org/jira/browse/HIVE-21948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21948: -- Status: Patch Available (was: Open) > Implement parallell processing in Pre Upgrade Tool > -- > > Key: HIVE-21948 > URL: https://issues.apache.org/jira/browse/HIVE-21948 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21948.1.patch > > > Pre Upgrade Tool scans for all databases and tables in the warehouse > sequentially which can be very slow in case of lots of tables. > Example: It took the process 8-10 hours to complete on ~500k tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21948) Implement parallell processing in Pre Upgrade Tool
[ https://issues.apache.org/jira/browse/HIVE-21948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21948: -- Attachment: HIVE-21948.1.patch > Implement parallell processing in Pre Upgrade Tool > -- > > Key: HIVE-21948 > URL: https://issues.apache.org/jira/browse/HIVE-21948 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21948.1.patch > > > Pre Upgrade Tool scans for all databases and tables in the warehouse > sequentially which can be very slow in case of lots of tables. > Example: It took the process 8-10 hours to complete on ~500k tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[ https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21923: Attachment: HIVE-21923.05.patch > Vectorized MapJoin may miss results when only the join key is selected > -- > > Key: HIVE-21923 > URL: https://issues.apache.org/jira/browse/HIVE-21923 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21923.01.patch, HIVE-21923.02.patch, > HIVE-21923.03.patch, HIVE-21923.04.patch, HIVE-21923.05.patch > > > HIVE-21189 have introduced some resultset changes > in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out > https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21713) Explain consistency Task/Operator
[ https://issues.apache.org/jira/browse/HIVE-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880091#comment-16880091 ] Zoltan Haindrich commented on HIVE-21713: - wow :) what a great overview about these things! sorry for getting back to this just now...I've overlooked the notification... there are quite a few "very" interesting onse; like "keys" for TopNKeyOperator :D I think [~mnarayanan2018] you have exposed this problem very well; some of them are very exotic - I think the best would be if you could came up with a logical naming scheme and propose to rename them! > Explain consistency Task/Operator > - > > Key: HIVE-21713 > URL: https://issues.apache.org/jira/browse/HIVE-21713 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Manoj Narayanan >Priority: Major > > there seems to be some inconsistency what gets printed in the explain; > for example "MoveTask" is shown as "Move Operator" in the explain > Becase Tasks and Operators are basically different things - this should be > corrected as it may cause confusion -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880094#comment-16880094
]
Hive QA commented on HIVE-21958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973875/HIVE-21958.03.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16361 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=222)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17902/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17902/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17902/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973875 - PreCommit-HIVE-Build
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-21225: Attachment: HIVE-21225.8.patch > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-21225: Status: Open (was: Patch Available) > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273126&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273126
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:47
Start Date: 08/Jul/19 07:47
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300961188
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -313,8 +313,8 @@ public void testIncorrectTablePolicyInReplDump() throws
Throwable {
}
Assert.assertTrue(failed);
-// Invalid pattern where we didn't enclose table pattern within single or
double quotes.
-replPolicy = primaryDbName + ".[t1].[t2]";
+// Invalid pattern where where table list is empty.
+replPolicy = primaryDbName + ".''.'t2'";
Review comment:
Yes, agreed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273126)
Time Spent: 2h 20m (was: 2h 10m)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[ https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-21225: Status: Patch Available (was: Open) > ACID: getAcidState() should cache a recursive dir listing locally > - > > Key: HIVE-21225 > URL: https://issues.apache.org/jira/browse/HIVE-21225 > Project: Hive > Issue Type: Improvement > Components: Transactions >Reporter: Gopal V >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch, > HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch, > HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch, > HIVE-21225.7.patch, HIVE-21225.8.patch, async-pid-44-2.svg > > > Currently getAcidState() makes 3 calls into the FS api which could be > answered by making a single recursive listDir call and reusing the same data > to check for isRawFormat() and isValidBase(). > All delta operations for a single partition can go against a single listed > directory snapshot instead of interacting with the NameNode or ObjectStore > within the inner loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273129&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273129
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:48
Start Date: 08/Jul/19 07:48
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300961527
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -263,12 +263,12 @@ public void testIncorrectTablePolicyInReplDump() throws
Throwable {
String[] replicatedTables = new String[] {};
boolean failed;
String[] invalidReplPolicies = new String[] {
-primaryDbName + ".t1.t2", // Two explicit table names not allowed.
-primaryDbName + ".['t1'].t2", // Table name and include list not
allowed.
-primaryDbName + ".t1.['t2']", // Table name and exclude list not
allowed.
-primaryDbName + ".[t1].t2", // Table name and include list not allowed.
-primaryDbName + ".['t1+'].", // Abrubtly ended dot.
-primaryDbName + "..[]" // Multiple dots
+primaryDbName + ".t1.t2", // Didn't enclose table pattern within
single quotes.
+primaryDbName + ".'t1'.t2", // Table name and include list not allowed.
+primaryDbName + ".t1.'t2'", // Table name and exclude list not allowed.
+primaryDbName + ".'t1+'.", // Abrubtly ended dot.
Review comment:
We can add the empty include/exclude list in this test case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273129)
Time Spent: 2h 40m (was: 2.5h)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273127&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273127
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:48
Start Date: 08/Jul/19 07:48
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300961527
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -263,12 +263,12 @@ public void testIncorrectTablePolicyInReplDump() throws
Throwable {
String[] replicatedTables = new String[] {};
boolean failed;
String[] invalidReplPolicies = new String[] {
-primaryDbName + ".t1.t2", // Two explicit table names not allowed.
-primaryDbName + ".['t1'].t2", // Table name and include list not
allowed.
-primaryDbName + ".t1.['t2']", // Table name and exclude list not
allowed.
-primaryDbName + ".[t1].t2", // Table name and include list not allowed.
-primaryDbName + ".['t1+'].", // Abrubtly ended dot.
-primaryDbName + "..[]" // Multiple dots
+primaryDbName + ".t1.t2", // Didn't enclose table pattern within
single quotes.
+primaryDbName + ".'t1'.t2", // Table name and include list not allowed.
+primaryDbName + ".t1.'t2'", // Table name and exclude list not allowed.
+primaryDbName + ".'t1+'.", // Abrubtly ended dot.
Review comment:
We can add the empty include/exclude list in this test case.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273127)
Time Spent: 2.5h (was: 2h 20m)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273131&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273131
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:49
Start Date: 08/Jul/19 07:49
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300961926
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -313,8 +313,8 @@ public void testIncorrectTablePolicyInReplDump() throws
Throwable {
}
Assert.assertTrue(failed);
-// Invalid pattern where we didn't enclose table pattern within single or
double quotes.
-replPolicy = primaryDbName + ".[t1].[t2]";
+// Invalid pattern where where table list is empty.
+replPolicy = primaryDbName + ".''.'t2'";
Review comment:
Check empty string for exclude list as well. Also add a test with more than
2 dots in the policy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273131)
Time Spent: 3h (was: 2h 50m)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273130&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273130
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 07:49
Start Date: 08/Jul/19 07:49
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300961926
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -313,8 +313,8 @@ public void testIncorrectTablePolicyInReplDump() throws
Throwable {
}
Assert.assertTrue(failed);
-// Invalid pattern where we didn't enclose table pattern within single or
double quotes.
-replPolicy = primaryDbName + ".[t1].[t2]";
+// Invalid pattern where where table list is empty.
+replPolicy = primaryDbName + ".''.'t2'";
Review comment:
Check empty list for exclude list as well. Also add a test with more than 2
dots in the policy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273130)
Time Spent: 2h 50m (was: 2h 40m)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21957) Create temporary table like should omit transactional properties
[ https://issues.apache.org/jira/browse/HIVE-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21957: - Attachment: HIVE-21957.02.patch > Create temporary table like should omit transactional properties > > > Key: HIVE-21957 > URL: https://issues.apache.org/jira/browse/HIVE-21957 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21957.01.patch, HIVE-21957.02.patch > > > In case of create temporary table like queries, where the source table is > transactional, the transactional properties should not be copied over to the > new table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?focusedWorklogId=273143&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273143
]
ASF GitHub Bot logged work on HIVE-21958:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 08:15
Start Date: 08/Jul/19 08:15
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #705: HIVE-21958 :
The list of table expression in the inclusion and exclusion list should be
separated by '|' instead of comma.
URL: https://github.com/apache/hive/pull/705#discussion_r300970848
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java
##
@@ -313,8 +313,8 @@ public void testIncorrectTablePolicyInReplDump() throws
Throwable {
}
Assert.assertTrue(failed);
-// Invalid pattern where we didn't enclose table pattern within single or
double quotes.
-replPolicy = primaryDbName + ".[t1].[t2]";
+// Invalid pattern where where table list is empty.
+replPolicy = primaryDbName + ".''.'t2'";
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273143)
Time Spent: 3h 10m (was: 3h)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Status: Open (was: Patch Available) > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch, HIVE-21938.4.patch, HIVE-21938.5.patch, HIVE-21938.6.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Status: Patch Available (was: Open) > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch, HIVE-21938.4.patch, HIVE-21938.5.patch, HIVE-21938.6.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Attachment: HIVE-21938.6.patch > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch, HIVE-21938.4.patch, HIVE-21938.5.patch, HIVE-21938.6.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-21958:
---
Attachment: HIVE-21958.04.patch
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch, HIVE-21958.04.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-21958:
---
Status: Open (was: Patch Available)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch, HIVE-21958.04.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mahesh kumar behera updated HIVE-21958:
---
Status: Patch Available (was: Open)
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch, HIVE-21958.04.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21948) Implement parallel processing in Pre Upgrade Tool
[ https://issues.apache.org/jira/browse/HIVE-21948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21948: -- Summary: Implement parallel processing in Pre Upgrade Tool (was: Implement parallell processing in Pre Upgrade Tool) > Implement parallel processing in Pre Upgrade Tool > - > > Key: HIVE-21948 > URL: https://issues.apache.org/jira/browse/HIVE-21948 > Project: Hive > Issue Type: Improvement > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21948.1.patch > > > Pre Upgrade Tool scans for all databases and tables in the warehouse > sequentially which can be very slow in case of lots of tables. > Example: It took the process 8-10 hours to complete on ~500k tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21965) Implement parallel processing in HiveStrictManagedMigration
[ https://issues.apache.org/jira/browse/HIVE-21965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-21965: - > Implement parallel processing in HiveStrictManagedMigration > --- > > Key: HIVE-21965 > URL: https://issues.apache.org/jira/browse/HIVE-21965 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > > This process, kicked off from Ambari can take many days for systems with > 1000's of tables. The process needs to support parallel execution as it > iterates through the Databases and Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21224) Upgrade tests JUnit3 to JUnit4
[ https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Pusztahazi updated HIVE-21224: Status: Open (was: Patch Available) > Upgrade tests JUnit3 to JUnit4 > -- > > Key: HIVE-21224 > URL: https://issues.apache.org/jira/browse/HIVE-21224 > Project: Hive > Issue Type: Improvement >Reporter: Bruno Pusztahazi >Assignee: Bruno Pusztahazi >Priority: Major > Attachments: HIVE-21224.1.patch, HIVE-21224.10.patch, > HIVE-21224.11.patch, HIVE-21224.12.patch, HIVE-21224.2.patch, > HIVE-21224.3.patch, HIVE-21224.4.patch, HIVE-21224.5.patch, > HIVE-21224.6.patch, HIVE-21224.7.patch, HIVE-21224.8.patch, HIVE-21224.9.patch > > > Old JUnit3 tests should be upgraded to JUnit4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21224) Upgrade tests JUnit3 to JUnit4
[ https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Pusztahazi updated HIVE-21224: Attachment: HIVE-21224.13.patch > Upgrade tests JUnit3 to JUnit4 > -- > > Key: HIVE-21224 > URL: https://issues.apache.org/jira/browse/HIVE-21224 > Project: Hive > Issue Type: Improvement >Reporter: Bruno Pusztahazi >Assignee: Bruno Pusztahazi >Priority: Major > Attachments: HIVE-21224.1.patch, HIVE-21224.10.patch, > HIVE-21224.11.patch, HIVE-21224.12.patch, HIVE-21224.13.patch, > HIVE-21224.2.patch, HIVE-21224.3.patch, HIVE-21224.4.patch, > HIVE-21224.5.patch, HIVE-21224.6.patch, HIVE-21224.7.patch, > HIVE-21224.8.patch, HIVE-21224.9.patch > > > Old JUnit3 tests should be upgraded to JUnit4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880135#comment-16880135
]
Hive QA commented on HIVE-21637:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
30s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
179 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
57s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} beeline in master has 44 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue}
standalone-metastore/metastore-tools/metastore-benchmarks in master has 3
extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
49s{color} | {color:blue} itests/util in master has 44 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} storage-api: The patch generated 1 new + 5 unchanged -
0 fixed = 6 total (was 5) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 9 new + 498 unchanged - 2 fixed = 507 total (was 500) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 164 new + 2193 unchanged - 65 fixed = 2357 total (was 2258) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 27 new + 1040 unchanged - 10
fixed = 1067 total (was 1050) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} standalone-metastore/metastore-tools/tools-common: The
patch generated 5 new + 31 unchanged - 0 fixed = 36 total (was 31) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} itests/hcatalog-unit: The patch generated 2 new + 24
unchanged - 3 fixed = 26 total (was 27) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} itests/hive-unit: The patch generated 3 new + 163
unchanged - 1 fixed = 166 total (was 164) {color} |
| {color:red}-1{color} | {color:red} checkst
[jira] [Updated] (HIVE-21224) Upgrade tests JUnit3 to JUnit4
[ https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Pusztahazi updated HIVE-21224: Status: Patch Available (was: Open) > Upgrade tests JUnit3 to JUnit4 > -- > > Key: HIVE-21224 > URL: https://issues.apache.org/jira/browse/HIVE-21224 > Project: Hive > Issue Type: Improvement >Reporter: Bruno Pusztahazi >Assignee: Bruno Pusztahazi >Priority: Major > Attachments: HIVE-21224.1.patch, HIVE-21224.10.patch, > HIVE-21224.11.patch, HIVE-21224.12.patch, HIVE-21224.13.patch, > HIVE-21224.2.patch, HIVE-21224.3.patch, HIVE-21224.4.patch, > HIVE-21224.5.patch, HIVE-21224.6.patch, HIVE-21224.7.patch, > HIVE-21224.8.patch, HIVE-21224.9.patch > > > Old JUnit3 tests should be upgraded to JUnit4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21637) Synchronized metastore cache
[
https://issues.apache.org/jira/browse/HIVE-21637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880139#comment-16880139
]
Hive QA commented on HIVE-21637:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973878/HIVE-21637.20.patch
{color:green}SUCCESS:{color} +1 due to 10 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 538 failed/errored test(s), 15544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver
(batchId=279)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_insert_overwrite]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats2] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats4] (batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_stats5] (batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_vectorization_partition]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allow_change_col_type_par]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_partition_update_status]
(batchId=99)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_table_update_status_disable_bitvector]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_whole_partition]
(batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_locks]
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_dynamic_partitioned]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lock1] (batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lock2] (batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lock3] (batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_wide_table]
(batchId=96)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_2_exim_basic]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[repl_3_exim_metadata]
(batchId=63)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[set_metaconf]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_nonpart]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_sizebug]
(batchId=89)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udtf_get_sql_schema]
(batchId=77)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_dynamic]
(batchId=190)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_insert_partition_static]
(batchId=187)
org.apache.hadoop.hive.cli.TestLocalSparkCliDriver.org.apache.hadoop.hive.cli.TestLocalSparkCliDriver
(batchId=289)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=179)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_all_partitioned]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[get_splits_0]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_3]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=168)
org.apache.hadoo
[jira] [Assigned] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia reassigned HIVE-21966:
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880154#comment-16880154
]
Hive QA commented on HIVE-21225:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
3s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 21 new + 903 unchanged - 1
fixed = 924 total (was 904) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 27 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
22s{color} | {color:red} ql generated 1 new + 2252 unchanged - 0 fixed = 2253
total (was 2252) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Unread field:AcidUtils.java:[line 1409] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17904/dev-support/hive-personality.sh
|
| git revision | master / 67e515f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17904/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17904/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17904/yetus/new-findbugs-ql.html
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17904/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ACID: getAcidState() should cache a recursive dir listing locally
> -
>
> Key: HIVE-21225
> URL: https://issues.apache.org/jira/browse/HIVE-21225
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch,
> HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch,
> HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch,
> HIVE-212
[jira] [Commented] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880155#comment-16880155
]
Sankar Hariappan commented on HIVE-21958:
-
+1, pending tests
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch, HIVE-21958.04.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880183#comment-16880183
]
Hive QA commented on HIVE-21225:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973895/HIVE-21225.8.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 51 failed/errored test(s), 16361 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[delete_tmp_table]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_tmp_table]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_cttas] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[temp_table_options1]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin2]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin3]
(batchId=13)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_cttas]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[approx_distinct]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[delete_tmp_table]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_tmp_table]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lvj_mapjoin]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mm_exim]
(batchId=186)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[strict_managed_tables1]
(batchId=181)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[temp_table]
(batchId=185)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_nway_join]
(batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr_2]
(batchId=177)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_like_2]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_udf2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_union_null]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_insert_into_bucketed_table]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_mapjoin3]
(batchId=163)
org.apache.hadoop.hive.ql.TestTxnCommands.testDeleteIn (batchId=341)
org.apache.hadoop.hive.ql.TestTxnCommands.testMmExim (batchId=341)
org.apache.hadoop.hive.ql.TestTxnCommands2.updateDeletePartitioned (batchId=322)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.updateDeletePartitioned
(batchId=336)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testDeleteIn
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMmExim
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnExIm.testImport (batchId=322)
org.apache.hadoop.hive.ql.TestTxnExIm.testImportNoTarget (batchId=322)
org.apache.hadoop.hive.ql.TestTxnExIm.testMM (batchId=322)
org.apache.hadoop.hive.ql.TestTxnExIm.testMMCreate (batchId=322)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadData (batchId=298)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion
(batchId=298)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataUpdate (batchId=298)
org.apache.hadoop.hive.ql.TestTxnLoadData.testMultiStatement (batchId=298)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testCompactStatsGather (batchId=322)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testEmptyCompactionResult
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testCompactStatsGather
(batchId=322)
org.apache.hadoop.hive.ql.TestTxnNoBucketsVectorized.testEmptyCompactionResult
(batchId=322)
org.apache.hadoop.hive.ql.io.TestAcidUtils.testObsoleteOriginals (batchId=310)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithPartitionDirMissing
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testDumpAcidTableWithTableDirMissing
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testMultiDBTxn
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithPartitionDirMissing
(batchId=262)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testDumpAcidTableWithTableDirMissing
(batchId=262)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testMultiDBTxn
(batchId=262)
org.apache.hadoop.hive.ql.parse.TestReplicationS
[jira] [Work logged] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?focusedWorklogId=273207&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273207
]
ASF GitHub Bot logged work on HIVE-21966:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 10:32
Start Date: 08/Jul/19 10:32
Worklog Time Spent: 10m
Work Description: ShubhamChaurasia commented on pull request #706:
HIVE-21966: Llap external client - Arrow Serializer throws
ArrayIndexOutOfBoundsException
URL: https://github.com/apache/hive/pull/706
This patch readjusts selected[] and size in VectorizedRowBatch according to
the actual vectors inside MultiValuedColumnVector.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273207)
Time Spent: 10m
Remaining Estimate: 0h
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21966:
--
Labels: pull-request-available (was: )
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia updated HIVE-21966:
-
Attachment: HIVE-21966.1.patch
Status: Patch Available (was: Open)
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21948) Implement parallel processing in Pre Upgrade Tool
[
https://issues.apache.org/jira/browse/HIVE-21948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880206#comment-16880206
]
Hive QA commented on HIVE-21948:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
22s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} upgrade-acid/pre-upgrade: The patch generated 46 new +
54 unchanged - 14 fixed = 100 total (was 68) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
29s{color} | {color:red} upgrade-acid/pre-upgrade generated 1 new + 1 unchanged
- 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:upgrade-acid/pre-upgrade |
| | Null passed for non-null parameter of new
java.util.concurrent.ForkJoinPool(int,
ForkJoinPool$ForkJoinWorkerThreadFactory, Thread$UncaughtExceptionHandler,
boolean) in
org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal()
At PreUpgradeTool.java:of new java.util.concurrent.ForkJoinPool(int,
ForkJoinPool$ForkJoinWorkerThreadFactory, Thread$UncaughtExceptionHandler,
boolean) in
org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal()
At PreUpgradeTool.java:[line 286] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17905/dev-support/hive-personality.sh
|
| git revision | master / 67e515f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17905/yetus/diff-checkstyle-upgrade-acid_pre-upgrade.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17905/yetus/new-findbugs-upgrade-acid_pre-upgrade.html
|
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17905/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement parallel processing in Pre Upgrade Tool
> -
>
> Key: HIVE-21948
> URL: https://issues.apache.org/jira/browse/HIVE-21948
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21948.1.patch
>
>
> Pre Upgrade Tool scans for all databases and tables in the warehouse
> sequentially which can be very slow in case of lots of t
[jira] [Updated] (HIVE-21965) Implement parallel processing in HiveStrictManagedMigration
[ https://issues.apache.org/jira/browse/HIVE-21965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21965: -- Status: Patch Available (was: Open) > Implement parallel processing in HiveStrictManagedMigration > --- > > Key: HIVE-21965 > URL: https://issues.apache.org/jira/browse/HIVE-21965 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21965.1.patch > > > This process, kicked off from Ambari can take many days for systems with > 1000's of tables. The process needs to support parallel execution as it > iterates through the Databases and Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21965) Implement parallel processing in HiveStrictManagedMigration
[ https://issues.apache.org/jira/browse/HIVE-21965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21965: -- Attachment: HIVE-21965.1.patch > Implement parallel processing in HiveStrictManagedMigration > --- > > Key: HIVE-21965 > URL: https://issues.apache.org/jira/browse/HIVE-21965 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21965.1.patch > > > This process, kicked off from Ambari can take many days for systems with > 1000's of tables. The process needs to support parallel execution as it > iterates through the Databases and Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21908) LlapDaemon node status should be reflected in the metrics
[ https://issues.apache.org/jira/browse/HIVE-21908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880207#comment-16880207 ] Peter Vary commented on HIVE-21908: --- +1 > LlapDaemon node status should be reflected in the metrics > - > > Key: HIVE-21908 > URL: https://issues.apache.org/jira/browse/HIVE-21908 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21908.01.patch, HIVE-21908.02.patch, > HIVE-21908.03.patch, HIVE-21908.04.patch, HIVE-21908.05.patch, > HIVE-21908.06.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > When we enable/disable a node it should be reflected in the LlapDaemon > metrics, so the administrator can act upon the disabled nodes. They can > manually check the status and either reenable them by restart, or fix the > existing issues causing the problems -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21909) Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity when scheduling new tasks
[ https://issues.apache.org/jira/browse/HIVE-21909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880208#comment-16880208 ] Peter Vary commented on HIVE-21909: --- +1 > Publish the LLAP Daemon capacity through ZooKeeper and honor the capacity > when scheduling new tasks > --- > > Key: HIVE-21909 > URL: https://issues.apache.org/jira/browse/HIVE-21909 > Project: Hive > Issue Type: Sub-task > Components: llap >Reporter: Peter Vary >Assignee: Antal Sinkovits >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21909.01.patch, HIVE-21909.02.patch, > HIVE-21909.03.patch, HIVE-21909.04.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > > The changes in LlapDaemonNode status (or executor number) should be reflected > in the ZooKeeper information, and we should make sure TezAMs are reading and > honor the changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?focusedWorklogId=273226&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273226
]
ASF GitHub Bot logged work on HIVE-21966:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 11:17
Start Date: 08/Jul/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #706: HIVE-21966:
Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
URL: https://github.com/apache/hive/pull/706#discussion_r301041379
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java
##
@@ -365,27 +369,74 @@ private void writeStruct(NonNullableStructVector
arrowVector, StructColumnVector
}
}
- private void writeList(ListVector arrowVector, ListColumnVector hiveVector,
ListTypeInfo typeInfo, int size,
+// selected[] points to the valid/filtered/selected records at row level.
+// for MultiValuedColumnVector such as ListColumnVector one record of
vector points to multiple nested records.
+// In child vectors we get these records in exploded manner i.e. the
number of records in child vectors can have size more
+// than actual the VectorizedRowBatch, consequently selected[] also needs
to be readjusted.
+// This method creates a shallow copy of VectorizedRowBatch with corrected
size and selected[]
+
+private static VectorizedRowBatch
correctSelectedAndSize(VectorizedRowBatch sourceVrb,
+ ListColumnVector
listColumnVector) {
+
+VectorizedRowBatch vrb = new VectorizedRowBatch(sourceVrb.numCols,
sourceVrb.size);
+vrb.cols = sourceVrb.cols;
+vrb.endOfFile = sourceVrb.endOfFile;
+vrb.projectedColumns = sourceVrb.projectedColumns;
+vrb.projectionSize = sourceVrb.projectionSize;
+vrb.selectedInUse = sourceVrb.selectedInUse;
Review comment:
Need to set vrb.setPartitionInfo() as well.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273226)
Time Spent: 20m (was: 10m)
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlass
[jira] [Work logged] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?focusedWorklogId=273227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273227
]
ASF GitHub Bot logged work on HIVE-21966:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 11:17
Start Date: 08/Jul/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #706: HIVE-21966:
Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
URL: https://github.com/apache/hive/pull/706#discussion_r301039505
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java
##
@@ -365,27 +369,74 @@ private void writeStruct(NonNullableStructVector
arrowVector, StructColumnVector
}
}
- private void writeList(ListVector arrowVector, ListColumnVector hiveVector,
ListTypeInfo typeInfo, int size,
+// selected[] points to the valid/filtered/selected records at row level.
+// for MultiValuedColumnVector such as ListColumnVector one record of
vector points to multiple nested records.
+// In child vectors we get these records in exploded manner i.e. the
number of records in child vectors can have size more
+// than actual the VectorizedRowBatch, consequently selected[] also needs
to be readjusted.
+// This method creates a shallow copy of VectorizedRowBatch with corrected
size and selected[]
+
+private static VectorizedRowBatch
correctSelectedAndSize(VectorizedRowBatch sourceVrb,
+ ListColumnVector
listColumnVector) {
+
+VectorizedRowBatch vrb = new VectorizedRowBatch(sourceVrb.numCols,
sourceVrb.size);
+vrb.cols = sourceVrb.cols;
+vrb.endOfFile = sourceVrb.endOfFile;
+vrb.projectedColumns = sourceVrb.projectedColumns;
+vrb.projectionSize = sourceVrb.projectionSize;
+vrb.selectedInUse = sourceVrb.selectedInUse;
+
+int correctedSize = 0;
+final int[] srcVrbSelected = sourceVrb.selected;
+for (int i = 0; i < sourceVrb.size; i++) {
+correctedSize += listColumnVector.lengths[srcVrbSelected[i]];
Review comment:
Used 4 space tab instead of 2 space one. Check for other places too.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273227)
Time Spent: 0.5h (was: 20m)
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT I
[jira] [Work logged] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?focusedWorklogId=273228&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273228
]
ASF GitHub Bot logged work on HIVE-21966:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 11:17
Start Date: 08/Jul/19 11:17
Worklog Time Spent: 10m
Work Description: sankarh commented on pull request #706: HIVE-21966:
Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
URL: https://github.com/apache/hive/pull/706#discussion_r301039096
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java
##
@@ -365,27 +369,74 @@ private void writeStruct(NonNullableStructVector
arrowVector, StructColumnVector
}
}
- private void writeList(ListVector arrowVector, ListColumnVector hiveVector,
ListTypeInfo typeInfo, int size,
+// selected[] points to the valid/filtered/selected records at row level.
+// for MultiValuedColumnVector such as ListColumnVector one record of
vector points to multiple nested records.
+// In child vectors we get these records in exploded manner i.e. the
number of records in child vectors can have size more
+// than actual the VectorizedRowBatch, consequently selected[] also needs
to be readjusted.
+// This method creates a shallow copy of VectorizedRowBatch with corrected
size and selected[]
+
+private static VectorizedRowBatch
correctSelectedAndSize(VectorizedRowBatch sourceVrb,
+ ListColumnVector
listColumnVector) {
Review comment:
Should we use MultiValuedColumnVector instead of ListColumnVector?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273228)
Time Spent: 40m (was: 0.5h)
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[
https://issues.apache.org/jira/browse/HIVE-21959?focusedWorklogId=273232&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273232
]
ASF GitHub Bot logged work on HIVE-21959:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 11:35
Start Date: 08/Jul/19 11:35
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #703: HIVE-21959
Clean up Concatenate and Msck DDL commands
URL: https://github.com/apache/hive/pull/703#discussion_r301047070
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
##
@@ -108,15 +108,16 @@ public int repair(MsckInfo msckInfo) {
boolean success = false;
long txnId = -1;
int ret = 0;
+long partitionExpirySeconds = msckInfo.getPartitionExpirySeconds();
Review comment:
this doesn't seem right; see other comment as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273232)
> Clean up Concatenate and Msck DDL commands
> --
>
> Key: HIVE-21959
> URL: https://issues.apache.org/jira/browse/HIVE-21959
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21959.01.patch, HIVE-21959.02.patch,
> HIVE-21959.03.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Concatenate and Msck DDL use basic data structure implementations instead of
> their interfaces for variable declaration, like ArrayList instead of List,
> LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in
> the code, which needs to be cleaned.
> Concatenate also had a non-immutable Desc class, that needs to be transformed
> into an immutable one. Concatenate operation code should be cut into smaller
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[
https://issues.apache.org/jira/browse/HIVE-21959?focusedWorklogId=273233&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273233
]
ASF GitHub Bot logged work on HIVE-21959:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 11:35
Start Date: 08/Jul/19 11:35
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #703: HIVE-21959
Clean up Concatenate and Msck DDL commands
URL: https://github.com/apache/hive/pull/703#discussion_r301046777
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
##
@@ -108,15 +108,16 @@ public int repair(MsckInfo msckInfo) {
boolean success = false;
long txnId = -1;
int ret = 0;
+long partitionExpirySeconds = msckInfo.getPartitionExpirySeconds();
try {
Table table = getMsc().getTable(msckInfo.getCatalogName(),
msckInfo.getDbName(), msckInfo.getTableName());
qualifiedTableName = Warehouse.getCatalogQualifiedTableName(table);
if
(getConf().getBoolean(MetastoreConf.ConfVars.MSCK_REPAIR_ENABLE_PARTITION_RETENTION.getHiveName(),
false)) {
-
msckInfo.setPartitionExpirySeconds(PartitionManagementTask.getRetentionPeriodInSeconds(table));
-LOG.info("{} - Retention period ({}s) for partition is enabled for
MSCK REPAIR..",
- qualifiedTableName, msckInfo.getPartitionExpirySeconds());
+partitionExpirySeconds =
PartitionManagementTask.getRetentionPeriodInSeconds(table);
Review comment:
earlier code had a "sideeffect"; namely: it set
mschInfo.setPartitionExpirySeconds
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273233)
Time Spent: 40m (was: 0.5h)
> Clean up Concatenate and Msck DDL commands
> --
>
> Key: HIVE-21959
> URL: https://issues.apache.org/jira/browse/HIVE-21959
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21959.01.patch, HIVE-21959.02.patch,
> HIVE-21959.03.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Concatenate and Msck DDL use basic data structure implementations instead of
> their interfaces for variable declaration, like ArrayList instead of List,
> LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in
> the code, which needs to be cleaned.
> Concatenate also had a non-immutable Desc class, that needs to be transformed
> into an immutable one. Concatenate operation code should be cut into smaller
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[
https://issues.apache.org/jira/browse/HIVE-21959?focusedWorklogId=273230&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273230
]
ASF GitHub Bot logged work on HIVE-21959:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 11:35
Start Date: 08/Jul/19 11:35
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #703: HIVE-21959
Clean up Concatenate and Msck DDL commands
URL: https://github.com/apache/hive/pull/703#discussion_r301041322
##
File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/misc/MsckDesc.java
##
@@ -36,19 +34,16 @@
private static final long serialVersionUID = 1L;
private final String tableName;
- private final ArrayList> partitionsSpecs;
+ private final List> partitionsSpecs;
private final String resFile;
private final boolean repairPartitions;
private final boolean addPartitions;
private final boolean dropPartitions;
- public MsckDesc(String tableName, List>
partitionSpecs, Path resFile,
+ public MsckDesc(String tableName, List> partitionsSpecs,
Path resFile,
boolean repairPartitions, boolean addPartitions, boolean dropPartitions)
{
this.tableName = tableName;
-this.partitionsSpecs = new ArrayList>(partitionSpecs.size());
Review comment:
earlier; the approach was to make a new copy
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273230)
Time Spent: 20m (was: 10m)
> Clean up Concatenate and Msck DDL commands
> --
>
> Key: HIVE-21959
> URL: https://issues.apache.org/jira/browse/HIVE-21959
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21959.01.patch, HIVE-21959.02.patch,
> HIVE-21959.03.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Concatenate and Msck DDL use basic data structure implementations instead of
> their interfaces for variable declaration, like ArrayList instead of List,
> LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in
> the code, which needs to be cleaned.
> Concatenate also had a non-immutable Desc class, that needs to be transformed
> into an immutable one. Concatenate operation code should be cut into smaller
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[
https://issues.apache.org/jira/browse/HIVE-21959?focusedWorklogId=273231&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273231
]
ASF GitHub Bot logged work on HIVE-21959:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 11:35
Start Date: 08/Jul/19 11:35
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on pull request #703: HIVE-21959
Clean up Concatenate and Msck DDL commands
URL: https://github.com/apache/hive/pull/703#discussion_r301044303
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
##
@@ -2061,11 +2057,9 @@ private void analyzeAlterTablePartMergeFiles(ASTNode
ast,
}
// throw a HiveException for other than rcfile and orcfile.
- if (!((inputFormatClass.equals(RCFileInputFormat.class) ||
- (inputFormatClass.equals(OrcInputFormat.class) {
+ if (!(inputFormatClass.equals(RCFileInputFormat.class) ||
inputFormatClass.equals(OrcInputFormat.class))) {
Review comment:
note: I think that instead of class.equals; we should use marker interfaces
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273231)
Time Spent: 0.5h (was: 20m)
> Clean up Concatenate and Msck DDL commands
> --
>
> Key: HIVE-21959
> URL: https://issues.apache.org/jira/browse/HIVE-21959
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21959.01.patch, HIVE-21959.02.patch,
> HIVE-21959.03.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Concatenate and Msck DDL use basic data structure implementations instead of
> their interfaces for variable declaration, like ArrayList instead of List,
> LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in
> the code, which needs to be cleaned.
> Concatenate also had a non-immutable Desc class, that needs to be transformed
> into an immutable one. Concatenate operation code should be cut into smaller
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[
https://issues.apache.org/jira/browse/HIVE-21959?focusedWorklogId=273238&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273238
]
ASF GitHub Bot logged work on HIVE-21959:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 11:50
Start Date: 08/Jul/19 11:50
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #703:
HIVE-21959 Clean up Concatenate and Msck DDL commands
URL: https://github.com/apache/hive/pull/703#discussion_r301052561
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
##
@@ -108,15 +108,16 @@ public int repair(MsckInfo msckInfo) {
boolean success = false;
long txnId = -1;
int ret = 0;
+long partitionExpirySeconds = msckInfo.getPartitionExpirySeconds();
try {
Table table = getMsc().getTable(msckInfo.getCatalogName(),
msckInfo.getDbName(), msckInfo.getTableName());
qualifiedTableName = Warehouse.getCatalogQualifiedTableName(table);
if
(getConf().getBoolean(MetastoreConf.ConfVars.MSCK_REPAIR_ENABLE_PARTITION_RETENTION.getHiveName(),
false)) {
-
msckInfo.setPartitionExpirySeconds(PartitionManagementTask.getRetentionPeriodInSeconds(table));
-LOG.info("{} - Retention period ({}s) for partition is enabled for
MSCK REPAIR..",
- qualifiedTableName, msckInfo.getPartitionExpirySeconds());
+partitionExpirySeconds =
PartitionManagementTask.getRetentionPeriodInSeconds(table);
Review comment:
Indeed, but it is not use anywhere else, but here, this is the only place
where the getter is invoked. The point here was that a request type object
should not be modified, it should be immutable, the request contained a certain
value, which is used here like this - may be overwritten by some config value.
Still the request is not modified.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273238)
Time Spent: 50m (was: 40m)
> Clean up Concatenate and Msck DDL commands
> --
>
> Key: HIVE-21959
> URL: https://issues.apache.org/jira/browse/HIVE-21959
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.1.1
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Fix For: 4.0.0
>
> Attachments: HIVE-21959.01.patch, HIVE-21959.02.patch,
> HIVE-21959.03.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Concatenate and Msck DDL use basic data structure implementations instead of
> their interfaces for variable declaration, like ArrayList instead of List,
> LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in
> the code, which needs to be cleaned.
> Concatenate also had a non-immutable Desc class, that needs to be transformed
> into an immutable one. Concatenate operation code should be cut into smaller
> functions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21948) Implement parallel processing in Pre Upgrade Tool
[
https://issues.apache.org/jira/browse/HIVE-21948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880248#comment-16880248
]
Hive QA commented on HIVE-21948:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973887/HIVE-21948.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16363 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17905/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17905/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17905/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973887 - PreCommit-HIVE-Build
> Implement parallel processing in Pre Upgrade Tool
> -
>
> Key: HIVE-21948
> URL: https://issues.apache.org/jira/browse/HIVE-21948
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21948.1.patch
>
>
> Pre Upgrade Tool scans for all databases and tables in the warehouse
> sequentially which can be very slow in case of lots of tables.
> Example: It took the process 8-10 hours to complete on ~500k tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shubham Chaurasia updated HIVE-21966:
-
Attachment: HIVE-21966.2.patch
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch, HIVE-21966.2.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?focusedWorklogId=273252&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273252
]
ASF GitHub Bot logged work on HIVE-21966:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 12:10
Start Date: 08/Jul/19 12:10
Worklog Time Spent: 10m
Work Description: ShubhamChaurasia commented on pull request #706:
HIVE-21966: Llap external client - Arrow Serializer throws
ArrayIndexOutOfBoundsException
URL: https://github.com/apache/hive/pull/706#discussion_r301059022
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java
##
@@ -365,27 +369,74 @@ private void writeStruct(NonNullableStructVector
arrowVector, StructColumnVector
}
}
- private void writeList(ListVector arrowVector, ListColumnVector hiveVector,
ListTypeInfo typeInfo, int size,
+// selected[] points to the valid/filtered/selected records at row level.
+// for MultiValuedColumnVector such as ListColumnVector one record of
vector points to multiple nested records.
+// In child vectors we get these records in exploded manner i.e. the
number of records in child vectors can have size more
+// than actual the VectorizedRowBatch, consequently selected[] also needs
to be readjusted.
+// This method creates a shallow copy of VectorizedRowBatch with corrected
size and selected[]
+
+private static VectorizedRowBatch
correctSelectedAndSize(VectorizedRowBatch sourceVrb,
+ ListColumnVector
listColumnVector) {
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273252)
Time Spent: 50m (was: 40m)
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch, HIVE-21966.2.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?focusedWorklogId=273253&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273253
]
ASF GitHub Bot logged work on HIVE-21966:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 12:10
Start Date: 08/Jul/19 12:10
Worklog Time Spent: 10m
Work Description: ShubhamChaurasia commented on pull request #706:
HIVE-21966: Llap external client - Arrow Serializer throws
ArrayIndexOutOfBoundsException
URL: https://github.com/apache/hive/pull/706#discussion_r301059071
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java
##
@@ -365,27 +369,74 @@ private void writeStruct(NonNullableStructVector
arrowVector, StructColumnVector
}
}
- private void writeList(ListVector arrowVector, ListColumnVector hiveVector,
ListTypeInfo typeInfo, int size,
+// selected[] points to the valid/filtered/selected records at row level.
+// for MultiValuedColumnVector such as ListColumnVector one record of
vector points to multiple nested records.
+// In child vectors we get these records in exploded manner i.e. the
number of records in child vectors can have size more
+// than actual the VectorizedRowBatch, consequently selected[] also needs
to be readjusted.
+// This method creates a shallow copy of VectorizedRowBatch with corrected
size and selected[]
+
+private static VectorizedRowBatch
correctSelectedAndSize(VectorizedRowBatch sourceVrb,
+ ListColumnVector
listColumnVector) {
+
+VectorizedRowBatch vrb = new VectorizedRowBatch(sourceVrb.numCols,
sourceVrb.size);
+vrb.cols = sourceVrb.cols;
+vrb.endOfFile = sourceVrb.endOfFile;
+vrb.projectedColumns = sourceVrb.projectedColumns;
+vrb.projectionSize = sourceVrb.projectionSize;
+vrb.selectedInUse = sourceVrb.selectedInUse;
+
+int correctedSize = 0;
+final int[] srcVrbSelected = sourceVrb.selected;
+for (int i = 0; i < sourceVrb.size; i++) {
+correctedSize += listColumnVector.lengths[srcVrbSelected[i]];
Review comment:
done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273253)
Time Spent: 1h (was: 50m)
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch, HIVE-21966.2.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STR
[jira] [Work logged] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?focusedWorklogId=273254&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273254
]
ASF GitHub Bot logged work on HIVE-21966:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 12:10
Start Date: 08/Jul/19 12:10
Worklog Time Spent: 10m
Work Description: ShubhamChaurasia commented on pull request #706:
HIVE-21966: Llap external client - Arrow Serializer throws
ArrayIndexOutOfBoundsException
URL: https://github.com/apache/hive/pull/706#discussion_r301059193
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/arrow/Serializer.java
##
@@ -365,27 +369,74 @@ private void writeStruct(NonNullableStructVector
arrowVector, StructColumnVector
}
}
- private void writeList(ListVector arrowVector, ListColumnVector hiveVector,
ListTypeInfo typeInfo, int size,
+// selected[] points to the valid/filtered/selected records at row level.
+// for MultiValuedColumnVector such as ListColumnVector one record of
vector points to multiple nested records.
+// In child vectors we get these records in exploded manner i.e. the
number of records in child vectors can have size more
+// than actual the VectorizedRowBatch, consequently selected[] also needs
to be readjusted.
+// This method creates a shallow copy of VectorizedRowBatch with corrected
size and selected[]
+
+private static VectorizedRowBatch
correctSelectedAndSize(VectorizedRowBatch sourceVrb,
+ ListColumnVector
listColumnVector) {
+
+VectorizedRowBatch vrb = new VectorizedRowBatch(sourceVrb.numCols,
sourceVrb.size);
+vrb.cols = sourceVrb.cols;
+vrb.endOfFile = sourceVrb.endOfFile;
+vrb.projectedColumns = sourceVrb.projectedColumns;
+vrb.projectionSize = sourceVrb.projectionSize;
+vrb.selectedInUse = sourceVrb.selectedInUse;
Review comment:
done. thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273254)
Time Spent: 1h 10m (was: 1h)
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch, HIVE-21966.2.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by A
[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880333#comment-16880333
]
Hive QA commented on HIVE-21923:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973888/HIVE-21923.05.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16361 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17906/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17906/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17906/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973888 - PreCommit-HIVE-Build
> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-21923.01.patch, HIVE-21923.02.patch,
> HIVE-21923.03.patch, HIVE-21923.04.patch, HIVE-21923.05.patch
>
>
> HIVE-21189 have introduced some resultset changes
> in ql/src/test/results/clientpositive/llap/hybridgrace_hashjoin_2.q.out
> https://github.com/apache/hive/commit/5799398450c17d06e8ef144ce835a8524f5abec9#diff-56b3ab96b6c90fdbebe2c4f84e8595afL500
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21959) Clean up Concatenate and Msck DDL commands
[ https://issues.apache.org/jira/browse/HIVE-21959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21959: -- Attachment: HIVE-21959.04.patch > Clean up Concatenate and Msck DDL commands > -- > > Key: HIVE-21959 > URL: https://issues.apache.org/jira/browse/HIVE-21959 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21959.01.patch, HIVE-21959.02.patch, > HIVE-21959.03.patch, HIVE-21959.04.patch > > Time Spent: 50m > Remaining Estimate: 0h > > Concatenate and Msck DDL use basic data structure implementations instead of > their interfaces for variable declaration, like ArrayList instead of List, > LinkedHashMap instead of Map, etc. This is due to a lot of similar issues in > the code, which needs to be cleaned. > Concatenate also had a non-immutable Desc class, that needs to be transformed > into an immutable one. Concatenate operation code should be cut into smaller > functions. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21225) ACID: getAcidState() should cache a recursive dir listing locally
[
https://issues.apache.org/jira/browse/HIVE-21225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880346#comment-16880346
]
Hive QA commented on HIVE-21225:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973895/HIVE-21225.8.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17907/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17907/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17907/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12973895/HIVE-21225.8.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973895 - PreCommit-HIVE-Build
> ACID: getAcidState() should cache a recursive dir listing locally
> -
>
> Key: HIVE-21225
> URL: https://issues.apache.org/jira/browse/HIVE-21225
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Reporter: Gopal V
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-21225.1.patch, HIVE-21225.2.patch,
> HIVE-21225.3.patch, HIVE-21225.4.patch, HIVE-21225.4.patch,
> HIVE-21225.5.patch, HIVE-21225.6.patch, HIVE-21225.7.patch,
> HIVE-21225.7.patch, HIVE-21225.8.patch, async-pid-44-2.svg
>
>
> Currently getAcidState() makes 3 calls into the FS api which could be
> answered by making a single recursive listDir call and reusing the same data
> to check for isRawFormat() and isValidBase().
> All delta operations for a single partition can go against a single listed
> directory snapshot instead of interacting with the NameNode or ObjectStore
> within the inner loop.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21957) Create temporary table like should omit transactional properties
[
https://issues.apache.org/jira/browse/HIVE-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880349#comment-16880349
]
Hive QA commented on HIVE-21957:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973901/HIVE-21957.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17908/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17908/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17908/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-07-08 13:29:46.950
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-17908/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-07-08 13:29:46.953
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 67e515f HIVE-21571: SHOW COMPACTIONS shows column names as its
first output row (Rajkumar Singh, reviewed by Daniel Dai)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 67e515f HIVE-21571: SHOW COMPACTIONS shows column names as its
first output row (Rajkumar Singh, reviewed by Daniel Dai)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-07-08 13:29:48.109
+ rm -rf ../yetus_PreCommit-HIVE-Build-17908
+ mkdir ../yetus_PreCommit-HIVE-Build-17908
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-17908
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17908/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc8234374177639365020.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc8234374177639365020.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-shims
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-17908
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973901 - PreCommit-HIVE-Build
> Create temporary table like should omit transactional properties
> -
[jira] [Commented] (HIVE-21923) Vectorized MapJoin may miss results when only the join key is selected
[
https://issues.apache.org/jira/browse/HIVE-21923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880358#comment-16880358
]
Hive QA commented on HIVE-21923:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 19m
20s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
29s{color} | {color:red} common: The patch generated 1 new + 1922 unchanged - 0
fixed = 1923 total (was 1922) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 12m
52s{color} | {color:red} root: The patch generated 1 new + 73140 unchanged - 0
fixed = 73141 total (was 73140) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
20s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17906/dev-support/hive-personality.sh
|
| git revision | master / 67e515f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17906/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17906/yetus/diff-checkstyle-root.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17906/yetus/patch-findbugs-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17906/yetus/patch-findbugs-ql.txt
|
| modules | C: common ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17906/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorized MapJoin may miss results when only the join key is selected
> --
>
> Key: HIVE-21923
> URL: https://issues.apache.org/jira/browse/HIVE-21923
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zolt
[jira] [Commented] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[
https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880397#comment-16880397
]
Hive QA commented on HIVE-21938:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} upgrade-acid/pre-upgrade: The patch generated 2 new +
56 unchanged - 12 fixed = 58 total (was 68) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17909/dev-support/hive-personality.sh
|
| git revision | master / 67e515f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17909/yetus/diff-checkstyle-upgrade-acid_pre-upgrade.txt
|
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17909/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add database and table filter options to PreUpgradeTool
> ---
>
> Key: HIVE-21938
> URL: https://issues.apache.org/jira/browse/HIVE-21938
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch,
> HIVE-21938.3.patch, HIVE-21938.4.patch, HIVE-21938.5.patch, HIVE-21938.6.patch
>
>
> By default pre upgrade tool scans all databases and tables in the warehouse.
> Add database and table filter options to run the tool for a specific subset
> of databases and tables only.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21191) I want to extends lag/lead functions to Implementing some special functions, And I met some problems
[
https://issues.apache.org/jira/browse/HIVE-21191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880419#comment-16880419
]
Ittay Dror commented on HIVE-21191:
---
This is happening to me as well. It seems like Hive doesn't support user
defined windowing functions
> I want to extends lag/lead functions to Implementing some special functions,
> And I met some problems
> -
>
> Key: HIVE-21191
> URL: https://issues.apache.org/jira/browse/HIVE-21191
> Project: Hive
> Issue Type: Wish
> Components: Hive, UDF, Windows
>Affects Versions: 1.1.0
>Reporter: one
>Priority: Minor
> Labels: LAG(), UDAF, UDF, window_function
>
> i want a distinctLag functions ,The function is like lag, but the difference
> is to select different values in front of it.
> Example:
> {color:#14892c}select * from active{color}
> ||session||sq||channel||
> |1|1|A|
> |1|2|B|
> |1|3|B|
> |1|4|C|
> |1|5|B|
> |2|1|C|
> |2|2|B|
> |2|3|B|
> |2|4|A|
> |2|5|B|
> {color:#14892c}
> select session,sq,lag(channel)over(partition by session order by sq) from
> active{color}
> ||session||sq||channel||
> |1|1|null|
> |1|2|A|
> |1|3|B|
> |1|4|B|
> |1|5|C|
> |2|1|null|
> |2|2|C|
> |2|3|B|
> |2|4|B|
> |2|5|A|
> The function I want is：{color:#14892c}
> select session,sq,distinctLag(channel)over(partition by session order by sq)
> from active{color}
> ||session||sq||channel||
> |1|1|null|
> |1|2|A|
> |1|3|A|
> |1|4|B|
> |1|5|C|
> |2|1|null|
> |2|2|C|
> |2|3|C|
> |2|4|B|
> |2|5|A|
>
> i try to extend GenericUDFLeadLag and Override:
> {code:java}
> import org.apache.hadoop.hive.ql.exec.Description;
> import org.apache.hadoop.hive.ql.metadata.HiveException;
> import org.apache.hadoop.hive.ql.udf.UDFType;
> import org.apache.hadoop.hive.ql.udf.generic.GenericUDFLeadLag;
> import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils;
> import
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.ObjectInspectorCopyOption;
> @Description(
> name = "distinctLag",
> value = "distinctLag (scalar_expression [,offset] [,default]) OVER
> ([query_partition_clause] order_by_clause); "
> + "The distinctLag function is used to access data from a
> distinct previous row.",
> extended = "Example:\n "
> + "select p1.p_mfgr, p1.p_name, p1.p_size,\n"
> + " p1.p_size - distinctLag(p1.p_size,1,p1.p_size) over( distribute
> by p1.p_mfgr sort by p1.p_name) as deltaSz\n"
> + " from part p1 join part p2 on p1.p_partkey = p2.p_partkey")
> @UDFType(impliesOrder = true)
> public class GenericUDFDistinctLag extends GenericUDFLeadLag {
> @Override
> public Object evaluate(DeferredObject[] arguments) throws HiveException
> {
> Object defaultVal = null;
> if (arguments.length == 3) {
> defaultVal =
> ObjectInspectorUtils.copyToStandardObject(getDefaultValueConverter().convert(arguments[2].get()),
> getDefaultArgOI());
> }
> int idx = getpItr().getIndex() - 1;
> int start = 0;
> int end = getpItr().getPartition().size();
> try {
> Object currValue =
> ObjectInspectorUtils.copyToStandardObject(getExprEvaluator().evaluate(getpItr().resetToIndex(idx)),
> getFirstArgOI(), ObjectInspectorCopyOption.WRITABLE);
> Object ret = null;
> int newIdx = idx;
> do {
> --newIdx;
> if (newIdx >= end || newIdx < start) {
> ret = defaultVal;
> return ret;
> }else{
> ret =
> ObjectInspectorUtils.copyToStandardObject(getExprEvaluator().evaluate(getpItr().lag(1)),
> getFirstArgOI(), ObjectInspectorCopyOption.WRITABLE);
> if(ret.equals(currValue)){
> setAmt(getAmt() - 1);
> }
> }
> } while (getAmt() > 0);
> return ret;
> } finally {
> Object currRow = getpItr().resetToIndex(idx);
> // reevaluate expression on current Row, to trigger the
> Lazy object
> // caches to be reset to the current row.
> getExprEvaluator().evaluate(currRow);
> }
> }
> @Override
> protected String _getFnName(){
>return "distinctLag";
> }
> @Override
> protected Ob
[jira] [Updated] (HIVE-21957) Create temporary table like should omit transactional properties
[ https://issues.apache.org/jira/browse/HIVE-21957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Pinter updated HIVE-21957: - Attachment: HIVE-21957.03.patch > Create temporary table like should omit transactional properties > > > Key: HIVE-21957 > URL: https://issues.apache.org/jira/browse/HIVE-21957 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-21957.01.patch, HIVE-21957.02.patch, > HIVE-21957.03.patch > > > In case of create temporary table like queries, where the source table is > transactional, the transactional properties should not be copied over to the > new table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?focusedWorklogId=273332&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273332
]
ASF GitHub Bot logged work on HIVE-18842:
-
Author: ASF GitHub Bot
Created on: 08/Jul/19 15:14
Start Date: 08/Jul/19 15:14
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #696: HIVE-18842
URL: https://github.com/apache/hive/pull/696#discussion_r301150194
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -7297,8 +7435,19 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
// Add NOT NULL constraint check
input = genConstraintsPlan(dest, qb, input);
- // Add sorting/bucketing if needed
- input = genBucketingSortingDest(dest, input, qb, tableDescriptor,
destinationTable, rsCtx);
+ if (destinationTable.isMaterializedView() &&
+ mvRebuildMode ==
MaterializationRebuildMode.INSERT_OVERWRITE_REBUILD) {
+// Data organization (DISTRIBUTED, SORTED, CLUSTERED) for materialized
view
+// TODO: We only do this for a full rebuild
Review comment:
I was thinking about this and there is no need to log this info. This is a
missing feature, and there is a `TODO` and a follow-up JIRA
(https://issues.apache.org/jira/browse/HIVE-21953).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 273332)
Time Spent: 1.5h (was: 1h 20m)
> CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available, todoc4.0
> Attachments: HIVE-18842.01.patch, HIVE-18842.01.patch,
> HIVE-18842.02.patch, HIVE-18842.03.patch, HIVE-18842.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON+SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | DISTRIBUTED ON (col_name, ...) SORTED ON
> (col_name, ...)] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18842) CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
[
https://issues.apache.org/jira/browse/HIVE-18842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-18842:
---
Attachment: HIVE-18842.03.patch
> CLUSTERED ON/DISTRIBUTED ON+SORTED ON support for materialized views
>
>
> Key: HIVE-18842
> URL: https://issues.apache.org/jira/browse/HIVE-18842
> Project: Hive
> Issue Type: New Feature
> Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Labels: pull-request-available, todoc4.0
> Attachments: HIVE-18842.01.patch, HIVE-18842.01.patch,
> HIVE-18842.02.patch, HIVE-18842.03.patch, HIVE-18842.03.patch,
> HIVE-18842.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We should support defining a {{CLUSTERED ON/DISTRIBUTED ON+SORTED ON}}
> specification for materialized views.
> The syntax should be extended as follows:
> {code:sql}
> CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name
> [COMMENT materialized_view_comment]
> [PARTITIONED ON (col_name, ...)]
> [CLUSTERED ON (col_name, ...) | DISTRIBUTED ON (col_name, ...) SORTED ON
> (col_name, ...)] -- NEW!
> [
>[ROW FORMAT row_format]
>[STORED AS file_format]
> | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)]
> ]
> [LOCATION hdfs_path]
> [TBLPROPERTIES (property_name=property_value, ...)]
> AS select_statement;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[
https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880460#comment-16880460
]
Hive QA commented on HIVE-21938:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973903/HIVE-21938.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 16363 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=350)
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testConcurrentDropPartitions
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=232)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17909/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17909/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17909/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973903 - PreCommit-HIVE-Build
> Add database and table filter options to PreUpgradeTool
> ---
>
> Key: HIVE-21938
> URL: https://issues.apache.org/jira/browse/HIVE-21938
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Blocker
> Fix For: 4.0.0
>
> Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch,
> HIVE-21938.3.patch, HIVE-21938.4.patch, HIVE-21938.5.patch, HIVE-21938.6.patch
>
>
> By default pre upgrade tool scans all databases and tables in the warehouse.
> Add database and table filter options to run the tool for a specific subset
> of databases and tables only.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21928) Fix for statistics annotation in nested AND expressions
[
https://issues.apache.org/jira/browse/HIVE-21928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21928:
---
Attachment: HIVE-21928.01.patch
> Fix for statistics annotation in nested AND expressions
> ---
>
> Key: HIVE-21928
> URL: https://issues.apache.org/jira/browse/HIVE-21928
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-21928.01.patch, HIVE-21928.01.patch,
> HIVE-21928.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Discovered while working on HIVE-21867. Having predicates with nested AND
> expressions may result in different stats, even if predicates are basically
> similar (from stats estimation standpoint).
> For instance, stats for {{AND(x=5, true, true)}} are different from {{x=5}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21928) Fix for statistics annotation in nested AND expressions
[
https://issues.apache.org/jira/browse/HIVE-21928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-21928:
---
Attachment: HIVE-21928.02.patch
> Fix for statistics annotation in nested AND expressions
> ---
>
> Key: HIVE-21928
> URL: https://issues.apache.org/jira/browse/HIVE-21928
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
> Labels: pull-request-available
> Attachments: HIVE-21928.01.patch, HIVE-21928.01.patch,
> HIVE-21928.02.patch, HIVE-21928.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Discovered while working on HIVE-21867. Having predicates with nested AND
> expressions may result in different stats, even if predicates are basically
> similar (from stats estimation standpoint).
> For instance, stats for {{AND(x=5, true, true)}} are different from {{x=5}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880503#comment-16880503
]
Hive QA commented on HIVE-21958:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
28s{color} | {color:blue} standalone-metastore/metastore-common in master has
31 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 106
unchanged - 0 fixed = 108 total (was 106) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
17s{color} | {color:red} ql generated 9 new + 2243 unchanged - 9 fixed = 2252
total (was 2252) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA235 be a _static_
inner class? At HiveParser.java:inner class? At HiveParser.java:[lines
48087-48100] |
| | Dead store to LA29_128 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47827] |
| | Dead store to LA29_130 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47840] |
| | Dead store to LA29_132 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47853] |
| | Dead store to LA29_134 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47866] |
| | Dead store to LA29_136 in
org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At
HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA29.specialStateTransition(int,
IntStream) At HiveParser.java:[line 47879] |
| | Dead store to LA29_138 in
org.apache.hadoop.hive.ql.pars
[jira] [Commented] (HIVE-21958) The list of table expression in the inclusion and exclusion list should be separated by '|' instead of comma.
[
https://issues.apache.org/jira/browse/HIVE-21958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880529#comment-16880529
]
Hive QA commented on HIVE-21958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973905/HIVE-21958.04.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 16361 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=232)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=232)
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=222)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17910/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17910/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17910/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973905 - PreCommit-HIVE-Build
> The list of table expression in the inclusion and exclusion list should be
> separated by '|' instead of comma.
> -
>
> Key: HIVE-21958
> URL: https://issues.apache.org/jira/browse/HIVE-21958
> Project: Hive
> Issue Type: Sub-task
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21958.01.patch, HIVE-21958.02.patch,
> HIVE-21958.03.patch, HIVE-21958.04.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Java regex expression does not support comma. If user wants multiple
> expression to be present in the include or exclude list, then the expressions
> can be provided separated by pipe ('|') character. The policy will look
> something like db_name.'(t1*)|(t3)'.'t100'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21967) Clean up CreateTableLikeOperation
[ https://issues.apache.org/jira/browse/HIVE-21967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21967: - > Clean up CreateTableLikeOperation > - > > Key: HIVE-21967 > URL: https://issues.apache.org/jira/browse/HIVE-21967 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > > CreateTableLikeOperation has two sub types, creating from view or table. A > lot of their codes is common, they should be reused. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21967) Clean up CreateTableLikeOperation
[ https://issues.apache.org/jira/browse/HIVE-21967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21967: -- Status: Patch Available (was: Open) > Clean up CreateTableLikeOperation > - > > Key: HIVE-21967 > URL: https://issues.apache.org/jira/browse/HIVE-21967 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21967.01.patch > > > CreateTableLikeOperation has two sub types, creating from view or table. A > lot of their codes is common, they should be reused. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21967) Clean up CreateTableLikeOperation
[ https://issues.apache.org/jira/browse/HIVE-21967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21967: -- Attachment: HIVE-21967.01.patch > Clean up CreateTableLikeOperation > - > > Key: HIVE-21967 > URL: https://issues.apache.org/jira/browse/HIVE-21967 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21967.01.patch > > > CreateTableLikeOperation has two sub types, creating from view or table. A > lot of their codes is common, they should be reused. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Status: Patch Available (was: Open) > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch, HIVE-21938.4.patch, HIVE-21938.5.patch, > HIVE-21938.6.patch, HIVE-21938.7.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Status: Open (was: Patch Available) > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch, HIVE-21938.4.patch, HIVE-21938.5.patch, > HIVE-21938.6.patch, HIVE-21938.7.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21938) Add database and table filter options to PreUpgradeTool
[ https://issues.apache.org/jira/browse/HIVE-21938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-21938: -- Attachment: HIVE-21938.7.patch > Add database and table filter options to PreUpgradeTool > --- > > Key: HIVE-21938 > URL: https://issues.apache.org/jira/browse/HIVE-21938 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Blocker > Fix For: 4.0.0 > > Attachments: HIVE-21938.1.patch, HIVE-21938.2.patch, > HIVE-21938.3.patch, HIVE-21938.4.patch, HIVE-21938.5.patch, > HIVE-21938.6.patch, HIVE-21938.7.patch > > > By default pre upgrade tool scans all databases and tables in the warehouse. > Add database and table filter options to run the tool for a specific subset > of databases and tables only. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21968) Remove index related tests
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-21968: - > Remove index related tests > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21968: -- Summary: Remove index related codes (was: Remove index related tests) > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21968: -- Description: Hive doesn't support indexes since 3.0.0, still some index related tests and some code were left behind, and some code to disable them. They should be removed. (was: Hive doesn't support indexes since 3.0.0, still some index related tests were left behind, and some code to disable them. They should be removed.) > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Hive doesn't support indexes since 3.0.0, still some index related tests and > some code were left behind, and some code to disable them. They should be > removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21968: -- Description: Hive doesn't support indexes since 3.0.0, still some index related tests were left behind, and some code to disable them. Also some index related code is still in the codebase. They should be removed. (was: Hive doesn't support indexes since 3.0.0, still some index related tests and some code were left behind, and some code to disable them. They should be removed.) > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21968: -- Attachment: HIVE-21968.01.patch > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21968.01.patch > > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21968: -- Status: Patch Available (was: Open) > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21968.01.patch > > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21968: -- Attachment: HIVE-21968.01.patch > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21968.01.patch > > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21224) Upgrade tests JUnit3 to JUnit4
[
https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880591#comment-16880591
]
Hive QA commented on HIVE-21224:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} serde in master has 193 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} cli in master has 8 extant Findbugs warnings. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} contrib in master has 10 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} druid-handler in master has 3 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} hcatalog/core in master has 29 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue} hcatalog/webhcat/java-client in master has 3 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} common: The patch generated 1 new + 24 unchanged - 3
fixed = 25 total (was 27) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} serde: The patch generated 271 new + 267 unchanged -
43 fixed = 538 total (was 310) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
56s{color} | {color:red} ql: The patch generated 8 new + 1546 unchanged - 132
fixed = 1554 total (was 1678) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} service: The patch generated 1 new + 17 unchanged - 6
fixed = 18 total (was 23) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch accumulo-handler passed checkstyle {color}
|
| {color:green}+1{color
[jira] [Updated] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-21968: -- Attachment: (was: HIVE-21968.01.patch) > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21968.01.patch > > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21968: -- Labels: pull-request-available (was: ) > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21968.01.patch > > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?focusedWorklogId=273446&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273446 ] ASF GitHub Bot logged work on HIVE-21968: - Author: ASF GitHub Bot Created on: 08/Jul/19 18:02 Start Date: 08/Jul/19 18:02 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #707: HIVE-21968 Remove index related codes URL: https://github.com/apache/hive/pull/707 Hive doesn't support indexes since 3.0.0, still some index related tests were left behind, and some code to disable them. Also some index related code is still in the codebase. They should be removed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 273446) Time Spent: 10m Remaining Estimate: 0h > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21968.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880595#comment-16880595 ] Miklos Gergely commented on HIVE-21968: --- Created [https://github.com/apache/hive/pull/707], [~jcamachorodriguez] please review. Also fixed tons of formatting issues in Operation2Privilege.java > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21968.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21224) Upgrade tests JUnit3 to JUnit4
[
https://issues.apache.org/jira/browse/HIVE-21224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880597#comment-16880597
]
Hive QA commented on HIVE-21224:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973908/HIVE-21224.13.patch
{color:green}SUCCESS:{color} +1 due to 165 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 16345 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.io.TestRCFile.initializationError (batchId=310)
org.apache.hadoop.hive.ql.udf.generic.TestGenericUDFMonthsBetween.initializationError
(batchId=305)
org.apache.hadoop.hive.serde2.binarysortable.TestBinarySortableFast.initializationError
(batchId=345)
org.apache.hadoop.hive.serde2.lazy.TestLazySimpleFast.initializationError
(batchId=345)
org.apache.hive.common.util.TestFixedSizedObjectPool.initializationError
(batchId=297)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17911/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17911/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17911/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973908 - PreCommit-HIVE-Build
> Upgrade tests JUnit3 to JUnit4
> --
>
> Key: HIVE-21224
> URL: https://issues.apache.org/jira/browse/HIVE-21224
> Project: Hive
> Issue Type: Improvement
>Reporter: Bruno Pusztahazi
>Assignee: Bruno Pusztahazi
>Priority: Major
> Attachments: HIVE-21224.1.patch, HIVE-21224.10.patch,
> HIVE-21224.11.patch, HIVE-21224.12.patch, HIVE-21224.13.patch,
> HIVE-21224.2.patch, HIVE-21224.3.patch, HIVE-21224.4.patch,
> HIVE-21224.5.patch, HIVE-21224.6.patch, HIVE-21224.7.patch,
> HIVE-21224.8.patch, HIVE-21224.9.patch
>
>
> Old JUnit3 tests should be upgraded to JUnit4
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21967) Clean up CreateTableLikeOperation
[ https://issues.apache.org/jira/browse/HIVE-21967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21967: -- Labels: pull-request-available refactor-ddl (was: refactor-ddl) > Clean up CreateTableLikeOperation > - > > Key: HIVE-21967 > URL: https://issues.apache.org/jira/browse/HIVE-21967 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21967.01.patch > > > CreateTableLikeOperation has two sub types, creating from view or table. A > lot of their codes is common, they should be reused. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21967) Clean up CreateTableLikeOperation
[ https://issues.apache.org/jira/browse/HIVE-21967?focusedWorklogId=273467&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-273467 ] ASF GitHub Bot logged work on HIVE-21967: - Author: ASF GitHub Bot Created on: 08/Jul/19 18:39 Start Date: 08/Jul/19 18:39 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #708: HIVE-21967 Clean up CreateTableLikeOperation URL: https://github.com/apache/hive/pull/708 CreateTableLikeOperation has two sub types, creating from view or table. A lot of their codes is common, they should be reused. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 273467) Time Spent: 10m Remaining Estimate: 0h > Clean up CreateTableLikeOperation > - > > Key: HIVE-21967 > URL: https://issues.apache.org/jira/browse/HIVE-21967 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21967.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > CreateTableLikeOperation has two sub types, creating from view or table. A > lot of their codes is common, they should be reused. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880623#comment-16880623
]
Hive QA commented on HIVE-21966:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
0s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 1 new + 211 unchanged - 0
fixed = 212 total (was 211) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17912/dev-support/hive-personality.sh
|
| git revision | master / 67e515f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17912/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17912/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch, HIVE-21966.2.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=tru
[jira] [Commented] (HIVE-21966) Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException in some cases
[
https://issues.apache.org/jira/browse/HIVE-21966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880654#comment-16880654
]
Hive QA commented on HIVE-21966:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12973923/HIVE-21966.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 16363 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/17912/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17912/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17912/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12973923 - PreCommit-HIVE-Build
> Llap external client - Arrow Serializer throws ArrayIndexOutOfBoundsException
> in some cases
> ---
>
> Key: HIVE-21966
> URL: https://issues.apache.org/jira/browse/HIVE-21966
> Project: Hive
> Issue Type: Bug
> Components: llap, Serializers/Deserializers
>Affects Versions: 3.1.1
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21966.1.patch, HIVE-21966.2.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When we submit query through llap-ext-client, arrow serializer throws
> ArrayIndexOutOfBoundsException when 1), 2) and 3) below are satisfied.
> 1) {{hive.vectorized.execution.filesink.arrow.native.enabled=true}} to take
> arrow serializer code path.
> 2) Query contains a filter or limit clause which enforces
> {{VectorizedRowBatch#selectedInUse=true}}
> 3) Projection involves a column of type {{MultiValuedColumnVector}}.
> Sample stacktrace:
> {code}
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 150
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeGeneric(Serializer.java:679)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writePrimitive(Serializer.java:518)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:276)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(Serializer.java:342)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:282)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.writeList(Serializer.java:365)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.write(Serializer.java:279)
> at
> org.apache.hadoop.hive.ql.io.arrow.Serializer.serializeBatch(Serializer.java:199)
> at
> org.apache.hadoop.hive.ql.exec.vector.filesink.VectorFileSinkArrowOperator.process(VectorFileSinkArrowOperator.java:135)
> ... 30 more
> {code}
> It can be reproduced by:
> from beeline:
> {code}
> CREATE TABLE complex_tbl(c1 array>) STORED AS ORC;
> INSERT INTO complex_tbl SELECT ARRAY(NAMED_STRUCT('f1','v11', 'f2','v21'),
> NAMED_STRUCT('f1','v21', 'f2','v22'));
> {code}
> and when we fire query: {{select * from complex_tbl limit 1}} through
> llap-ext-client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21915) Hive with TEZ UNION ALL and UDTF results in data loss
[ https://issues.apache.org/jira/browse/HIVE-21915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21915: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) +1. Pushed to master. Thanks for providing the patch [~zhangweilst] > Hive with TEZ UNION ALL and UDTF results in data loss > - > > Key: HIVE-21915 > URL: https://issues.apache.org/jira/browse/HIVE-21915 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 1.2.1 >Reporter: Wei Zhang >Assignee: Wei Zhang >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21915.01.patch, HIVE-21915.02.patch, > HIVE-21915.03.patch, HIVE-21915.04.patch > > > The HQL syntax is like this: > CREATE TEMPORARY TABLE tez_union_all_loss_data AS > SELECT xxx, yyy, zzz,1 as tag > FROM ods_1 > UNION ALL > SELECT xxx, yyy, zzz, tag > FROM > ( > SELECT xxx > ,get_json_object(get_json_object(tb,'$.a'),'$.b') AS yyy > ,zzz > ,2 as tag > FROM ods_2 > LATERAL VIEW EXPLODE(some_udf(uuu)) team_number AS tb > ) tbl > ; > > With above HQL, we are expecting that rows with both tag = 2 and tag = 1 > appear. In our case however, all the rows with tag = 1 are lost. > Dig deeper we can find that the generated two maps have identical task tmp > paths. And that results from when UDTF is present, the FileSinkOperator will > be processed twice generating the tmp path in > GenTezUtils.removeUnionOperators(); > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21968) Remove index related codes
[ https://issues.apache.org/jira/browse/HIVE-21968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880672#comment-16880672 ] Jesus Camacho Rodriguez commented on HIVE-21968: +1 (pending tests) > Remove index related codes > -- > > Key: HIVE-21968 > URL: https://issues.apache.org/jira/browse/HIVE-21968 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21968.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Hive doesn't support indexes since 3.0.0, still some index related tests were > left behind, and some code to disable them. Also some index related code is > still in the codebase. They should be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21965) Implement parallel processing in HiveStrictManagedMigration
[
https://issues.apache.org/jira/browse/HIVE-21965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16880677#comment-16880677
]
Hive QA commented on HIVE-21965:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
1s{color} | {color:blue} ql in master has 2252 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 11 new + 40 unchanged - 12
fixed = 51 total (was 52) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-17913/dev-support/hive-personality.sh
|
| git revision | master / 67e515f |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17913/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-17913/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement parallel processing in HiveStrictManagedMigration
> ---
>
> Key: HIVE-21965
> URL: https://issues.apache.org/jira/browse/HIVE-21965
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Affects Versions: 3.1.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21965.1.patch
>
>
> This process, kicked off from Ambari can take many days for systems with
> 1000's of tables. The process needs to support parallel execution as it
> iterates through the Databases and Tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
