[jira] [Commented] (HIVE-23011) Shared work optimizer should check residual predicates when comparing joins
[
https://issues.apache.org/jira/browse/HIVE-23011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060675#comment-17060675
]
Hive QA commented on HIVE-23011:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
47s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21139/dev-support/hive-personality.sh
|
| git revision | master / 2be3080 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21139/yetus/whitespace-eol.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21139/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Shared work optimizer should check residual predicates when comparing joins
> ---
>
> Key: HIVE-23011
> URL: https://issues.apache.org/jira/browse/HIVE-23011
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch,
> HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch,
> HIVE-23011.patch, HIVE-23011.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.7.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch, HIVE-22997.7.patch > > Time Spent: 2h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404474&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404474
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 06:16
Start Date: 17/Mar/20 06:16
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #951: HIVE-22997 : Copy
external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393463746
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOnHDFSEncryptedZones.java
##
@@ -102,12 +104,20 @@ public void
targetAndSourceHaveDifferentEncryptionZoneKeys() throws Throwable {
put(HiveConf.ConfVars.REPLDIR.varname, primary.repldDir);
}}, "test_key123");
+List dumpWithClause = Arrays.asList(
+"'hive.repl.add.raw.reserved.namespace'='true'",
+"'" + HiveConf.ConfVars.REPL_EXTERNAL_TABLE_BASE_DIR.varname +
"'='"
++ replica.externalTableWarehouseRoot + "'",
+"'distcp.options.skipcrccheck'=''",
+"'" + HiveConf.ConfVars.HIVE_SERVER2_ENABLE_DOAS.varname +
"'='false'",
+"'" + HiveConf.ConfVars.HIVE_DISTCP_DOAS_USER.varname + "'='"
++ UserGroupInformation.getCurrentUser().getUserName()
+"'");
Review comment:
Please add a test with scheduler as well
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404474)
Time Spent: 2h 40m (was: 2.5h)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch,
> HIVE-22997.5.patch, HIVE-22997.6.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404470&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404470
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 06:13
Start Date: 17/Mar/20 06:13
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393463008
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -122,25 +127,31 @@ public String getName() {
@Override
public int execute() {
try {
- Hive hiveDb = getHive();
- Path dumpRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLDIR),
-
Base64.getEncoder().encodeToString(work.dbNameOrPattern.toLowerCase()
- .getBytes(StandardCharsets.UTF_8.name(;
- Path currentDumpPath = new Path(dumpRoot, getNextDumpDir());
- Path hiveDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_HIVE_BASE_DIR);
- DumpMetaData dmd = new DumpMetaData(hiveDumpRoot, conf);
- // Initialize ReplChangeManager instance since we will require it to
encode file URI.
- ReplChangeManager.getInstance(conf);
- Path cmRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLCMDIR));
- Long lastReplId;
- if (!dumpRoot.getFileSystem(conf).exists(dumpRoot)
- || dumpRoot.getFileSystem(conf).listStatus(dumpRoot).length ==
0) {
-lastReplId = bootStrapDump(hiveDumpRoot, dmd, cmRoot, hiveDb);
+ //First Check if external table copy work has been initialized, if so,
just do that and return.
+ if (work.dirCopyIteratorInitialized() ||
work.replPathIteratorInitialized()) {
+intitiateDataCopyTasks();
Review comment:
We would like to have a control on how many tasks in parallel can run
including external table copy. That was the existing behavior even in case of
external table copy.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404470)
Time Spent: 2.5h (was: 2h 20m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch,
> HIVE-22997.5.patch, HIVE-22997.6.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404468&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404468 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 06:11 Start Date: 17/Mar/20 06:11 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393462313 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java ## @@ -386,8 +399,10 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive } } dumpTableListToDumpLocation(tableList, dumpRoot, dbName, conf); + List extTableCopyWorks = dirLocationsToCopy(extTableLocations); Review comment: Sure, will fix that. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404468) Time Spent: 2h 20m (was: 2h 10m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch > > Time Spent: 2h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PRAVIN KUMAR SINHA updated HIVE-22997: -- Attachment: HIVE-22997.6.patch > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, > HIVE-22997.5.patch, HIVE-22997.6.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404447&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404447
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 04:39
Start Date: 17/Mar/20 04:39
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #951: HIVE-22997 :
Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r392815132
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -122,25 +127,31 @@ public String getName() {
@Override
public int execute() {
try {
- Hive hiveDb = getHive();
- Path dumpRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLDIR),
-
Base64.getEncoder().encodeToString(work.dbNameOrPattern.toLowerCase()
- .getBytes(StandardCharsets.UTF_8.name(;
- Path currentDumpPath = new Path(dumpRoot, getNextDumpDir());
- Path hiveDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_HIVE_BASE_DIR);
- DumpMetaData dmd = new DumpMetaData(hiveDumpRoot, conf);
- // Initialize ReplChangeManager instance since we will require it to
encode file URI.
- ReplChangeManager.getInstance(conf);
- Path cmRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLCMDIR));
- Long lastReplId;
- if (!dumpRoot.getFileSystem(conf).exists(dumpRoot)
- || dumpRoot.getFileSystem(conf).listStatus(dumpRoot).length ==
0) {
-lastReplId = bootStrapDump(hiveDumpRoot, dmd, cmRoot, hiveDb);
+ //First Check if external table copy work has been initialized, if so,
just do that and return.
+ if (work.dirCopyIteratorInitialized() ||
work.replPathIteratorInitialized()) {
+intitiateDataCopyTasks();
Review comment:
why cant external table copy task goes in parallel to other tasks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404447)
Time Spent: 2h 10m (was: 2h)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404446&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404446 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 04:39 Start Date: 17/Mar/20 04:39 Worklog Time Spent: 10m Work Description: maheshk114 commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r392815969 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java ## @@ -386,8 +399,10 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive } } dumpTableListToDumpLocation(tableList, dumpRoot, dbName, conf); + List extTableCopyWorks = dirLocationsToCopy(extTableLocations); Review comment: if extTableLocations is empty ..no need to create the task This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404446) Time Spent: 2h 10m (was: 2h) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[
https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404445&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404445
]
ASF GitHub Bot logged work on HIVE-22990:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 04:31
Start Date: 17/Mar/20 04:31
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #949: HIVE-22990
Add file based ack for replication
URL: https://github.com/apache/hive/pull/949#discussion_r393437333
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadCompleteAckTask.java
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.parse.repl.dump.Utils;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+
+import java.io.Serializable;
+
+/**
+ * ReplLoadCompleteAckTask.
+ *
+ * Add the load complete acknoledgement.
+ **/
+public class ReplLoadCompleteAckTask extends Task
implements Serializable {
Review comment:
No ..when repl load / dump task ..returns the status ..it makes sure that
..all child tasks are done ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404445)
Time Spent: 4h (was: 3h 50m)
> Build acknowledgement mechanism for repl dump and load
> --
>
> Key: HIVE-22990
> URL: https://issues.apache.org/jira/browse/HIVE-22990
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch,
> HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch,
> HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch,
> HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch,
> HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch,
> HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch,
> HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[
https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404443&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404443
]
ASF GitHub Bot logged work on HIVE-22990:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 04:22
Start Date: 17/Mar/20 04:22
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #949: HIVE-22990
Add file based ack for replication
URL: https://github.com/apache/hive/pull/949#discussion_r393437622
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -146,7 +146,7 @@ public int execute() {
}
prepareReturnValues(Arrays.asList(currentDumpPath.toUri().toString(),
String.valueOf(lastReplId)));
writeDumpCompleteAck(hiveDumpRoot);
-deletePreviousDumpMeta(previousDumpMetaPath);
+deleteAllPreviousDumpMeta(dumpRoot, currentDumpPath);
Review comment:
But load ..reads all the path and loads only the latest one ..so there is a
race condition ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404443)
Time Spent: 3h 50m (was: 3h 40m)
> Build acknowledgement mechanism for repl dump and load
> --
>
> Key: HIVE-22990
> URL: https://issues.apache.org/jira/browse/HIVE-22990
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch,
> HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch,
> HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch,
> HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch,
> HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch,
> HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch,
> HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[
https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=404442&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404442
]
ASF GitHub Bot logged work on HIVE-22990:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 04:20
Start Date: 17/Mar/20 04:20
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #949: HIVE-22990
Add file based ack for replication
URL: https://github.com/apache/hive/pull/949#discussion_r393437333
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadCompleteAckTask.java
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.parse.repl.dump.Utils;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+
+import java.io.Serializable;
+
+/**
+ * ReplLoadCompleteAckTask.
+ *
+ * Add the load complete acknoledgement.
+ **/
+public class ReplLoadCompleteAckTask extends Task
implements Serializable {
Review comment:
No ..when repl load / dump task ..returns the status ..it makes sure that
..all child tasks are done ..
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404442)
Time Spent: 3h 40m (was: 3.5h)
> Build acknowledgement mechanism for repl dump and load
> --
>
> Key: HIVE-22990
> URL: https://issues.apache.org/jira/browse/HIVE-22990
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch,
> HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch,
> HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch,
> HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch,
> HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch,
> HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch,
> HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.patch
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404441&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404441
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 04:19
Start Date: 17/Mar/20 04:19
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393436614
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -367,13 +379,14 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData
dmd, Path cmRoot, Hive
// Dump external table locations if required.
if (TableType.EXTERNAL_TABLE.equals(table.getTableType())
&& shouldDumpExternalTableLocation()) {
- writer.dataLocationDump(table);
+ extTableLocations.addAll(writer.dataLocationDump(table));
}
// Dump the table to be bootstrapped if required.
if (shouldBootstrapDumpTable(table)) {
HiveWrapper.Tuple tableTuple = new HiveWrapper(hiveDb,
dbName).table(table);
- dumpTable(dbName, tableName, validTxnList, dbRoot, dumpRoot,
bootDumpBeginReplId, hiveDb, tableTuple);
+ replPathMappings.addAll(dumpTable(dbName, tableName,
validTxnList, dbRoot, dumpRoot, bootDumpBeginReplId,
Review comment:
That's the existing issue with the way current iterators are being used and
the patch doesn't address that. May be later, we may want to solve that in
general. May be by using some disk based iterators. Will file a tracking JIRA
for that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404441)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404439&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404439 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 04:19 Start Date: 17/Mar/20 04:19 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393436940 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java ## @@ -918,7 +922,9 @@ public void testRenameTableScenariosWithReplaceExternalTable() throws Throwable String newPolicy = primaryDbName + ".'(in[0-9]+)|(out1500)|(in2)'"; dumpWithClause = Arrays.asList( "'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + "'='true'", -"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'" +"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'", Review comment: Will see if we can do that. Problem was that the parameters were not fixed and were being set on case-to-case basis. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404439) Time Spent: 2h (was: 1h 50m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch > > Time Spent: 2h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404438&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404438
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 04:19
Start Date: 17/Mar/20 04:19
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393436153
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -122,25 +127,31 @@ public String getName() {
@Override
public int execute() {
try {
- Hive hiveDb = getHive();
- Path dumpRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLDIR),
-
Base64.getEncoder().encodeToString(work.dbNameOrPattern.toLowerCase()
- .getBytes(StandardCharsets.UTF_8.name(;
- Path currentDumpPath = new Path(dumpRoot, getNextDumpDir());
- Path hiveDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_HIVE_BASE_DIR);
- DumpMetaData dmd = new DumpMetaData(hiveDumpRoot, conf);
- // Initialize ReplChangeManager instance since we will require it to
encode file URI.
- ReplChangeManager.getInstance(conf);
- Path cmRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLCMDIR));
- Long lastReplId;
- if (!dumpRoot.getFileSystem(conf).exists(dumpRoot)
- || dumpRoot.getFileSystem(conf).listStatus(dumpRoot).length ==
0) {
-lastReplId = bootStrapDump(hiveDumpRoot, dmd, cmRoot, hiveDb);
+ //First Check if external table copy work has been initialized, if so,
just do that and return.
Review comment:
Will do
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404438)
Time Spent: 1h 50m (was: 1h 40m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404437&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404437
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 04:19
Start Date: 17/Mar/20 04:19
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393432287
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -397,17 +397,19 @@ public void externalTableWithPartitions() throws
Throwable {
primary.run("use " + primaryDbName)
.run("insert into table t2 partition(country='france') values
('lyon')")
.run("alter table t2 set location '" + tmpLocation2 + "'")
-.dump(primaryDbName);
+.dump(primaryDbName, withClause);
-replica.load(replicatedDbName, primaryDbName, loadWithClause);
+replica.load(replicatedDbName, primaryDbName, withClause);
assertTablePartitionLocation(primaryDbName + ".t2", replicatedDbName +
".t2");
}
@Test
public void externalTableIncrementalReplication() throws Throwable {
-WarehouseInstance.Tuple tuple = primary.dumpWithCommand("repl dump " +
primaryDbName);
+List withClause = externalTableBasePathWithClause();
+String replDumpCommand = "repl dump " + primaryDbName
++ " WITH (" + withClause.get(0) + "," + withClause.get(1) + ")";
+WarehouseInstance.Tuple tuple = primary.dumpWithCommand(replDumpCommand);
Review comment:
Will change
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404437)
Time Spent: 1h 40m (was: 1.5h)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404440&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404440
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 04:19
Start Date: 17/Mar/20 04:19
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393433210
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -436,16 +438,29 @@ public void externalTableIncrementalReplication() throws
Throwable {
}
List loadWithClause = externalTableBasePathWithClause();
-replica.load(replicatedDbName, primaryDbName, loadWithClause)
+replica.load(replicatedDbName, primaryDbName, withClause)
.run("use " + replicatedDbName)
.run("show tables like 't1'")
.verifyResult("t1")
.run("show partitions t1")
.verifyResults(new String[] { "country=india", "country=us" })
.run("select place from t1 order by place")
-.verifyResults(new String[] { "bangalore", "mumbai", "pune" })
+.verifyResults(new String[] {})
Review comment:
Since the modification happened after dump, data will be seen only after
next dump-load cycle, which is being tested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404440)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404428&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404428
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 03:38
Start Date: 17/Mar/20 03:38
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on pull request #951: HIVE-22997
: Copy external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393427998
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOnHDFSEncryptedZones.java
##
@@ -102,12 +104,20 @@ public void
targetAndSourceHaveDifferentEncryptionZoneKeys() throws Throwable {
put(HiveConf.ConfVars.REPLDIR.varname, primary.repldDir);
}}, "test_key123");
+List dumpWithClause = Arrays.asList(
Review comment:
Since the external table copy is happening on source, the configs are
required to be passed at dump.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404428)
Time Spent: 1.5h (was: 1h 20m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[
https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060583#comment-17060583
]
Hive QA commented on HIVE-22539:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996835/HIVE-22539.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18105 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21138/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21138/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21138/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996835 - PreCommit-HIVE-Build
> HiveServer2 SPNEGO authentication should skip if authorization header is empty
> --
>
> Key: HIVE-22539
> URL: https://issues.apache.org/jira/browse/HIVE-22539
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch,
> HIVE-22539.3.patch, HIVE-22539.4.patch, HIVE-22539.5.patch,
> HIVE-22539.6.patch, HIVE-22539.7.patch
>
>
> Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos
> before checking header. This can be checked up front to avoid doing any
> Kerberos related work if the header is empty. This is helpful in a lot of
> cases since typically the first request is empty with the client waiting for
> a 401 before returning the Authorization header.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404399&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404399
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 01:55
Start Date: 17/Mar/20 01:55
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #951: HIVE-22997 : Copy
external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393402843
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -367,13 +379,14 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData
dmd, Path cmRoot, Hive
// Dump external table locations if required.
if (TableType.EXTERNAL_TABLE.equals(table.getTableType())
&& shouldDumpExternalTableLocation()) {
- writer.dataLocationDump(table);
+ extTableLocations.addAll(writer.dataLocationDump(table));
}
// Dump the table to be bootstrapped if required.
if (shouldBootstrapDumpTable(table)) {
HiveWrapper.Tuple tableTuple = new HiveWrapper(hiveDb,
dbName).table(table);
- dumpTable(dbName, tableName, validTxnList, dbRoot, dumpRoot,
bootDumpBeginReplId, hiveDb, tableTuple);
+ replPathMappings.addAll(dumpTable(dbName, tableName,
validTxnList, dbRoot, dumpRoot, bootDumpBeginReplId,
Review comment:
If there are too many external tables OOM?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404399)
Time Spent: 1h 20m (was: 1h 10m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404398&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404398
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 01:52
Start Date: 17/Mar/20 01:52
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #951: HIVE-22997 : Copy
external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393402297
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -122,25 +127,31 @@ public String getName() {
@Override
public int execute() {
try {
- Hive hiveDb = getHive();
- Path dumpRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLDIR),
-
Base64.getEncoder().encodeToString(work.dbNameOrPattern.toLowerCase()
- .getBytes(StandardCharsets.UTF_8.name(;
- Path currentDumpPath = new Path(dumpRoot, getNextDumpDir());
- Path hiveDumpRoot = new Path(currentDumpPath,
ReplUtils.REPL_HIVE_BASE_DIR);
- DumpMetaData dmd = new DumpMetaData(hiveDumpRoot, conf);
- // Initialize ReplChangeManager instance since we will require it to
encode file URI.
- ReplChangeManager.getInstance(conf);
- Path cmRoot = new Path(conf.getVar(HiveConf.ConfVars.REPLCMDIR));
- Long lastReplId;
- if (!dumpRoot.getFileSystem(conf).exists(dumpRoot)
- || dumpRoot.getFileSystem(conf).listStatus(dumpRoot).length ==
0) {
-lastReplId = bootStrapDump(hiveDumpRoot, dmd, cmRoot, hiveDb);
+ //First Check if external table copy work has been initialized, if so,
just do that and return.
Review comment:
Please add more comments here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404398)
Time Spent: 1h 10m (was: 1h)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[ https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404397&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404397 ] ASF GitHub Bot logged work on HIVE-22997: - Author: ASF GitHub Bot Created on: 17/Mar/20 01:48 Start Date: 17/Mar/20 01:48 Worklog Time Spent: 10m Work Description: aasha commented on pull request #951: HIVE-22997 : Copy external table to target during Repl Dump operation URL: https://github.com/apache/hive/pull/951#discussion_r393401308 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestTableLevelReplicationScenarios.java ## @@ -918,7 +922,9 @@ public void testRenameTableScenariosWithReplaceExternalTable() throws Throwable String newPolicy = primaryDbName + ".'(in[0-9]+)|(out1500)|(in2)'"; dumpWithClause = Arrays.asList( "'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname + "'='true'", -"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'" +"'" + HiveConf.ConfVars.REPL_BOOTSTRAP_EXTERNAL_TABLES.varname + "'='false'", Review comment: Can be added to a separate method This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 404397) Time Spent: 1h (was: 50m) > Copy external table to target during Repl Dump operation > > > Key: HIVE-22997 > URL: https://issues.apache.org/jira/browse/HIVE-22997 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch, > HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404396&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404396
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 01:45
Start Date: 17/Mar/20 01:45
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #951: HIVE-22997 : Copy
external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393400475
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -713,9 +732,11 @@ public void testExternalTableDataPath() throws Exception {
@Test
public void testExternalTablesIncReplicationWithConcurrentDropTable() throws
Throwable {
-List dumpWithClause = Collections.singletonList(
-"'" + HiveConf.ConfVars.REPL_INCLUDE_EXTERNAL_TABLES.varname +
"'='true'"
-);
+List dumpWithClause = Arrays.asList(
Review comment:
Can be separated in a common method
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404396)
Time Spent: 50m (was: 40m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404395&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404395
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 01:44
Start Date: 17/Mar/20 01:44
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #951: HIVE-22997 : Copy
external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393400307
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -436,16 +438,29 @@ public void externalTableIncrementalReplication() throws
Throwable {
}
List loadWithClause = externalTableBasePathWithClause();
-replica.load(replicatedDbName, primaryDbName, loadWithClause)
+replica.load(replicatedDbName, primaryDbName, withClause)
.run("use " + replicatedDbName)
.run("show tables like 't1'")
.verifyResult("t1")
.run("show partitions t1")
.verifyResults(new String[] { "country=india", "country=us" })
.run("select place from t1 order by place")
-.verifyResults(new String[] { "bangalore", "mumbai", "pune" })
+.verifyResults(new String[] {})
Review comment:
Why is there no data here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404395)
Time Spent: 40m (was: 0.5h)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404393&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404393
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 01:42
Start Date: 17/Mar/20 01:42
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #951: HIVE-22997 : Copy
external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393399950
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationScenariosExternalTables.java
##
@@ -397,17 +397,19 @@ public void externalTableWithPartitions() throws
Throwable {
primary.run("use " + primaryDbName)
.run("insert into table t2 partition(country='france') values
('lyon')")
.run("alter table t2 set location '" + tmpLocation2 + "'")
-.dump(primaryDbName);
+.dump(primaryDbName, withClause);
-replica.load(replicatedDbName, primaryDbName, loadWithClause);
+replica.load(replicatedDbName, primaryDbName, withClause);
assertTablePartitionLocation(primaryDbName + ".t2", replicatedDbName +
".t2");
}
@Test
public void externalTableIncrementalReplication() throws Throwable {
-WarehouseInstance.Tuple tuple = primary.dumpWithCommand("repl dump " +
primaryDbName);
+List withClause = externalTableBasePathWithClause();
+String replDumpCommand = "repl dump " + primaryDbName
++ " WITH (" + withClause.get(0) + "," + withClause.get(1) + ")";
+WarehouseInstance.Tuple tuple = primary.dumpWithCommand(replDumpCommand);
Review comment:
can use withClause directly
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404393)
Time Spent: 0.5h (was: 20m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22997) Copy external table to target during Repl Dump operation
[
https://issues.apache.org/jira/browse/HIVE-22997?focusedWorklogId=404392&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-404392
]
ASF GitHub Bot logged work on HIVE-22997:
-
Author: ASF GitHub Bot
Created on: 17/Mar/20 01:40
Start Date: 17/Mar/20 01:40
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #951: HIVE-22997 : Copy
external table to target during Repl Dump operation
URL: https://github.com/apache/hive/pull/951#discussion_r393399236
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOnHDFSEncryptedZones.java
##
@@ -102,12 +104,20 @@ public void
targetAndSourceHaveDifferentEncryptionZoneKeys() throws Throwable {
put(HiveConf.ConfVars.REPLDIR.varname, primary.repldDir);
}}, "test_key123");
+List dumpWithClause = Arrays.asList(
Review comment:
Why is this needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 404392)
Time Spent: 20m (was: 10m)
> Copy external table to target during Repl Dump operation
>
>
> Key: HIVE-22997
> URL: https://issues.apache.org/jira/browse/HIVE-22997
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22997.03.patch, HIVE-22997.04.patch,
> HIVE-22997.1.patch, HIVE-22997.2.patch, HIVE-22997.4.patch, HIVE-22997.5.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[
https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060558#comment-17060558
]
Hive QA commented on HIVE-22539:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 51 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} service: The patch generated 0 new + 25 unchanged -
6 fixed = 25 total (was 31) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} service generated 0 new + 50 unchanged - 1 fixed =
50 total (was 51) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21138/dev-support/hive-personality.sh
|
| git revision | master / 2be3080 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: service U: service |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21138/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HiveServer2 SPNEGO authentication should skip if authorization header is empty
> --
>
> Key: HIVE-22539
> URL: https://issues.apache.org/jira/browse/HIVE-22539
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Minor
> Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch,
> HIVE-22539.3.patch, HIVE-22539.4.patch, HIVE-22539.5.patch,
> HIVE-22539.6.patch, HIVE-22539.7.patch
>
>
> Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos
> before checking header. This can be checked up front to avoid doing any
> Kerberos related work if the header is empty. This is helpful in a lot of
> cases since typically the first request is empty with the client waiting for
> a 401 before returning the Authorization header.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23027) Fix syntax error in llap package.py
[ https://issues.apache.org/jira/browse/HIVE-23027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060532#comment-17060532 ] Rajesh Balamohan commented on HIVE-23027: - Committed to master. Thanks [~gopalv] . > Fix syntax error in llap package.py > --- > > Key: HIVE-23027 > URL: https://issues.apache.org/jira/browse/HIVE-23027 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-23027.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23027) Fix syntax error in llap package.py
[ https://issues.apache.org/jira/browse/HIVE-23027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23027: Fix Version/s: 4.0.0 > Fix syntax error in llap package.py > --- > > Key: HIVE-23027 > URL: https://issues.apache.org/jira/browse/HIVE-23027 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-23027.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23027) Fix syntax error in llap package.py
[ https://issues.apache.org/jira/browse/HIVE-23027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23027: Assignee: Rajesh Balamohan Resolution: Fixed Status: Resolved (was: Patch Available) > Fix syntax error in llap package.py > --- > > Key: HIVE-23027 > URL: https://issues.apache.org/jira/browse/HIVE-23027 > Project: Hive > Issue Type: Bug >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Trivial > Attachments: HIVE-23027.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060528#comment-17060528
]
Hive QA commented on HIVE-22841:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996834/HIVE-22841.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18108 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21137/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21137/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21137/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996834 - PreCommit-HIVE-Build
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch, HIVE-22841.2.patch,
> HIVE-22841.3.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060507#comment-17060507
]
Hive QA commented on HIVE-22841:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 51 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} jdbc in master has 16 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} service: The patch generated 0 new + 31 unchanged -
6 fixed = 31 total (was 37) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch jdbc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21137/dev-support/hive-personality.sh
|
| git revision | master / 87a560b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: service jdbc itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21137/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch, HIVE-22841.2.patch,
> HIVE-22841.3.patch
>
>
> Currently CookieSigner throws an IllegalArgum
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[
https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060441#comment-17060441
]
Denys Kuzmenko edited comment on HIVE-23032 at 3/16/20, 8:28 PM:
-
[~pvary], same for MySQL, however for Oracle/Postgres it gives better
performance.
*postgres:9.3* (4 tables x 1800 partitions)
{code}
batch insert
lockReq: {count=200, sum=720917, min=1790, average=3604.585000, max=6952}
total: {count=200, sum=430610, min=749, average=2153.05, max=5913}
batch insert+ reWriteBatchedInserts
lockReq: {count=200, sum=488272, min=1370, average=2441.36, max=3813}
total: {count=200, sum=283623, min=667, average=1418.115000, max=2780}
Multi-row insert
lockReq: {count=200, sum=771952, min=1853, average=3859.76, max=7817}
total: {count=200, sum=455352, min=768, average=2276.76, max=5880}
{code}
was (Author: dkuzmenko):
[~pvary], same for MySQL, however for Oracle/Postgres it gives better
performance.
*PostgreSQL* (4 tables x 1800 partitions)
{code}
batch insert
lockReq: {count=200, sum=720917, min=1790, average=3604.585000, max=6952}
total: {count=200, sum=430610, min=749, average=2153.05, max=5913}
batch insert+ reWriteBatchedInserts
lockReq: {count=200, sum=488272, min=1370, average=2441.36, max=3813}
total: {count=200, sum=283623, min=667, average=1418.115000, max=2780}
Multi-row insert
lockReq: {count=200, sum=771952, min=1853, average=3859.76, max=7817}
total: {count=200, sum=455352, min=768, average=2276.76, max=5880}
{code}
> Add batching in Lock generation
> ---
>
> Key: HIVE-23032
> URL: https://issues.apache.org/jira/browse/HIVE-23032
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23032.1.patch
>
>
> Replace multi-row insert in Oracle with batching. Performance tests showed
> significant performance improvement after turning batching on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23032) Add batching in Lock generation
[
https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060441#comment-17060441
]
Denys Kuzmenko edited comment on HIVE-23032 at 3/16/20, 8:20 PM:
-
[~pvary], same for MySQL, however for Oracle/Postgres it gives better
performance.
*PostgreSQL* (4 tables x 1800 partitions)
{code}
batch insert
lockReq: {count=200, sum=720917, min=1790, average=3604.585000, max=6952}
total: {count=200, sum=430610, min=749, average=2153.05, max=5913}
batch insert+ reWriteBatchedInserts
lockReq: {count=200, sum=488272, min=1370, average=2441.36, max=3813}
total: {count=200, sum=283623, min=667, average=1418.115000, max=2780}
Multi-row insert
lockReq: {count=200, sum=771952, min=1853, average=3859.76, max=7817}
total: {count=200, sum=455352, min=768, average=2276.76, max=5880}
{code}
was (Author: dkuzmenko):
[~pvary], same for MySQL, however for Oracle/Postgres it gives better
performance.
> Add batching in Lock generation
> ---
>
> Key: HIVE-23032
> URL: https://issues.apache.org/jira/browse/HIVE-23032
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23032.1.patch
>
>
> Replace multi-row insert in Oracle with batching. Performance tests showed
> significant performance improvement after turning batching on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[
https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060447#comment-17060447
]
Sungwoo commented on HIVE-21164:
[~kuczoram] In my testing, both select queries return non-empty lists, but the
final ORC table is empty (which implies that Tez is okay while our execution
engine has a bug):
{code:sql}
select
ss.ss_sold_time_sk,
...
ss.ss_net_profit,
ss.ss_sold_date_sk
where ss.ss_sold_date_sk is not null
{code}
{code:sql}
select
ss.ss_sold_time_sk,
...
ss.ss_net_profit,
ss.ss_sold_date_sk
where ss.ss_sold_date_sk is null
sort by ss.ss_sold_date_sk
{code}
If we use another second select query returning an empty list, the final ORC
table is still empty, but this observation is not useful here because even when
the second select query returns a non-empty list, the final ORC table is empty
anyway.
Let me try to set up an environment for testing Hive 4 on Tez (hopefully by the
end of this week) and report the result.
> ACID: explore how we can avoid a move step during inserts/compaction
>
>
> Key: HIVE-21164
> URL: https://issues.apache.org/jira/browse/HIVE-21164
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch,
> HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.12.patch,
> HIVE-21164.13.patch, HIVE-21164.14.patch, HIVE-21164.14.patch,
> HIVE-21164.15.patch, HIVE-21164.16.patch, HIVE-21164.17.patch,
> HIVE-21164.18.patch, HIVE-21164.19.patch, HIVE-21164.2.patch,
> HIVE-21164.20.patch, HIVE-21164.21.patch, HIVE-21164.22.patch,
> HIVE-21164.3.patch, HIVE-21164.4.patch, HIVE-21164.5.patch,
> HIVE-21164.6.patch, HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch
>
>
> Currently, we write compacted data to a temporary location and then move the
> files to a final location, which is an expensive operation on some cloud file
> systems. Since HIVE-20823 is already in, it can control the visibility of
> compacted data for the readers. Therefore, we can perhaps avoid writing data
> to a temporary location and directly write compacted data to the intended
> final path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23002) Optimise LazyBinaryUtils.writeVLong
[
https://issues.apache.org/jira/browse/HIVE-23002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060443#comment-17060443
]
Hive QA commented on HIVE-23002:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996823/HIVE-23002.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18090 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=162)
[unionDistinct_1.q,table_nonprintable.q,file_with_header_footer_aggregation.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,parquet_struct_type_vectorization.q,results_cache_diff_fs.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q,orc_merge3.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21136/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21136/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21136/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996823 - PreCommit-HIVE-Build
> Optimise LazyBinaryUtils.writeVLong
> ---
>
> Key: HIVE-23002
> URL: https://issues.apache.org/jira/browse/HIVE-23002
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23002.1.patch, HIVE-23002.2.patch,
> HIVE-23002.3.patch, Screenshot 2020-03-10 at 5.01.34 AM.jpg
>
>
> [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java#L420]
> It would be good to add a method which accepts scratch bytes.
>
> !Screenshot 2020-03-10 at 5.01.34 AM.jpg|width=452,height=321!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.19.patch Status: Patch Available (was: In Progress) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Status: In Progress (was: Patch Available) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.19.patch, HIVE-22990.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.18.patch Status: Patch Available (was: In Progress) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Status: In Progress (was: Patch Available) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.18.patch, HIVE-22990.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060441#comment-17060441 ] Denys Kuzmenko commented on HIVE-23032: --- [~pvary], same for MySQL, however for Oracle/Postgres it gives better performance. > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23031) Add option to enable transparent rewrite of count(distinct) into sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060439#comment-17060439 ] Zoltan Haindrich commented on HIVE-23031: - I should have been more brief in the description :) Thank you [~bslim] for thinking it through; I wanted to first concentrate on doing only the rewriting and do it for a concreate sketch impl (hll) - and see how well that works. bq. One sketches return an approximate and user want exact reporting. I don't want to force every query into this world - probably a feature toggle could be used to enable it. bq. how you will be mapping the sketching implementation to actual execution given that there is multiple sketches algorithms I thinked that rewriting for the udfs which are of the desired sketch family would make it happen. bq. let's treat whatever sketch you have in mind as a UDF and maybe add some as defaults udf that are trusted by the system Right now I'm not sure how this could be incorporated; but I keep this idea in mind - it could make it more easily customizable... > Add option to enable transparent rewrite of count(distinct) into sketch > functions > - > > Key: HIVE-23031 > URL: https://issues.apache.org/jira/browse/HIVE-23031 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23002) Optimise LazyBinaryUtils.writeVLong
[
https://issues.apache.org/jira/browse/HIVE-23002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060429#comment-17060429
]
Hive QA commented on HIVE-23002:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
39s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} serde: The patch generated 2 new + 55 unchanged - 0
fixed = 57 total (was 55) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21136/dev-support/hive-personality.sh
|
| git revision | master / 87a560b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21136/yetus/diff-checkstyle-serde.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21136/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimise LazyBinaryUtils.writeVLong
> ---
>
> Key: HIVE-23002
> URL: https://issues.apache.org/jira/browse/HIVE-23002
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23002.1.patch, HIVE-23002.2.patch,
> HIVE-23002.3.patch, Screenshot 2020-03-10 at 5.01.34 AM.jpg
>
>
> [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java#L420]
> It would be good to add a method which accepts scratch bytes.
>
> !Screenshot 2020-03-10 at 5.01.34 AM.jpg|width=452,height=321!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060422#comment-17060422
]
Hive QA commented on HIVE-22940:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996818/HIVE-22940.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18096 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver
(batchId=298)
org.apache.hadoop.hive.cli.TestKuduCliDriver.org.apache.hadoop.hive.cli.TestKuduCliDriver
(batchId=297)
org.apache.hadoop.hive.cli.TestKuduNegativeCliDriver.org.apache.hadoop.hive.cli.TestKuduNegativeCliDriver
(batchId=297)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21135/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21135/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21135/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996818 - PreCommit-HIVE-Build
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch,
> HIVE-22940.03.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.17.patch Status: Patch Available (was: In Progress) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.17.patch, > HIVE-22990.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22940) Make the datasketches functions available as predefined functions
[
https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060410#comment-17060410
]
Hive QA commented on HIVE-22940:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
38s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
30s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} ql: The patch generated 0 new + 82 unchanged - 2
fixed = 82 total (was 84) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
58s{color} | {color:green} root: The patch generated 0 new + 82 unchanged - 2
fixed = 82 total (was 84) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21135/dev-support/hive-personality.sh
|
| git revision | master / 5112a9e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21135/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make the datasketches functions available as predefined functions
> --
>
> Key: HIVE-22940
> URL: https://issues.apache.org/jira/browse/HIVE-22940
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch,
> HIVE-22940.03.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Status: In Progress (was: Patch Available) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work stopped] (HIVE-22928) Allow hive.exec.stagingdir to be a fully qualified directory name
[
https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-22928 stopped by Thomas Poepping.
--
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Affects Versions: 3.1.2
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
>Priority: Minor
> Attachments: HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory
> name that, for operations like {{insert}} or {{insert overwrite}}, will be
> placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very
> large (greater than the capacity of the HDFS cluster, as mentioned in a
> comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their
> staging directory to be an explicit blobstore path (or any filesystem path),
> rather than relying on Hive to intelligently build the blobstore path based
> on an interpretation of the job. We may lose locality guarantees, but because
> renames are just as expensive on blobstores no matter what the prefix is,
> this isn't considered a terribly large loss (assuming only blobstore
> customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually
> suffice in this case, as the stagingdir is not the same.
> This commit enables Hive customers to set an absolute location for all
> staging directories. For instances where the configured stagingdir scheme is
> not the same as the scheme for the table location, the default stagingdir
> configuration is used. This avoids a cross-filesystem rename, which is
> impossible anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22928) Allow hive.exec.stagingdir to be a fully qualified directory name
[
https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Poepping updated HIVE-22928:
---
Status: Patch Available (was: Open)
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Affects Versions: 3.1.2
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
>Priority: Minor
> Attachments: HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory
> name that, for operations like {{insert}} or {{insert overwrite}}, will be
> placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very
> large (greater than the capacity of the HDFS cluster, as mentioned in a
> comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their
> staging directory to be an explicit blobstore path (or any filesystem path),
> rather than relying on Hive to intelligently build the blobstore path based
> on an interpretation of the job. We may lose locality guarantees, but because
> renames are just as expensive on blobstores no matter what the prefix is,
> this isn't considered a terribly large loss (assuming only blobstore
> customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually
> suffice in this case, as the stagingdir is not the same.
> This commit enables Hive customers to set an absolute location for all
> staging directories. For instances where the configured stagingdir scheme is
> not the same as the scheme for the table location, the default stagingdir
> configuration is used. This avoids a cross-filesystem rename, which is
> impossible anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22928) Allow hive.exec.stagingdir to be a fully qualified directory name
[
https://issues.apache.org/jira/browse/HIVE-22928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Poepping updated HIVE-22928:
---
Status: In Progress (was: Patch Available)
> Allow hive.exec.stagingdir to be a fully qualified directory name
> -
>
> Key: HIVE-22928
> URL: https://issues.apache.org/jira/browse/HIVE-22928
> Project: Hive
> Issue Type: Improvement
> Components: Configuration, Hive
>Affects Versions: 3.1.2
>Reporter: Thomas Poepping
>Assignee: Thomas Poepping
>Priority: Minor
> Attachments: HIVE-22928.patch
>
>
> Currently, {{hive.exec.stagingdir}} can only be set as a relative directory
> name that, for operations like {{insert}} or {{insert overwrite}}, will be
> placed either under the table directory or the partition directory.
> For cases where an HDFS cluster is small but the data being inserted is very
> large (greater than the capacity of the HDFS cluster, as mentioned in a
> comment by [~ashutoshc] on [HIVE-14270]), the client may want to set their
> staging directory to be an explicit blobstore path (or any filesystem path),
> rather than relying on Hive to intelligently build the blobstore path based
> on an interpretation of the job. We may lose locality guarantees, but because
> renames are just as expensive on blobstores no matter what the prefix is,
> this isn't considered a terribly large loss (assuming only blobstore
> customers use this functionality).
> Note that {{hive.blobstore.use.blobstore.as.scratchdir}} doesn't actually
> suffice in this case, as the stagingdir is not the same.
> This commit enables Hive customers to set an absolute location for all
> staging directories. For instances where the configured stagingdir scheme is
> not the same as the scheme for the table location, the default stagingdir
> configuration is used. This avoids a cross-filesystem rename, which is
> impossible anyway.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22959) Extend storage-api to expose FilterContext
[ https://issues.apache.org/jira/browse/HIVE-22959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22959: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Panos! > Extend storage-api to expose FilterContext > -- > > Key: HIVE-22959 > URL: https://issues.apache.org/jira/browse/HIVE-22959 > Project: Hive > Issue Type: Sub-task > Components: storage-api >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22959.1.patch, HIVE-22959.2.patch, > HIVE-22959.3.patch, HIVE-22959.4.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > To enable row-level filtering at the ORC level ORC-577, or as an extension > ProDecode MapJoin HIVE-22731 we need a common context class that will hold > all the needed information for the filter. > I propose this class to be part of the storage-api – similar to > VectorizedRowBatch class and hold the information below: > * A boolean variable showing if the filter is enabled > * A int array storing the row Ids that are actually selected (passing the > filter) > * An int variable storing the the number or rows that passed the filter > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23033) MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE
[ https://issues.apache.org/jira/browse/HIVE-23033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati updated HIVE-23033: Attachment: HIVE-23033.1.patch Status: Patch Available (was: Open) > MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE > --- > > Key: HIVE-23033 > URL: https://issues.apache.org/jira/browse/HIVE-23033 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2, 3.1.1, 3.1.0, 3.0.0 >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0, 3.1.3 > > Attachments: HIVE-23033.1.patch > > > * The inital value for this table in the schema scripts was removed in > HIVE-17566: > https://github.com/apache/hive/commit/32b7abac961ca3879d23b074357f211fc7c49131#diff-3d1a4bae0d5d53c8e4ea79951ebf5eceL598 > * This was fixed in a number of scripts in HIVE-18781, but not for mssql: > https://github.com/apache/hive/commit/59483bca262880d3e7ef1b873d3c21176e9294cb#diff-4f43efd5a45cc362cb138287d90dbf82 > * This is as is since then > When using the schematool, the table gets initialized by other means. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22959) Extend storage-api to expose FilterContext
[ https://issues.apache.org/jira/browse/HIVE-22959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060386#comment-17060386 ] Gopal Vijayaraghavan commented on HIVE-22959: - LGTM - +1 > Extend storage-api to expose FilterContext > -- > > Key: HIVE-22959 > URL: https://issues.apache.org/jira/browse/HIVE-22959 > Project: Hive > Issue Type: Sub-task > Components: storage-api >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22959.1.patch, HIVE-22959.2.patch, > HIVE-22959.3.patch, HIVE-22959.4.patch > > Time Spent: 1h 20m > Remaining Estimate: 0h > > To enable row-level filtering at the ORC level ORC-577, or as an extension > ProDecode MapJoin HIVE-22731 we need a common context class that will hold > all the needed information for the filter. > I propose this class to be part of the storage-api – similar to > VectorizedRowBatch class and hold the information below: > * A boolean variable showing if the filter is enabled > * A int array storing the row Ids that are actually selected (passing the > filter) > * An int variable storing the the number or rows that passed the filter > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23032) Add batching in Lock generation
[
https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060384#comment-17060384
]
Peter Vary commented on HIVE-23032:
---
By my tests on the openTxn speedup, I feel that we will have a performance
degradation in mysql. Nothing conclusive yet (need some rechecks etc), but even
with optimization settings below the generated sql seems to executing slower
than the original one:
{code:title=HikariCPDataSourceProvider.java}
if (determineDatabaseProduct(driverUrl) == MYSQL) {
config.setConnectionInitSql("SET @@session.sql_mode=ANSI_QUOTES");
config.addDataSourceProperty("cachePrepStmts","true");
config.addDataSourceProperty("prepStmtCacheSize","250");
config.addDataSourceProperty("prepStmtCacheSqlLimit","2048");
config.addDataSourceProperty("useServerPrepStmts","true");
config.addDataSourceProperty("useLocalSessionState","true");
config.addDataSourceProperty("rewriteBatchedStatements","true");
config.addDataSourceProperty("cacheResultSetMetadata","true");
config.addDataSourceProperty("cacheServerConfiguration","true");
config.addDataSourceProperty("elideSetAutoCommits","true");
config.addDataSourceProperty("maintainTimeStats","false");
}
{code}
> Add batching in Lock generation
> ---
>
> Key: HIVE-23032
> URL: https://issues.apache.org/jira/browse/HIVE-23032
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-23032.1.patch
>
>
> Replace multi-row insert in Oracle with batching. Performance tests showed
> significant performance improvement after turning batching on.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060381#comment-17060381
]
Peter Vary commented on HIVE-22888:
---
[~dkuzmenko]: Please verify that MSSQL is accepting the new query.
Also which version of MySQL/PostgreSQL have you tried? Do we know what is the
consensus, which version of MySQL, PostgreSQL, Oracle, MSSql is supported?
CC: [~thejas], [~ngangam]
Also the query generation is a little bit convoluted, would it be possible to
provide a simple example query in a comment to show the end result?
Do we have a test code changes for the performance tests?
Thanks,
Peter
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.10.patch,
> HIVE-22888.2.patch, HIVE-22888.3.patch, HIVE-22888.4.patch,
> HIVE-22888.5.patch, HIVE-22888.6.patch, HIVE-22888.8.patch,
> HIVE-22888.9.patch, acid-lock-perf-test.pdf
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
> HL_LOCK_TYPE FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 333) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
> HL_LOCK_EXT_ID = 333) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
> LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR
> LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
> AND (LBC.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
> IS NOT NULL )
> OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
> OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23033) MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE
[ https://issues.apache.org/jira/browse/HIVE-23033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Lavati reassigned HIVE-23033: --- > MSSQL metastore schema init script doesn't initialize NOTIFICATION_SEQUENCE > --- > > Key: HIVE-23033 > URL: https://issues.apache.org/jira/browse/HIVE-23033 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2, 3.1.1, 3.1.0, 3.0.0 >Reporter: David Lavati >Assignee: David Lavati >Priority: Major > Fix For: 4.0.0, 3.1.3 > > > * The inital value for this table in the schema scripts was removed in > HIVE-17566: > https://github.com/apache/hive/commit/32b7abac961ca3879d23b074357f211fc7c49131#diff-3d1a4bae0d5d53c8e4ea79951ebf5eceL598 > * This was fixed in a number of scripts in HIVE-18781, but not for mssql: > https://github.com/apache/hive/commit/59483bca262880d3e7ef1b873d3c21176e9294cb#diff-4f43efd5a45cc362cb138287d90dbf82 > * This is as is since then > When using the schematool, the table gets initialized by other means. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[
https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060373#comment-17060373
]
Marta Kuczora commented on HIVE-21164:
--
Hi [~glapark],
I was trying to reproduce the issue. I can reproduce it only if the result of
the select in the last "INSERT OVERWRITE" part is empty. With your example
query:
{noformat}
from tpcds_text_2.store_sales ss
insert overwrite table store_sales partition (ss_sold_date_sk)
select
ss.ss_sold_time_sk,
...
ss.ss_net_profit,
ss.ss_sold_date_sk
where ss.ss_sold_date_sk is not null
insert overwrite table store_sales partition (ss_sold_date_sk)
select
ss.ss_sold_time_sk,
...
ss.ss_net_profit,
ss.ss_sold_date_sk
where ss.ss_sold_date_sk is null
sort by ss.ss_sold_date_sk
;
{noformat}
If the result of the
{noformat}
select
ss.ss_sold_time_sk,
...
ss.ss_net_profit,
ss.ss_sold_date_sk
where ss.ss_sold_date_sk is null
sort by ss.ss_sold_date_sk
{noformat}
is empty then the store_sales will be also empty after the query is finished.
Do you know the result of this last part in your environment? Is it possible
that your table contains no rows with ss_sold_date_sk=null?
What I see is that with this patch when I do an insert overwrite even with
dynamic partitioning, the data in the table will be overwritten. Without this
patch, this didn't happen with dynamic partitioning. When I do two insert
overwrites without this patch and dynamic partitioning is happening then the
result will contain the data from both inserts. The second one won't overwrite
the result of the first one as I would expect. So this behaviour seems to be
changed in this patch.
This is a good finding actually, thanks a lot for bringing it to my attention.
I will keep investigating this and will find a fix.
Until then, you can turn off this feature by setting the
'hive.acid.direct.insert.enabled' config parameter to false. With this, the
insert will happens just as before this patch. Or you can also try to do an
INSERT instead of the INSERT OVERWRITE.
> ACID: explore how we can avoid a move step during inserts/compaction
>
>
> Key: HIVE-21164
> URL: https://issues.apache.org/jira/browse/HIVE-21164
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.1.1
>Reporter: Vaibhav Gumashta
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch,
> HIVE-21164.11.patch, HIVE-21164.11.patch, HIVE-21164.12.patch,
> HIVE-21164.13.patch, HIVE-21164.14.patch, HIVE-21164.14.patch,
> HIVE-21164.15.patch, HIVE-21164.16.patch, HIVE-21164.17.patch,
> HIVE-21164.18.patch, HIVE-21164.19.patch, HIVE-21164.2.patch,
> HIVE-21164.20.patch, HIVE-21164.21.patch, HIVE-21164.22.patch,
> HIVE-21164.3.patch, HIVE-21164.4.patch, HIVE-21164.5.patch,
> HIVE-21164.6.patch, HIVE-21164.7.patch, HIVE-21164.8.patch, HIVE-21164.9.patch
>
>
> Currently, we write compacted data to a temporary location and then move the
> files to a final location, which is an expensive operation on some cloud file
> systems. Since HIVE-20823 is already in, it can control the visibility of
> compacted data for the readers. Therefore, we can perhaps avoid writing data
> to a temporary location and directly write compacted data to the intended
> final path.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23023) MR compaction ignores column schema evolution
[
https://issues.apache.org/jira/browse/HIVE-23023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060369#comment-17060369
]
Hive QA commented on HIVE-23023:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996816/HIVE-23023.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18102 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21134/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21134/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21134/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996816 - PreCommit-HIVE-Build
> MR compaction ignores column schema evolution
> -
>
> Key: HIVE-23023
> URL: https://issues.apache.org/jira/browse/HIVE-23023
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23023.01.patch, HIVE-23023.02.patch,
> HIVE-23023.03.patch
>
>
> Repro:
> {code:java}
> create table compaction_error(i int)
> partitioned by (`part1` string)
> stored as orc
> TBLPROPERTIES ('transactional'='true');
> insert into table compaction_error values (1, 'aa');
> ALTER TABLE compaction_error ADD COLUMNS (newcol string);
> insert into table compaction_error values (2, 2000, 'aa');
> alter table compaction_error partition (part1='aa') compact 'minor'; --or
> major
> {code}
> data row will look like:
> {code:java}
> 1, NULL, 'aa'
> 2, NULL, 'aa'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Status: In Progress (was: Patch Available) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[ https://issues.apache.org/jira/browse/HIVE-22990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22990: --- Attachment: HIVE-22990.16.patch Status: Patch Available (was: In Progress) > Build acknowledgement mechanism for repl dump and load > -- > > Key: HIVE-22990 > URL: https://issues.apache.org/jira/browse/HIVE-22990 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch, > HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch, > HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch, > HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch, > HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch, > HIVE-22990.15.patch, HIVE-22990.16.patch, HIVE-22990.patch > > Time Spent: 3.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23023) MR compaction ignores column schema evolution
[
https://issues.apache.org/jira/browse/HIVE-23023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060320#comment-17060320
]
Hive QA commented on HIVE-23023:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
31s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21134/dev-support/hive-personality.sh
|
| git revision | master / 5112a9e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21134/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MR compaction ignores column schema evolution
> -
>
> Key: HIVE-23023
> URL: https://issues.apache.org/jira/browse/HIVE-23023
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23023.01.patch, HIVE-23023.02.patch,
> HIVE-23023.03.patch
>
>
> Repro:
> {code:java}
> create table compaction_error(i int)
> partitioned by (`part1` string)
> stored as orc
> TBLPROPERTIES ('transactional'='true');
> insert into table compaction_error values (1, 'aa');
> ALTER TABLE compaction_error ADD COLUMNS (newcol string);
> insert into table compaction_error values (2, 2000, 'aa');
> alter table compaction_error partition (part1='aa') compact 'minor'; --or
> major
> {code}
> data row will look like:
> {code:java}
> 1, NULL, 'aa'
> 2, NULL, 'aa'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23031) Add option to enable transparent rewrite of count(distinct) into sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060299#comment-17060299 ] Slim Bouguerra commented on HIVE-23031: --- I do not see how this can work well given the followings: One sketches return an approximate and user want exact reporting. Second how you will be mapping the sketching implementation to actual execution given that there is multiple sketches algorithms Finally each sketch algorithm has some parameters like number of buckets etc how are you gonna allow the user to inject that. In a nutshell am saying let's treat whatever sketch you have in mind as a UDF and maybe add some as defaults udf that are trusted by the system. > Add option to enable transparent rewrite of count(distinct) into sketch > functions > - > > Key: HIVE-23031 > URL: https://issues.apache.org/jira/browse/HIVE-23031 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22955) PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
[
https://issues.apache.org/jira/browse/HIVE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060298#comment-17060298
]
Hive QA commented on HIVE-22955:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996815/HIVE-22955.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18101 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=185)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21133/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21133/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21133/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996815 - PreCommit-HIVE-Build
> PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
>
>
> Key: HIVE-22955
> URL: https://issues.apache.org/jira/browse/HIVE-22955
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22955.1.patch, HIVE-22955.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> 2020-02-26 20:22:49,683 ERROR [main] acid.PreUpgradeTool
> (PreUpgradeTool.java:main(150)) - PreUpgradeTool failed
> org.apache.hadoop.hive.ql.metadata.HiveException at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:283)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:146)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.IllegalStateException: Current state = RESET, new state = FLUSHED
> ...
> Caused by: java.lang.IllegalStateException: Current state = RESET, new state
> = FLUSHED at
> java.nio.charset.CharsetDecoder.throwIllegalStateException(CharsetDecoder.java:992)
> at java.nio.charset.CharsetDecoder.flush(CharsetDecoder.java:675) at
> java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:804) at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:606)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:567)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:464)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.processTable(PreUpgradeTool.java:374)
> {code}
> This is probably caused by HIVE-21948.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23011) Shared work optimizer should check residual predicates when comparing joins
[ https://issues.apache.org/jira/browse/HIVE-23011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-23011: --- Attachment: HIVE-23011.patch > Shared work optimizer should check residual predicates when comparing joins > --- > > Key: HIVE-23011 > URL: https://issues.apache.org/jira/browse/HIVE-23011 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch, > HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch, HIVE-23011.patch, > HIVE-23011.patch, HIVE-23011.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22940) Make the datasketches functions available as predefined functions
[ https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060294#comment-17060294 ] Jesus Camacho Rodriguez commented on HIVE-22940: +1 (pending tests) > Make the datasketches functions available as predefined functions > -- > > Key: HIVE-22940 > URL: https://issues.apache.org/jira/browse/HIVE-22940 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch, > HIVE-22940.03.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22970) Add a qoption to enable tests to use transactional mode
[ https://issues.apache.org/jira/browse/HIVE-22970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22970: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) pushed to master. Thank you Peter for reviewing the changes! > Add a qoption to enable tests to use transactional mode > --- > > Key: HIVE-22970 > URL: https://issues.apache.org/jira/browse/HIVE-22970 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22970.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > in scheduled query tests; the executor is launched form a qoption - however > scheduled queries do make a snapshot of the actual hiveconf and as such there > is no way to alter hiveconf keys for scheduled executions in the tests. > moving the "usual" transactional enabler settings to a qoption may also help > clean up our tests a bit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22992) ZkRegistryBase caching mechanism only caches the first instance
[
https://issues.apache.org/jira/browse/HIVE-22992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060266#comment-17060266
]
Prasanth Jayachandran commented on HIVE-22992:
--
{quote}When a node de-registers itself from zk, it will throw away all
instances from the cache running on that host.
{quote}
This is expected behaviour.
I am still wondering how were you able to run into this issue? If the port is
dynamic even running multiple daemons in a single host shouldn't be a problem.
Also IIRC the shutdown handler should trigger a deregister of daemon which
should throw away the cache.
The changes looks good but still I am unclear as in how this could happen.
> ZkRegistryBase caching mechanism only caches the first instance
> ---
>
> Key: HIVE-22992
> URL: https://issues.apache.org/jira/browse/HIVE-22992
> Project: Hive
> Issue Type: Bug
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Minor
> Attachments: HIVE-22992.01.patch, HIVE-22992.02.patch,
> HIVE-22992.03.patch
>
>
> ZkRegistryBase caching mechanism only caches the first instance of the llap
> node running on the same host.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23032: -- Attachment: HIVE-23032.1.patch > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-23032.1.patch > > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23032: -- Component/s: Locking > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-23032: -- Description: Replace multi-row insert in Oracle with batching. Performance tests showed significant performance improvement after turning batching on. > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement > Components: Locking >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > > Replace multi-row insert in Oracle with batching. Performance tests showed > significant performance improvement after turning batching on. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23032) Add batching in Lock generation
[ https://issues.apache.org/jira/browse/HIVE-23032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-23032: - Assignee: Denys Kuzmenko > Add batching in Lock generation > --- > > Key: HIVE-23032 > URL: https://issues.apache.org/jira/browse/HIVE-23032 > Project: Hive > Issue Type: Improvement >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22955) PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
[
https://issues.apache.org/jira/browse/HIVE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060248#comment-17060248
]
Hive QA commented on HIVE-22955:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
20s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} upgrade-acid/pre-upgrade: The patch generated 0 new
+ 15 unchanged - 38 fixed = 15 total (was 53) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21133/dev-support/hive-personality.sh
|
| git revision | master / 8abaea6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: upgrade-acid/pre-upgrade U: upgrade-acid/pre-upgrade |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21133/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
>
>
> Key: HIVE-22955
> URL: https://issues.apache.org/jira/browse/HIVE-22955
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22955.1.patch, HIVE-22955.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> 2020-02-26 20:22:49,683 ERROR [main] acid.PreUpgradeTool
> (PreUpgradeTool.java:main(150)) - PreUpgradeTool failed
> org.apache.hadoop.hive.ql.metadata.HiveException at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:283)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:146)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.IllegalStateException: Current state = RESET, new state = FLUSHED
> ...
> Caused by: java.lang.IllegalStateException: Current state = RESET, new state
> = FLUSHED at
> java.nio.charset.CharsetDecoder.throwIllegalStateException(CharsetDecoder.java:992)
> at java.nio.charset.CharsetDecoder.flush(CharsetDecoder.java:675) at
> java.nio.ch
[jira] [Updated] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HIVE-22841:
Status: Patch Available (was: Open)
Rebased patch on latest master - no conflicts. Thanks [~kgyrtkirk] for looking
it over.
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch, HIVE-22841.2.patch,
> HIVE-22841.3.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[ https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden updated HIVE-22539: Attachment: HIVE-22539.7.patch > HiveServer2 SPNEGO authentication should skip if authorization header is empty > -- > > Key: HIVE-22539 > URL: https://issues.apache.org/jira/browse/HIVE-22539 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Minor > Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch, > HIVE-22539.3.patch, HIVE-22539.4.patch, HIVE-22539.5.patch, > HIVE-22539.6.patch, HIVE-22539.7.patch > > > Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos > before checking header. This can be checked up front to avoid doing any > Kerberos related work if the header is empty. This is helpful in a lot of > cases since typically the first request is empty with the client waiting for > a 401 before returning the Authorization header. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HIVE-22841:
Status: Open (was: Patch Available)
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch, HIVE-22841.2.patch,
> HIVE-22841.3.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevin Risden updated HIVE-22841:
Attachment: HIVE-22841.3.patch
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch, HIVE-22841.2.patch,
> HIVE-22841.3.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[ https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden updated HIVE-22539: Attachment: HIVE-22539.6.patch > HiveServer2 SPNEGO authentication should skip if authorization header is empty > -- > > Key: HIVE-22539 > URL: https://issues.apache.org/jira/browse/HIVE-22539 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Minor > Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch, > HIVE-22539.3.patch, HIVE-22539.4.patch, HIVE-22539.5.patch, HIVE-22539.6.patch > > > Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos > before checking header. This can be checked up front to avoid doing any > Kerberos related work if the header is empty. This is helpful in a lot of > cases since typically the first request is empty with the client waiting for > a 401 before returning the Authorization header. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[ https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden updated HIVE-22539: Status: Patch Available (was: Open) Rebased patch to latest master - no conflicts. Thanks [~kgyrtkirk] for looking it over. > HiveServer2 SPNEGO authentication should skip if authorization header is empty > -- > > Key: HIVE-22539 > URL: https://issues.apache.org/jira/browse/HIVE-22539 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Minor > Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch, > HIVE-22539.3.patch, HIVE-22539.4.patch, HIVE-22539.5.patch, HIVE-22539.6.patch > > > Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos > before checking header. This can be checked up front to avoid doing any > Kerberos related work if the header is empty. This is helpful in a lot of > cases since typically the first request is empty with the client waiting for > a 401 before returning the Authorization header. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[ https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Risden updated HIVE-22539: Status: Open (was: Patch Available) > HiveServer2 SPNEGO authentication should skip if authorization header is empty > -- > > Key: HIVE-22539 > URL: https://issues.apache.org/jira/browse/HIVE-22539 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Minor > Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch, > HIVE-22539.3.patch, HIVE-22539.4.patch, HIVE-22539.5.patch, HIVE-22539.6.patch > > > Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos > before checking header. This can be checked up front to avoid doing any > Kerberos related work if the header is empty. This is helpful in a lot of > cases since typically the first request is empty with the client waiting for > a 401 before returning the Authorization header. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23029) LLAP: Shuffle Handler should support Index Cache configuration
[
https://issues.apache.org/jira/browse/HIVE-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060213#comment-17060213
]
Hive QA commented on HIVE-23029:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996810/HIVE-23029.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18096 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query24]
(batchId=306)
org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigrationEx.org.apache.hadoop.hive.ql.parse.TestReplicationWithTableMigrationEx
(batchId=274)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth (batchId=307)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21132/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21132/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21132/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996810 - PreCommit-HIVE-Build
> LLAP: Shuffle Handler should support Index Cache configuration
> --
>
> Key: HIVE-23029
> URL: https://issues.apache.org/jira/browse/HIVE-23029
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-23029.1.patch, Screenshot 2020-03-16 at 12.08.44
> PM.jpg
>
>
> !Screenshot 2020-03-16 at 12.08.44 PM.jpg|width=1592,height=1112!
>
> Queries like Q78 at large scale misses index cache with unordered edges. (24
> * 1009 = 24216. With the default 10 MB cache size, it can accommodate only
> 400+ entries).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22970) Add a qoption to enable tests to use transactional mode
[ https://issues.apache.org/jira/browse/HIVE-22970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060210#comment-17060210 ] Peter Vary commented on HIVE-22970: --- +1 > Add a qoption to enable tests to use transactional mode > --- > > Key: HIVE-22970 > URL: https://issues.apache.org/jira/browse/HIVE-22970 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22970.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > in scheduled query tests; the executor is launched form a qoption - however > scheduled queries do make a snapshot of the actual hiveconf and as such there > is no way to alter hiveconf keys for scheduled executions in the tests. > moving the "usual" transactional enabler settings to a qoption may also help > clean up our tests a bit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23029) LLAP: Shuffle Handler should support Index Cache configuration
[
https://issues.apache.org/jira/browse/HIVE-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060162#comment-17060162
]
Hive QA commented on HIVE-23029:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} llap-server: The patch generated 6 new + 5 unchanged -
0 fixed = 11 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21132/dev-support/hive-personality.sh
|
| git revision | master / 8abaea6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21132/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21132/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> LLAP: Shuffle Handler should support Index Cache configuration
> --
>
> Key: HIVE-23029
> URL: https://issues.apache.org/jira/browse/HIVE-23029
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Rajesh Balamohan
>Priority: Trivial
> Attachments: HIVE-23029.1.patch, Screenshot 2020-03-16 at 12.08.44
> PM.jpg
>
>
> !Screenshot 2020-03-16 at 12.08.44 PM.jpg|width=1592,height=1112!
>
> Queries like Q78 at large scale misses index cache with unordered edges. (24
> * 1009 = 24216. With the default 10 MB cache size, it can accommodate only
> 400+ entries).
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23031) Add option to enable transparent rewrite of count(distinct) into sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-23031: --- > Add option to enable transparent rewrite of count(distinct) into sketch > functions > - > > Key: HIVE-23031 > URL: https://issues.apache.org/jira/browse/HIVE-23031 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22990) Build acknowledgement mechanism for repl dump and load
[
https://issues.apache.org/jira/browse/HIVE-22990?focusedWorklogId=403881&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-403881
]
ASF GitHub Bot logged work on HIVE-22990:
-
Author: ASF GitHub Bot
Created on: 16/Mar/20 11:37
Start Date: 16/Mar/20 11:37
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #949: HIVE-22990 Add
file based ack for replication
URL: https://github.com/apache/hive/pull/949#discussion_r392789944
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplLoadCompleteAckTask.java
##
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.repl;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.ql.ErrorMsg;
+import org.apache.hadoop.hive.ql.exec.Task;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.parse.repl.dump.Utils;
+import org.apache.hadoop.hive.ql.plan.api.StageType;
+
+import java.io.Serializable;
+
+/**
+ * ReplLoadCompleteAckTask.
+ *
+ * Add the load complete acknoledgement.
+ **/
+public class ReplLoadCompleteAckTask extends Task
implements Serializable {
Review comment:
If we directly add the ack at the ReplLoadTask complete, its not correct.
There can be DDL Tasks or Copy Tasks which are not yet finished but the Repl
Load Task is complete. So we need to add this task at the end of the DAG so
that once all other tasks are complete without any error only then it writes
the ack
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 403881)
Time Spent: 3.5h (was: 3h 20m)
> Build acknowledgement mechanism for repl dump and load
> --
>
> Key: HIVE-22990
> URL: https://issues.apache.org/jira/browse/HIVE-22990
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22990.01.patch, HIVE-22990.02.patch,
> HIVE-22990.03.patch, HIVE-22990.04.patch, HIVE-22990.05.patch,
> HIVE-22990.06.patch, HIVE-22990.07.patch, HIVE-22990.08.patch,
> HIVE-22990.09.patch, HIVE-22990.10.patch, HIVE-22990.11.patch,
> HIVE-22990.12.patch, HIVE-22990.13.patch, HIVE-22990.14.patch,
> HIVE-22990.15.patch, HIVE-22990.patch
>
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23030) Enable sketch union-s to be rolled up
[ https://issues.apache.org/jira/browse/HIVE-23030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-23030: --- > Enable sketch union-s to be rolled up > - > > Key: HIVE-23030 > URL: https://issues.apache.org/jira/browse/HIVE-23030 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > > Enabling rolling up sketch aggregates could enable the matching of > materialized views created for higher dimensions to be applied for lower > dimension cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22970) Add a qoption to enable tests to use transactional mode
[ https://issues.apache.org/jira/browse/HIVE-22970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060108#comment-17060108 ] Zoltan Haindrich commented on HIVE-22970: - [~pvary] Could you please take a look? > Add a qoption to enable tests to use transactional mode > --- > > Key: HIVE-22970 > URL: https://issues.apache.org/jira/browse/HIVE-22970 > Project: Hive > Issue Type: Bug > Components: Tests >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22970.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > in scheduled query tests; the executor is launched form a qoption - however > scheduled queries do make a snapshot of the actual hiveconf and as such there > is no way to alter hiveconf keys for scheduled executions in the tests. > moving the "usual" transactional enabler settings to a qoption may also help > clean up our tests a bit -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060103#comment-17060103
]
Hive QA commented on HIVE-23004:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12996809/HIVE-23004.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18101 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_decimal_vectorized]
(batchId=177)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/21131/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/21131/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-21131/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12996809 - PreCommit-HIVE-Build
> Support Decimal64 operations across multiple vertices
> -
>
> Key: HIVE-23004
> URL: https://issues.apache.org/jira/browse/HIVE-23004
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch,
> HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch
>
>
> Support Decimal64 operations across multiple vertices
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23002) Optimise LazyBinaryUtils.writeVLong
[ https://issues.apache.org/jira/browse/HIVE-23002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan updated HIVE-23002: Attachment: HIVE-23002.3.patch > Optimise LazyBinaryUtils.writeVLong > --- > > Key: HIVE-23002 > URL: https://issues.apache.org/jira/browse/HIVE-23002 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23002.1.patch, HIVE-23002.2.patch, > HIVE-23002.3.patch, Screenshot 2020-03-10 at 5.01.34 AM.jpg > > > [https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java#L420] > It would be good to add a method which accepts scratch bytes. > > !Screenshot 2020-03-10 at 5.01.34 AM.jpg|width=452,height=321! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22841) ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner IllegalArgumentException on invalid cookie signature
[
https://issues.apache.org/jira/browse/HIVE-22841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060102#comment-17060102
]
Zoltan Haindrich commented on HIVE-22841:
-
+1
[~thejas]: do you have any concerns regarding this change?
> ThriftHttpServlet#getClientNameFromCookie should handle CookieSigner
> IllegalArgumentException on invalid cookie signature
> -
>
> Key: HIVE-22841
> URL: https://issues.apache.org/jira/browse/HIVE-22841
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Attachments: HIVE-22841.1.patch, HIVE-22841.2.patch
>
>
> Currently CookieSigner throws an IllegalArgumentException if the cookie
> signature is invalid.
> {code:java}
> if (!MessageDigest.isEqual(originalSignature.getBytes(),
> currentSignature.getBytes())) {
> throw new IllegalArgumentException("Invalid sign, original = " +
> originalSignature +
> " current = " + currentSignature);
> }
> {code}
> CookieSigner is only used in the ThriftHttpServlet#getClientNameFromCookie
> and doesn't handle the IllegalArgumentException. It is only checking if the
> value from the cookie is null or not.
> https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpServlet.java#L295
> {code:java}
> currValue = signer.verifyAndExtract(currValue);
> // Retrieve the user name, do the final validation step.
> if (currValue != null) {
> {code}
> This should be fixed to either:
> a) Have CookieSigner not return an IllegalArgumentException
> b) Improve ThriftHttpServlet to handle CookieSigner throwing an
> IllegalArgumentException
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22539) HiveServer2 SPNEGO authentication should skip if authorization header is empty
[ https://issues.apache.org/jira/browse/HIVE-22539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060096#comment-17060096 ] Zoltan Haindrich commented on HIVE-22539: - the patch makes sense to me +1 > HiveServer2 SPNEGO authentication should skip if authorization header is empty > -- > > Key: HIVE-22539 > URL: https://issues.apache.org/jira/browse/HIVE-22539 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Kevin Risden >Assignee: Kevin Risden >Priority: Minor > Attachments: HIVE-22539.1.patch, HIVE-22539.2.patch, > HIVE-22539.3.patch, HIVE-22539.4.patch, HIVE-22539.5.patch > > > Currently HiveServer2 SPNEGO authentication waits until setting up Kerberos > before checking header. This can be checked up front to avoid doing any > Kerberos related work if the header is empty. This is helpful in a lot of > cases since typically the first request is empty with the client waiting for > a 401 before returning the Authorization header. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22976) Oracle and MSSQL upgrade script missing the addition of WM_RESOURCEPLAN_FK1 constraint
[ https://issues.apache.org/jira/browse/HIVE-22976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22976: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to master. Thx Barna! > Oracle and MSSQL upgrade script missing the addition of WM_RESOURCEPLAN_FK1 > constraint > -- > > Key: HIVE-22976 > URL: https://issues.apache.org/jira/browse/HIVE-22976 > Project: Hive > Issue Type: Bug >Reporter: Barnabas Maidics >Assignee: Barnabas Maidics >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22976.1.patch, HIVE-22976.2.patch, > HIVE-22976.3.patch, HIVE-22976.4.patch > > > The schema init script (>=hive-schema-3.0.0) contains a constraint addition, > which is missing from the upgrade scripts in oracle and mssql. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22888) Rewrite checkLock inner select with JOIN operator
[
https://issues.apache.org/jira/browse/HIVE-22888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060088#comment-17060088
]
Denys Kuzmenko commented on HIVE-22888:
---
[~pvary], attached performance test measurements for MySql & Postgres.
In case of Oracle - the main bottleneck is multi-insert for locks generation,
replacing it with batching.
> Rewrite checkLock inner select with JOIN operator
> -
>
> Key: HIVE-22888
> URL: https://issues.apache.org/jira/browse/HIVE-22888
> Project: Hive
> Issue Type: Improvement
> Components: Locking
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22888.1.patch, HIVE-22888.10.patch,
> HIVE-22888.2.patch, HIVE-22888.3.patch, HIVE-22888.4.patch,
> HIVE-22888.5.patch, HIVE-22888.6.patch, HIVE-22888.8.patch,
> HIVE-22888.9.patch, acid-lock-perf-test.pdf
>
>
> - Created extra (db, tbl, part) index on HIVE_LOCKS table;
> - Replaced inner select under checkLocks using multiple IN statements with
> JOIN operator;
> generated query looks like :
> {code}
> SELECT LS.* FROM (
> SELECT HL_LOCK_EXT_ID, HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_STATE,
> HL_LOCK_TYPE FROM HIVE_LOCKS
> WHERE HL_LOCK_EXT_ID < 333) LS
> INNER JOIN (
> SELECT HL_DB, HL_TABLE, HL_PARTITION, HL_LOCK_TYPE FROM HIVE_LOCKS WHERE
> HL_LOCK_EXT_ID = 333) LBC
> ON LS.HL_DB = LBC.HL_DB
> AND (LS.HL_TABLE IS NULL OR LBC.HL_TABLE IS NULL OR LS.HL_TABLE =
> LBC.HL_TABLE
> AND (LS.HL_PARTITION IS NULL OR LBC.HL_PARTITION IS NULL OR
> LS.HL_PARTITION = LBC.HL_PARTITION))
> WHERE (LBC.HL_TXNID = 0 OR LS.HL_TXNID != LBC.HL_TXNID)
> AND (LBC.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NULL AND LS.HL_LOCK_TYPE='r' AND LBC.HL_TABLE
> IS NOT NULL )
> OR LBC.HL_LOCK_TYPE='w' AND LS.HL_LOCK_TYPE IN ('w','e')
> OR LBC.HL_LOCK_TYPE='r' AND LS.HL_LOCK_TYPE='e'
>AND !(LS.HL_TABLE IS NOT NULL AND LBC.HL_TABLE IS NULL))
> LIMIT 1;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22985) Failed compaction always throws TxnAbortedException
[
https://issues.apache.org/jira/browse/HIVE-22985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-22985:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~klcopp]!
> Failed compaction always throws TxnAbortedException
> ---
>
> Key: HIVE-22985
> URL: https://issues.apache.org/jira/browse/HIVE-22985
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Labels: compaction
> Fix For: 4.0.0
>
> Attachments: HIVE-22985.01.patch, HIVE-22985.02.patch,
> HIVE-22985.02.patch
>
>
> If compaction fails, its txn is aborted, however Worker attempts to commit it
> again in a finally statement. This results in a TxnAbortedException [1]
> thrown from TxnHandler#commitTxn
> We need to add a check and only try to commit at the end if the txn is not
> aborted.(TxnHandler#commitTxn does nothing if txn is already committed.)
> [1]
> {code:java}
> ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler -
> TxnAbortedException(message:Transaction txnid:16 already aborted)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.raiseTxnUnexpectedState(TxnHandler.java:4843)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.commitTxn(TxnHandler.java:1141)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.commit_txn(HiveMetaStore.java:8101)
> ...
> at
> org.apache.hadoop.hive.ql.txn.compactor.Worker.commitTxn(Worker.java:291)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:269)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23015) Fix HIVE_VECTORIZATION_GROUPBY_COMPLEX_TYPES_ENABLED definition
[
https://issues.apache.org/jira/browse/HIVE-23015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-23015:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~pvargacl]!
> Fix HIVE_VECTORIZATION_GROUPBY_COMPLEX_TYPES_ENABLED definition
> ---
>
> Key: HIVE-23015
> URL: https://issues.apache.org/jira/browse/HIVE-23015
> Project: Hive
> Issue Type: Bug
> Components: Vectorization
>Reporter: Peter Vary
>Assignee: Peter Varga
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-23015.patch
>
>
> The _HIVE_VECTORIZATION_GROUPBY_COMPLEX_TYPES_ENABLED_ is definied like this:
> {code}
> HIVE_VECTORIZATION_GROUPBY_COMPLEX_TYPES_ENABLED("hive.vectorized.groupby.complex.types.enabled",
> true,
> "This flag should be set to true to enable group by vectorization\n" +
> "of aggregations that use complex types.\n",
> "For example, AVG uses a complex type (STRUCT) for partial aggregation
> results" +
> "The default value is true."),
> {code}
> Notice that the end of the "of aggregations that use complex types.\n" line
> is a "," instead of "+".
> So this will end up using the following constructor using the second sentence
> as an "altName" :)
> {code}
> ConfVars(String varname, Object defaultVal, String description, String
> altName) {
> this(varname, defaultVal, null, description, true, false, altName);
> }
> {code}
> We should fix this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23019) Fix TestTxnCommandsForMmTable test case
[ https://issues.apache.org/jira/browse/HIVE-23019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060076#comment-17060076 ] Peter Vary commented on HIVE-23019: --- +1 > Fix TestTxnCommandsForMmTable test case > --- > > Key: HIVE-23019 > URL: https://issues.apache.org/jira/browse/HIVE-23019 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23019.2.patch, HIVE-23019.patch > > > TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable was fixed > in HIVE-19084 to avoid being dependent on the order of the element returned > by FileSystem.listStatus. > However the fix introduced a new bug, as now the assertion for the base > directory name doesn't run for the second partition, instead it runs twice > for the first one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22940) Make the datasketches functions available as predefined functions
[ https://issues.apache.org/jira/browse/HIVE-22940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22940: Attachment: HIVE-22940.03.patch > Make the datasketches functions available as predefined functions > -- > > Key: HIVE-22940 > URL: https://issues.apache.org/jira/browse/HIVE-22940 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22940.01.patch, HIVE-22940.02.patch, > HIVE-22940.03.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23004) Support Decimal64 operations across multiple vertices
[
https://issues.apache.org/jira/browse/HIVE-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060066#comment-17060066
]
Hive QA commented on HIVE-23004:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
49s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
46s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 2 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-21131/dev-support/hive-personality.sh
|
| git revision | master / 632ce66 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21131/yetus/whitespace-tabs.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-21131/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support Decimal64 operations across multiple vertices
> -
>
> Key: HIVE-23004
> URL: https://issues.apache.org/jira/browse/HIVE-23004
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-23004.1.patch, HIVE-23004.2.patch,
> HIVE-23004.4.patch, HIVE-23004.6.patch, HIVE-23004.7.patch
>
>
> Support Decimal64 operations across multiple vertices
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23019) Fix TestTxnCommandsForMmTable test case
[ https://issues.apache.org/jira/browse/HIVE-23019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060065#comment-17060065 ] Peter Varga commented on HIVE-23019: [~pvary] The sort was introduced in the previous attempt, the fix the test being dependent on the order which the FileSystem returns the elements, but it was only dependent on the order because of the indexing problem. Now that is fixed, the sort is unnecessary. > Fix TestTxnCommandsForMmTable test case > --- > > Key: HIVE-23019 > URL: https://issues.apache.org/jira/browse/HIVE-23019 > Project: Hive > Issue Type: Bug > Components: Test >Reporter: Peter Varga >Assignee: Peter Varga >Priority: Minor > Attachments: HIVE-23019.2.patch, HIVE-23019.patch > > > TestTxnCommandsForMmTable.testInsertOverwriteForPartitionedMmTable was fixed > in HIVE-19084 to avoid being dependent on the order of the element returned > by FileSystem.listStatus. > However the fix introduced a new bug, as now the assertion for the base > directory name doesn't run for the second partition, instead it runs twice > for the first one. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23023) MR compaction ignores column schema evolution
[
https://issues.apache.org/jira/browse/HIVE-23023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17060052#comment-17060052
]
László Pintér commented on HIVE-23023:
--
+1 (pending tests)
> MR compaction ignores column schema evolution
> -
>
> Key: HIVE-23023
> URL: https://issues.apache.org/jira/browse/HIVE-23023
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23023.01.patch, HIVE-23023.02.patch,
> HIVE-23023.03.patch
>
>
> Repro:
> {code:java}
> create table compaction_error(i int)
> partitioned by (`part1` string)
> stored as orc
> TBLPROPERTIES ('transactional'='true');
> insert into table compaction_error values (1, 'aa');
> ALTER TABLE compaction_error ADD COLUMNS (newcol string);
> insert into table compaction_error values (2, 2000, 'aa');
> alter table compaction_error partition (part1='aa') compact 'minor'; --or
> major
> {code}
> data row will look like:
> {code:java}
> 1, NULL, 'aa'
> 2, NULL, 'aa'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23023) MR compaction ignores column schema evolution
[
https://issues.apache.org/jira/browse/HIVE-23023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karen Coppage updated HIVE-23023:
-
Attachment: HIVE-23023.03.patch
> MR compaction ignores column schema evolution
> -
>
> Key: HIVE-23023
> URL: https://issues.apache.org/jira/browse/HIVE-23023
> Project: Hive
> Issue Type: Bug
>Reporter: Karen Coppage
>Assignee: Karen Coppage
>Priority: Major
> Attachments: HIVE-23023.01.patch, HIVE-23023.02.patch,
> HIVE-23023.03.patch
>
>
> Repro:
> {code:java}
> create table compaction_error(i int)
> partitioned by (`part1` string)
> stored as orc
> TBLPROPERTIES ('transactional'='true');
> insert into table compaction_error values (1, 'aa');
> ALTER TABLE compaction_error ADD COLUMNS (newcol string);
> insert into table compaction_error values (2, 2000, 'aa');
> alter table compaction_error partition (part1='aa') compact 'minor'; --or
> major
> {code}
> data row will look like:
> {code:java}
> 1, NULL, 'aa'
> 2, NULL, 'aa'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22955) PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
[
https://issues.apache.org/jira/browse/HIVE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hankó Gergely updated HIVE-22955:
-
Attachment: HIVE-22955.2.patch
Status: Patch Available (was: Open)
> PreUpgradeTool can fail because access to CharsetDecoder is not synchronized
>
>
> Key: HIVE-22955
> URL: https://issues.apache.org/jira/browse/HIVE-22955
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Hankó Gergely
>Assignee: Hankó Gergely
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22955.1.patch, HIVE-22955.2.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {code:java}
> 2020-02-26 20:22:49,683 ERROR [main] acid.PreUpgradeTool
> (PreUpgradeTool.java:main(150)) - PreUpgradeTool failed
> org.apache.hadoop.hive.ql.metadata.HiveException at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:283)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:146)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.IllegalStateException: Current state = RESET, new state = FLUSHED
> ...
> Caused by: java.lang.IllegalStateException: Current state = RESET, new state
> = FLUSHED at
> java.nio.charset.CharsetDecoder.throwIllegalStateException(CharsetDecoder.java:992)
> at java.nio.charset.CharsetDecoder.flush(CharsetDecoder.java:675) at
> java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:804) at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:606)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:567)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:464)
> at
> org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.processTable(PreUpgradeTool.java:374)
> {code}
> This is probably caused by HIVE-21948.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
