[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[ https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395606&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395606 ] ASF GitHub Bot logged work on HIVE-22865: - Author: ASF GitHub Bot Created on: 01/Mar/20 13:56 Start Date: 01/Mar/20 13:56 Worklog Time Spent: 10m Work Description: pkumarsinha commented on pull request #911: HIVE-22865 Include data in replication staging directory URL: https://github.com/apache/hive/pull/911#discussion_r386110262 ## File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java ## @@ -305,7 +309,7 @@ private Long incrementalDump(Path dumpRoot, DumpMetaData dmd, Path cmRoot, Hive NotificationEvent ev = evIter.next(); lastReplId = ev.getEventId(); Path evRoot = new Path(dumpRoot, String.valueOf(lastReplId)); - dumpEvent(ev, evRoot, cmRoot, hiveDb); + dumpEvent(ev, evRoot, dumpRoot, cmRoot, hiveDb); Review comment: hiveDumpRoot is recieved as dumpRoot in the current method. I haven't renamed existing parameter names. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 395606) Time Spent: 2.5h (was: 2h 20m) > Include data in replication staging directory > - > > Key: HIVE-22865 > URL: https://issues.apache.org/jira/browse/HIVE-22865 > Project: Hive > Issue Type: Task >Reporter: PRAVIN KUMAR SINHA >Assignee: PRAVIN KUMAR SINHA >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch, > HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22453) Describe table unnecessarily fetches partitions
[ https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048618#comment-17048618 ] Toshihiko Uchida commented on HIVE-22453: - [~vgarg] Thanks for your review and support, too. > Describe table unnecessarily fetches partitions > --- > > Key: HIVE-22453 > URL: https://issues.apache.org/jira/browse/HIVE-22453 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2, 2.3.6 >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22453.2.patch, HIVE-22453.2.patch, > HIVE-22453.3.patch, HIVE-22453.4.patch, HIVE-22453.patch > > > The simple describe table command without EXTENDED and FORMATTED (i.e., > DESCRIBE table_name) fetches all partitions when no partition is specified, > although it does not display partition statistics in nature. > The command should not fetch partitions since it can take a long time for a > large amount of partitions. > For instance, in our environment, the command takes around 8 seconds for a > table with 8760 (24 * 365) partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22944) Upgrade to Kryo5
[ https://issues.apache.org/jira/browse/HIVE-22944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Bodor updated HIVE-22944: Description: experimental ticket Maybe we should consider upgrading to kryo5 (plan ser/deser). Not sure about performance benefits, but looking at the code, e.g. FieldSerializer in Kryo5 seems to let us extend it easier (less private fields), which could be a benefit if we want to change its behavior, e.g. defining different logic for different fields of an object. Kryo 4 FieldSerializer: https://github.com/EsotericSoftware/kryo/blob/kryo-4/src/com/esotericsoftware/kryo/serializers/FieldSerializer.java Kryo 5 FieldSerialier: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/serializers/FieldSerializer.java versions, according to maven central: 5.0.0-RC4 - Apr, 2019 5.0.0-RC3 - Apr, 2019 5.0.0-RC2 - Feb, 2019 5.0.0-RC1 - Jun, 2018 there is no "5.0.0" yet TODO: why kryo-shaded artifact has been used so far? https://javalibs.com/artifact/com.esotericsoftware/kryo-shaded "This contains the shaded reflectasm jar to prevent conflicts with other versions of asm." was: Maybe we should consider upgrading to kryo5 (plan ser/deser). Not sure about performance benefits, but looking at the code, e.g. FieldSerializer in Kryo5 seems to let us extend it easier (less private fields), which could be a benefit if we want to change its behavior, e.g. defining different logic for different fields of an object. Kryo 4 FieldSerializer: https://github.com/EsotericSoftware/kryo/blob/kryo-4/src/com/esotericsoftware/kryo/serializers/FieldSerializer.java Kryo 5 FieldSerialier: https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/serializers/FieldSerializer.java versions, according to maven central: 5.0.0-RC4 - Apr, 2019 5.0.0-RC3 - Apr, 2019 5.0.0-RC2 - Feb, 2019 5.0.0-RC1 - Jun, 2018 there is no "5.0.0" yet TODO: why kryo-shaded artifact has been used so far? https://javalibs.com/artifact/com.esotericsoftware/kryo-shaded "This contains the shaded reflectasm jar to prevent conflicts with other versions of asm." > Upgrade to Kryo5 > > > Key: HIVE-22944 > URL: https://issues.apache.org/jira/browse/HIVE-22944 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: HIVE-22944.01.patch > > > experimental ticket > Maybe we should consider upgrading to kryo5 (plan ser/deser). Not sure about > performance benefits, but looking at the code, e.g. FieldSerializer in Kryo5 > seems to let us extend it easier (less private fields), which could be a > benefit if we want to change its behavior, e.g. defining different logic for > different fields of an object. > Kryo 4 FieldSerializer: > https://github.com/EsotericSoftware/kryo/blob/kryo-4/src/com/esotericsoftware/kryo/serializers/FieldSerializer.java > Kryo 5 FieldSerialier: > https://github.com/EsotericSoftware/kryo/blob/master/src/com/esotericsoftware/kryo/serializers/FieldSerializer.java > versions, according to maven central: > 5.0.0-RC4 - Apr, 2019 > 5.0.0-RC3 - Apr, 2019 > 5.0.0-RC2 - Feb, 2019 > 5.0.0-RC1 - Jun, 2018 > there is no "5.0.0" yet > TODO: why kryo-shaded artifact has been used so far? > https://javalibs.com/artifact/com.esotericsoftware/kryo-shaded > "This contains the shaded reflectasm jar to prevent conflicts with other > versions of asm." -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[
https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395656&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395656
]
ASF GitHub Bot logged work on HIVE-22865:
-
Author: ASF GitHub Bot
Created on: 01/Mar/20 18:32
Start Date: 01/Mar/20 18:32
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #911: HIVE-22865
Include data in replication staging directory
URL: https://github.com/apache/hive/pull/911#discussion_r386129803
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -582,10 +586,19 @@ void dumpTable(String dbName, String tblName, String
validTxnList, Path dbRoot,
}
MmContext mmCtx = MmContext.createIfNeeded(tableSpec.tableHandle);
tuple.replicationSpec.setRepl(true);
-new TableExport(
-exportPaths, tableSpec, tuple.replicationSpec, hiveDb, distCpDoAsUser,
conf, mmCtx).write();
-
+List replPathMappings = new TableExport(
+exportPaths, tableSpec, tuple.replicationSpec, hiveDb,
distCpDoAsUser, conf, mmCtx).write(false);
replLogger.tableLog(tblName, tableSpec.tableHandle.getTableType());
+if (Utils.shouldDumpMetaDataOnly(tuple.object, conf)) {
Review comment:
Why is this check needed here? Is it not done previously
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395656)
Time Spent: 2h 40m (was: 2.5h)
> Include data in replication staging directory
> -
>
> Key: HIVE-22865
> URL: https://issues.apache.org/jira/browse/HIVE-22865
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch,
> HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[
https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395657&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395657
]
ASF GitHub Bot logged work on HIVE-22865:
-
Author: ASF GitHub Bot
Created on: 01/Mar/20 18:32
Start Date: 01/Mar/20 18:32
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #911: HIVE-22865
Include data in replication staging directory
URL: https://github.com/apache/hive/pull/911#discussion_r386129824
##
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java
##
@@ -582,10 +586,19 @@ void dumpTable(String dbName, String tblName, String
validTxnList, Path dbRoot,
}
MmContext mmCtx = MmContext.createIfNeeded(tableSpec.tableHandle);
tuple.replicationSpec.setRepl(true);
-new TableExport(
-exportPaths, tableSpec, tuple.replicationSpec, hiveDb, distCpDoAsUser,
conf, mmCtx).write();
-
+List replPathMappings = new TableExport(
+exportPaths, tableSpec, tuple.replicationSpec, hiveDb,
distCpDoAsUser, conf, mmCtx).write(false);
replLogger.tableLog(tblName, tableSpec.tableHandle.getTableType());
+if (Utils.shouldDumpMetaDataOnly(tuple.object, conf)) {
+ return;
+}
+for (ReplPathMapping replPathMapping: replPathMappings) {
+ Task copyTask = ReplCopyTask.getLoadCopyTask(
+ tuple.replicationSpec, replPathMapping.getSrcPath(),
replPathMapping.getTargetPath(), conf, false);
+ this.addDependentTask(copyTask);
Review comment:
Dynamic DAG generation needed here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395657)
Time Spent: 2h 50m (was: 2h 40m)
> Include data in replication staging directory
> -
>
> Key: HIVE-22865
> URL: https://issues.apache.org/jira/browse/HIVE-22865
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch,
> HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[
https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395658&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395658
]
ASF GitHub Bot logged work on HIVE-22865:
-
Author: ASF GitHub Bot
Created on: 01/Mar/20 18:37
Start Date: 01/Mar/20 18:37
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #911: HIVE-22865
Include data in replication staging directory
URL: https://github.com/apache/hive/pull/911#discussion_r386130216
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AbstractEventHandler.java
##
@@ -71,4 +85,31 @@ public long fromEventId() {
public long toEventId() {
return event.getEventId();
}
+
+ public void writeFileEntry(String dbName, Table table, String file,
BufferedWriter fileListWriter,
+ Context withinContext) throws IOException,
LoginException {
+HiveConf hiveConf = withinContext.hiveConf;
+String distCpDoAsUser =
hiveConf.getVar(HiveConf.ConfVars.HIVE_DISTCP_DOAS_USER);
+if (Utils.shouldDumpMetaDataOnly(table, withinContext.hiveConf)) {
Review comment:
If its metadata only, why are we dumping data
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395658)
Time Spent: 3h (was: 2h 50m)
> Include data in replication staging directory
> -
>
> Key: HIVE-22865
> URL: https://issues.apache.org/jira/browse/HIVE-22865
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch,
> HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[
https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395660&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395660
]
ASF GitHub Bot logged work on HIVE-22865:
-
Author: ASF GitHub Bot
Created on: 01/Mar/20 18:38
Start Date: 01/Mar/20 18:38
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #911: HIVE-22865
Include data in replication staging directory
URL: https://github.com/apache/hive/pull/911#discussion_r386130273
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AbstractEventHandler.java
##
@@ -71,4 +85,31 @@ public long fromEventId() {
public long toEventId() {
return event.getEventId();
}
+
+ public void writeFileEntry(String dbName, Table table, String file,
BufferedWriter fileListWriter,
Review comment:
This can be in util class like before. Don't think its appropriate in a
event handler class
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395660)
Time Spent: 3h 20m (was: 3h 10m)
> Include data in replication staging directory
> -
>
> Key: HIVE-22865
> URL: https://issues.apache.org/jira/browse/HIVE-22865
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch,
> HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch
>
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22865) Include data in replication staging directory
[
https://issues.apache.org/jira/browse/HIVE-22865?focusedWorklogId=395659&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395659
]
ASF GitHub Bot logged work on HIVE-22865:
-
Author: ASF GitHub Bot
Created on: 01/Mar/20 18:38
Start Date: 01/Mar/20 18:38
Worklog Time Spent: 10m
Work Description: aasha commented on pull request #911: HIVE-22865
Include data in replication staging directory
URL: https://github.com/apache/hive/pull/911#discussion_r386130273
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/events/AbstractEventHandler.java
##
@@ -71,4 +85,31 @@ public long fromEventId() {
public long toEventId() {
return event.getEventId();
}
+
+ public void writeFileEntry(String dbName, Table table, String file,
BufferedWriter fileListWriter,
Review comment:
This can be in util class like before.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395659)
Time Spent: 3h 10m (was: 3h)
> Include data in replication staging directory
> -
>
> Key: HIVE-22865
> URL: https://issues.apache.org/jira/browse/HIVE-22865
> Project: Hive
> Issue Type: Task
>Reporter: PRAVIN KUMAR SINHA
>Assignee: PRAVIN KUMAR SINHA
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22865.1.patch, HIVE-22865.2.patch,
> HIVE-22865.3.patch, HIVE-22865.4.patch, HIVE-22865.5.patch, HIVE-22865.6.patch
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048652#comment-17048652 ] Vineet Garg commented on HIVE-22136: [~hashutosh] [~jcamachorodriguez] Can you take a look? > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch, HIVE-22136.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048658#comment-17048658 ] Jesus Camacho Rodriguez commented on HIVE-22136: +1 > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch, HIVE-22136.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.12.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Description: [https://github.com/apache/hive/pull/927] > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.patch > > > [https://github.com/apache/hive/pull/927] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?focusedWorklogId=395669&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395669
]
ASF GitHub Bot logged work on HIVE-21778:
-
Author: ASF GitHub Bot
Created on: 01/Mar/20 19:47
Start Date: 01/Mar/20 19:47
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #928: HIVE-21778
CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
miss in the query
URL: https://github.com/apache/hive/pull/928#discussion_r386135020
##
File path: ql/src/test/queries/clientpositive/structin.q
##
@@ -21,3 +21,14 @@ IN (
struct('1234--0074578664','3'),
struct('1234--0074578695',1)
);
+
+CREATE TABLE test_struct
+(
+ f1 string,
+ demo_struct struct,
+ datestr string
+);
+
Review comment:
Can we insert a few rows here and add the SELECT query? Just in case that
`IS NOT NULL` predicate in the explain goes away with a future change, we do
not confuse it with an optimization.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395669)
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0, 2.3.5
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, test_null.q, test_null.q.out
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean) <- Note that demo_struct filter is added when CBO is
> turned off
> Filter Operator
> predicate: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate
> this filter.
> {noformat}
> RexNode factoredFilterExpr = RexUtil
> .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter
> correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?focusedWorklogId=395667&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395667
]
ASF GitHub Bot logged work on HIVE-21778:
-
Author: ASF GitHub Bot
Created on: 01/Mar/20 19:47
Start Date: 01/Mar/20 19:47
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #928: HIVE-21778
CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
miss in the query
URL: https://github.com/apache/hive/pull/928#discussion_r386134508
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTBuilder.java
##
@@ -243,6 +243,7 @@ public static ASTNode literal(RexLiteral literal) {
case INTERVAL_SECOND:
case INTERVAL_YEAR:
case INTERVAL_YEAR_MONTH:
+case ROW:
Review comment:
`ROW` is missing in the type switch in L267. AFAIK it makes sense because
ROW type could only be a literal if it is NULL. However, can we add it at the
end of the switch with a short comment so we recall this (e.g., as it was done
with `BINARY`)?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395667)
Time Spent: 20m (was: 10m)
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0, 2.3.5
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, test_null.q, test_null.q.out
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean) <- Note that demo_struct filter is added when CBO is
> turned off
> Filter Operator
> predicate: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate
> this filter.
> {noformat}
> RexNode factoredFilterExpr = RexUtil
> .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter
> correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
[
https://issues.apache.org/jira/browse/HIVE-21778?focusedWorklogId=395668&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395668
]
ASF GitHub Bot logged work on HIVE-21778:
-
Author: ASF GitHub Bot
Created on: 01/Mar/20 19:47
Start Date: 01/Mar/20 19:47
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #928: HIVE-21778
CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
miss in the query
URL: https://github.com/apache/hive/pull/928#discussion_r386134721
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/TypeConverter.java
##
@@ -160,7 +160,7 @@ public static RelDataType convert(TypeInfo type,
RelDataTypeFactory dtFactory)
convertedType = convert((UnionTypeInfo) type, dtFactory);
break;
}
-return convertedType;
+return dtFactory.createTypeWithNullability(convertedType, true);
Review comment:
This makes sense because Hive does not have a concept of not nullable type.
Can we add a comment?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395668)
Time Spent: 0.5h (was: 20m)
> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter
> miss in the query
> --
>
> Key: HIVE-21778
> URL: https://issues.apache.org/jira/browse/HIVE-21778
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 4.0.0, 2.3.5
>Reporter: Rajesh Balamohan
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch,
> HIVE-21778.3.patch, test_null.q, test_null.q.out
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
> f1 string,
> demo_struct struct,
> datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: (datestr = '2019-01-01') (type: boolean) <- Note
> that demo_struct filter is not added here
> Filter Operator
> predicate: (datestr = '2019-01-01') (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and
> demo_struct is not null;
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: -1
> Processor Tree:
> TableScan
> alias: test_struct
> filterExpr: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean) <- Note that demo_struct filter is added when CBO is
> turned off
> Filter Operator
> predicate: ((datestr = '2019-01-01') and demo_struct is not null)
> (type: boolean)
> Select Operator
> expressions: f1 (type: string), demo_struct (type:
> struct), '2019-01-01' (type: string)
> outputColumnNames: _col0, _col1, _col2
> ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate
> this filter.
> {noformat}
> RexNode factoredFilterExpr = RexUtil
> .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter
> correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.13.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.patch > > > [https://github.com/apache/hive/pull/927] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.patch > > > [https://github.com/apache/hive/pull/927] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048671#comment-17048671
]
Hive QA commented on HIVE-22926:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
56s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 1322
unchanged - 9 fixed = 1324 total (was 1331) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20900/dev-support/hive-personality.sh
|
| git revision | master / e529a16 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20900/yetus/diff-checkstyle-itests_hive-unit.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20900/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20900/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.patch
>
>
> [https://github.com/apache/hive/pull/927]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048678#comment-17048678
]
Hive QA commented on HIVE-22926:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995077/HIVE-22926.12.patch
{color:green}SUCCESS:{color} +1 due to 22 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 18089 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=261)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootStrapDumpOfWarehouse
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testIncrementalDumpMultiIteration
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.testTaskCreationOptimization
(batchId=260)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTaskCreationOptimization
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootStrapDumpOfWarehouse
(batchId=273)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testIncrementalDumpMultiIteration
(batchId=273)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTables.testIncrementalDumpEmptyDumpDirectory
(batchId=267)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosExternalTablesMetaDataOnly.testIncrementalDumpEmptyDumpDirectory
(batchId=270)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosBasic
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosExternalTable
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosUpgrade
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestTableLevelReplicationScenarios.testRenameTableScenariosWithReplacePolicyDMLOperattion
(batchId=264)
org.apache.hive.minikdc.TestJdbcWithMiniKdcSQLAuthBinary.testAuthorization1
(batchId=307)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20900/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20900/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20900/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995077 - PreCommit-HIVE-Build
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.patch
>
>
> [https://github.com/apache/hive/pull/927]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048691#comment-17048691
]
Hive QA commented on HIVE-22926:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
54s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 1322
unchanged - 9 fixed = 1324 total (was 1331) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20901/dev-support/hive-personality.sh
|
| git revision | master / e529a16 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20901/yetus/diff-checkstyle-itests_hive-unit.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20901/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20901/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.patch
>
>
> [https://github.com/apache/hive/pull/927]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048703#comment-17048703
]
Hive QA commented on HIVE-22926:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995078/HIVE-22926.13.patch
{color:green}SUCCESS:{color} +1 due to 22 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 18093 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=78)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testBootStrapDumpOfWarehouse
(batchId=268)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testBootStrapDumpOfWarehouse
(batchId=273)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20901/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20901/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20901/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995078 - PreCommit-HIVE-Build
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.patch
>
>
> [https://github.com/apache/hive/pull/927]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.patch > > > [https://github.com/apache/hive/pull/927] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.14.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.14.patch, > HIVE-22926.patch > > > [https://github.com/apache/hive/pull/927] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22872) Support multiple executors for scheduled queries
[
https://issues.apache.org/jira/browse/HIVE-22872?focusedWorklogId=395698&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395698
]
ASF GitHub Bot logged work on HIVE-22872:
-
Author: ASF GitHub Bot
Created on: 02/Mar/20 01:35
Start Date: 02/Mar/20 01:35
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #924: HIVE-22872
schq executors
URL: https://github.com/apache/hive/pull/924#discussion_r386162020
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/scheduled/ScheduledQueryExecutionService.java
##
@@ -48,18 +53,29 @@
private static ScheduledQueryExecutionService INSTANCE = null;
private ScheduledQueryExecutionContext context;
- private ScheduledQueryExecutor worker;
private AtomicInteger forcedScheduleCheckCounter = new AtomicInteger();
+ private ScheduledQueryPoller poller;
+ AtomicInteger usedExecutors = new AtomicInteger(0);
+ List runningWorkers = new LinkedList<>();
public static ScheduledQueryExecutionService
startScheduledQueryExecutorService(HiveConf inputConf) {
HiveConf conf = new HiveConf(inputConf);
MetastoreBasedScheduledQueryService qService = new
MetastoreBasedScheduledQueryService(conf);
-ExecutorService executor = Executors.newCachedThreadPool(
-new ThreadFactoryBuilder().setDaemon(true).setNameFormat("Scheduled
Query Thread %d").build());
+ExecutorService executor = buildExecutor(conf);
ScheduledQueryExecutionContext ctx = new
ScheduledQueryExecutionContext(executor, conf, qService);
return startScheduledQueryExecutorService(ctx);
}
+ private static ExecutorService buildExecutor(HiveConf conf) {
+ThreadFactory threadFactory = new
ThreadFactoryBuilder().setDaemon(true).setNameFormat("Scheduled Query Thread
%d").build();
+int systemThreads = 2; // poller,reporter
+int maxServiceThreads =
conf.getIntVar(ConfVars.HIVE_SCHEDULED_QUERIES_MAX_EXECUTORS);
+return new ThreadPoolExecutor(systemThreads + 1, systemThreads +
maxServiceThreads,
Review comment:
Can we leave a comment here? It seems we are always keeping at least poller,
reporter, and an executor? It seems to me that minimum number of conf value
HIVE_SCHEDULED_QUERIES_MAX_EXECUTORS should be 1 for this to work properly;
though this is probably evident, we may leave a bit of this information to
users in HiveConf description of the property.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395698)
> Support multiple executors for scheduled queries
>
>
> Key: HIVE-22872
> URL: https://issues.apache.org/jira/browse/HIVE-22872
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch,
> HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch,
> HIVE-22872.06.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22872) Support multiple executors for scheduled queries
[
https://issues.apache.org/jira/browse/HIVE-22872?focusedWorklogId=395700&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395700
]
ASF GitHub Bot logged work on HIVE-22872:
-
Author: ASF GitHub Bot
Created on: 02/Mar/20 01:35
Start Date: 02/Mar/20 01:35
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #924: HIVE-22872
schq executors
URL: https://github.com/apache/hive/pull/924#discussion_r386163163
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/scheduled/ScheduledQueryExecutionService.java
##
@@ -183,33 +232,34 @@ private String getErrorStringForException(Throwable t) {
@Override
public void run() {
- while (true) {
+ while (!context.executor.isShutdown()) {
try {
Thread.sleep(context.getProgressReporterSleepTime());
} catch (InterruptedException e) {
LOG.warn("interrupt discarded");
}
try {
- worker.reportQueryProgress();
+ for (ScheduledQueryExecutor worker : runningWorkers) {
Review comment:
Can we hit a concurrent modification exception if a worker finishes and is
deleted, while we are iterating here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395700)
Time Spent: 50m (was: 40m)
> Support multiple executors for scheduled queries
>
>
> Key: HIVE-22872
> URL: https://issues.apache.org/jira/browse/HIVE-22872
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch,
> HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch,
> HIVE-22872.06.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22872) Support multiple executors for scheduled queries
[ https://issues.apache.org/jira/browse/HIVE-22872?focusedWorklogId=395696&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395696 ] ASF GitHub Bot logged work on HIVE-22872: - Author: ASF GitHub Bot Created on: 02/Mar/20 01:35 Start Date: 02/Mar/20 01:35 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #924: HIVE-22872 schq executors URL: https://github.com/apache/hive/pull/924#discussion_r386141036 ## File path: ql/src/java/org/apache/hadoop/hive/ql/scheduled/ScheduledQueryExecutionService.java ## @@ -48,18 +53,29 @@ private static ScheduledQueryExecutionService INSTANCE = null; private ScheduledQueryExecutionContext context; - private ScheduledQueryExecutor worker; private AtomicInteger forcedScheduleCheckCounter = new AtomicInteger(); + private ScheduledQueryPoller poller; + AtomicInteger usedExecutors = new AtomicInteger(0); + List runningWorkers = new LinkedList<>(); Review comment: private? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 395696) Time Spent: 0.5h (was: 20m) > Support multiple executors for scheduled queries > > > Key: HIVE-22872 > URL: https://issues.apache.org/jira/browse/HIVE-22872 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch, > HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch, > HIVE-22872.06.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22872) Support multiple executors for scheduled queries
[
https://issues.apache.org/jira/browse/HIVE-22872?focusedWorklogId=395701&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395701
]
ASF GitHub Bot logged work on HIVE-22872:
-
Author: ASF GitHub Bot
Created on: 02/Mar/20 01:35
Start Date: 02/Mar/20 01:35
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #924: HIVE-22872
schq executors
URL: https://github.com/apache/hive/pull/924#discussion_r386141211
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/schq/ScheduledQueryMaintenanceTask.java
##
@@ -46,7 +47,8 @@ public int execute() {
ScheduledQueryMaintenanceRequest request = buildScheduledQueryRequest();
try {
Hive.get().getMSC().scheduledQueryMaintenance(request);
- if (work.getScheduledQuery().isSetNextExecution()) {
+ if (work.getScheduledQuery().isSetNextExecution()
Review comment:
Can we add short comment explaining when we trigger a `forceScheduleCheck`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395701)
Time Spent: 1h (was: 50m)
> Support multiple executors for scheduled queries
>
>
> Key: HIVE-22872
> URL: https://issues.apache.org/jira/browse/HIVE-22872
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch,
> HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch,
> HIVE-22872.06.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22872) Support multiple executors for scheduled queries
[
https://issues.apache.org/jira/browse/HIVE-22872?focusedWorklogId=395697&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395697
]
ASF GitHub Bot logged work on HIVE-22872:
-
Author: ASF GitHub Bot
Created on: 02/Mar/20 01:35
Start Date: 02/Mar/20 01:35
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #924: HIVE-22872
schq executors
URL: https://github.com/apache/hive/pull/924#discussion_r386162395
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/scheduled/ScheduledQueryExecutionService.java
##
@@ -73,41 +89,42 @@ public static ScheduledQueryExecutionService
startScheduledQueryExecutorService(
private ScheduledQueryExecutionService(ScheduledQueryExecutionContext ctx) {
context = ctx;
-ctx.executor.submit(worker = new ScheduledQueryExecutor());
+ctx.executor.submit(poller = new ScheduledQueryPoller());
ctx.executor.submit(new ProgressReporter());
}
static boolean isTerminalState(QueryState state) {
return state == QueryState.FINISHED || state == QueryState.FAILED;
}
- class ScheduledQueryExecutor implements Runnable {
-
-private ScheduledQueryProgressInfo info;
+ class ScheduledQueryPoller implements Runnable {
Review comment:
Can we leave a comment with high level purpose of the Poller class? Also,
probably mentioning that there should be only one-at-the-time per HS2.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395697)
Time Spent: 40m (was: 0.5h)
> Support multiple executors for scheduled queries
>
>
> Key: HIVE-22872
> URL: https://issues.apache.org/jira/browse/HIVE-22872
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch,
> HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch,
> HIVE-22872.06.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22872) Support multiple executors for scheduled queries
[
https://issues.apache.org/jira/browse/HIVE-22872?focusedWorklogId=395699&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395699
]
ASF GitHub Bot logged work on HIVE-22872:
-
Author: ASF GitHub Bot
Created on: 02/Mar/20 01:35
Start Date: 02/Mar/20 01:35
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #924: HIVE-22872
schq executors
URL: https://github.com/apache/hive/pull/924#discussion_r386162944
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/scheduled/ScheduledQueryExecutionService.java
##
@@ -116,6 +133,37 @@ private void sleep(long idleSleepTime) throws
InterruptedException {
}
}
+ }
+
+ private void workerStarted(ScheduledQueryExecutor executor) {
+runningWorkers.add(executor);
+usedExecutors.incrementAndGet();
+ }
+
+ private void workerStopped(ScheduledQueryExecutor executor) {
+usedExecutors.decrementAndGet();
+runningWorkers.remove(executor);
+forceScheduleCheck();
+ }
+
+ class ScheduledQueryExecutor implements Runnable {
Review comment:
Similar to `ScheduledQueryPoller` , can we leave a description comment for
`ScheduledQueryExecutor` too (even if evident)? Thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 395699)
Time Spent: 50m (was: 40m)
> Support multiple executors for scheduled queries
>
>
> Key: HIVE-22872
> URL: https://issues.apache.org/jira/browse/HIVE-22872
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch,
> HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch,
> HIVE-22872.06.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22872) Support multiple executors for scheduled queries
[ https://issues.apache.org/jira/browse/HIVE-22872?focusedWorklogId=395695&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-395695 ] ASF GitHub Bot logged work on HIVE-22872: - Author: ASF GitHub Bot Created on: 02/Mar/20 01:35 Start Date: 02/Mar/20 01:35 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #924: HIVE-22872 schq executors URL: https://github.com/apache/hive/pull/924#discussion_r386136963 ## File path: ql/src/java/org/apache/hadoop/hive/ql/scheduled/ScheduledQueryExecutionService.java ## @@ -48,18 +53,29 @@ private static ScheduledQueryExecutionService INSTANCE = null; private ScheduledQueryExecutionContext context; - private ScheduledQueryExecutor worker; private AtomicInteger forcedScheduleCheckCounter = new AtomicInteger(); + private ScheduledQueryPoller poller; + AtomicInteger usedExecutors = new AtomicInteger(0); Review comment: private? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 395695) Time Spent: 20m (was: 10m) > Support multiple executors for scheduled queries > > > Key: HIVE-22872 > URL: https://issues.apache.org/jira/browse/HIVE-22872 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22872.01.patch, HIVE-22872.02.patch, > HIVE-22872.03.patch, HIVE-22872.04.patch, HIVE-22872.05.patch, > HIVE-22872.06.patch > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22953) Update Apache Arrow and flatbuffer versions
[ https://issues.apache.org/jira/browse/HIVE-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-22953: -- > Update Apache Arrow and flatbuffer versions > --- > > Key: HIVE-22953 > URL: https://issues.apache.org/jira/browse/HIVE-22953 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > HIVE-22827 updated flatbuffer version to 1.6.0.1. Current Arrow version > consumed by Hive uses 1.2.0 (com.vlkan:flatbuffers version). > This issue is to update Arrow to at least 0.15.1 and flatbuffers to 1.11.0 > (from official flatbuffers release, same version used by Arrow). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22953) Update Apache Arrow and flatbuffer versions
[ https://issues.apache.org/jira/browse/HIVE-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22953 started by Jesus Camacho Rodriguez. -- > Update Apache Arrow and flatbuffer versions > --- > > Key: HIVE-22953 > URL: https://issues.apache.org/jira/browse/HIVE-22953 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > HIVE-22827 updated flatbuffer version to 1.6.0.1. Current Arrow version > consumed by Hive uses 1.2.0 (com.vlkan:flatbuffers version). > This issue is to update Arrow to at least 0.15.1 and flatbuffers to 1.11.0 > (from official flatbuffers release, same version used by Arrow). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048741#comment-17048741
]
Hive QA commented on HIVE-22926:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
56s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
10s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
31s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 1322
unchanged - 9 fixed = 1323 total (was 1331) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20902/dev-support/hive-personality.sh
|
| git revision | master / e529a16 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20902/yetus/diff-checkstyle-itests_hive-unit.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20902/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20902/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.14.patch,
> HIVE-22926.patch
>
>
> [https://github.com/apache/hive/pull/927]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Attachment: HIVE-22926.15.patch Status: Patch Available (was: In Progress) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.14.patch, > HIVE-22926.15.patch, HIVE-22926.patch > > > [https://github.com/apache/hive/pull/927] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[ https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-22926: --- Status: In Progress (was: Patch Available) > Schedule Repl Dump Task using Hive scheduler > > > Key: HIVE-22926 > URL: https://issues.apache.org/jira/browse/HIVE-22926 > Project: Hive > Issue Type: Bug >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch, > HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch, > HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch, > HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch, > HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.14.patch, > HIVE-22926.15.patch, HIVE-22926.patch > > > [https://github.com/apache/hive/pull/927] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048758#comment-17048758
]
Hive QA commented on HIVE-22926:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995081/HIVE-22926.14.patch
{color:green}SUCCESS:{color} +1 due to 22 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18093 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20902/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20902/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20902/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995081 - PreCommit-HIVE-Build
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.14.patch,
> HIVE-22926.15.patch, HIVE-22926.patch
>
>
> [https://github.com/apache/hive/pull/927]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22136) Turn on tez.bucket.pruning
[ https://issues.apache.org/jira/browse/HIVE-22136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22136: Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks, Vineet! > Turn on tez.bucket.pruning > --- > > Key: HIVE-22136 > URL: https://issues.apache.org/jira/browse/HIVE-22136 > Project: Hive > Issue Type: Task >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22136.1.patch, HIVE-22136.2.patch, > HIVE-22136.3.patch, HIVE-22136.4.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[ https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22786: Status: Open (was: Patch Available) > Vectorization: Agg with distinct can be optimised in HASH mode > -- > > Key: HIVE-22786 > URL: https://issues.apache.org/jira/browse/HIVE-22786 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch, > HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch, > HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-21971:
Status: Open (was: Patch Available)
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
> select dummy();
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> at
> org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was
> removed in 2.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[ https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22786: Attachment: HIVE-22786.9.patch > Vectorization: Agg with distinct can be optimised in HASH mode > -- > > Key: HIVE-22786 > URL: https://issues.apache.org/jira/browse/HIVE-22786 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch, > HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch, > HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch, HIVE-22786.9.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[ https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-22786: Status: Patch Available (was: Open) > Vectorization: Agg with distinct can be optimised in HASH mode > -- > > Key: HIVE-22786 > URL: https://issues.apache.org/jira/browse/HIVE-22786 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch, > HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch, > HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch, HIVE-22786.9.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-21971:
Status: Patch Available (was: Open)
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch, HIVE-21971.4.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
> select dummy();
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> at
> org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was
> removed in 2.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-21971:
Attachment: HIVE-21971.4.patch
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch, HIVE-21971.4.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
> select dummy();
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> at
> org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was
> removed in 2.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048769#comment-17048769
]
Hive QA commented on HIVE-22926:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
59s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 1322
unchanged - 9 fixed = 1322 total (was 1331) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 52s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20903/dev-support/hive-personality.sh
|
| git revision | master / e529a16 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20903/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20903/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.14.patch,
> HIVE-22926.15.patch, HIVE-22926.patch
>
>
> [https://github.com/apache/hive/pull/927]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22903) Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause
[
https://issues.apache.org/jira/browse/HIVE-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048774#comment-17048774
]
Shubham Chaurasia commented on HIVE-22903:
--
[~rameshkumar] Tests are all green now. Can you please have a look at the patch
?
> Vectorized row_number() resets the row number after one batch in case of
> constant expression in partition clause
>
>
> Key: HIVE-22903
> URL: https://issues.apache.org/jira/browse/HIVE-22903
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22903.01.patch, HIVE-22903.02.patch,
> HIVE-22903.03.patch, HIVE-22903.04.patch, HIVE-22903.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorized row number implementation resets the row number when constant
> expression is passed in partition clause.
> Repro Query
> {code}
> select row_number() over(partition by 1) r1, t from over10k_n8;
> Or
> select row_number() over() r1, t from over10k_n8;
> {code}
> where table over10k_n8 contains more than 1024 records.
> This happens because currently in VectorPTFOperator, we reset evaluators if
> only partition clause is there.
> {code:java}
> // If we are only processing a PARTITION BY, reset our evaluators.
> if (!isPartitionOrderBy) {
> groupBatches.resetEvaluators();
> }
> {code}
> To resolve, we should also check if the entire partition clause is a constant
> expression, if it is so then we should not do
> {{groupBatches.resetEvaluators()}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22840) Race condition in formatters of TimestampColumnVector and DateColumnVector
[
https://issues.apache.org/jira/browse/HIVE-22840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048776#comment-17048776
]
Shubham Chaurasia commented on HIVE-22840:
--
[~jcamachorodriguez] Tests are all green now.
> Race condition in formatters of TimestampColumnVector and DateColumnVector
> ---
>
> Key: HIVE-22840
> URL: https://issues.apache.org/jira/browse/HIVE-22840
> Project: Hive
> Issue Type: Bug
> Components: storage-api
>Reporter: László Bodor
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22840.03.patch, HIVE-22840.04.patch,
> HIVE-22840.05.patch, HIVE-22840.1.patch, HIVE-22840.2.patch, HIVE-22840.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HIVE-22405 added support for proleptic calendar. It uses java's
> SimpleDateFormat/Calendar APIs which are not thread-safe and cause race in
> some scenarios.
> As a result of those race conditions, we see some exceptions like
> {code:java}
> 1) java.lang.NumberFormatException: For input string: ""
> OR
> java.lang.NumberFormatException: For input string: ".821582E.821582E44"
> OR
> 2) Caused by: java.lang.ArrayIndexOutOfBoundsException: -5325980
> at
> sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:453)
> at
> java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2397)
> {code}
> This issue is to address those thread-safety issues/race conditions.
> cc [~jcamachorodriguez] [~abstractdog] [~omalley]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22840) Race condition in formatters of TimestampColumnVector and DateColumnVector
[
https://issues.apache.org/jira/browse/HIVE-22840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048777#comment-17048777
]
Shubham Chaurasia commented on HIVE-22840:
--
[~jcamachorodriguez] Oh you already committed. Thanks!
> Race condition in formatters of TimestampColumnVector and DateColumnVector
> ---
>
> Key: HIVE-22840
> URL: https://issues.apache.org/jira/browse/HIVE-22840
> Project: Hive
> Issue Type: Bug
> Components: storage-api
>Reporter: László Bodor
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22840.03.patch, HIVE-22840.04.patch,
> HIVE-22840.05.patch, HIVE-22840.1.patch, HIVE-22840.2.patch, HIVE-22840.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HIVE-22405 added support for proleptic calendar. It uses java's
> SimpleDateFormat/Calendar APIs which are not thread-safe and cause race in
> some scenarios.
> As a result of those race conditions, we see some exceptions like
> {code:java}
> 1) java.lang.NumberFormatException: For input string: ""
> OR
> java.lang.NumberFormatException: For input string: ".821582E.821582E44"
> OR
> 2) Caused by: java.lang.ArrayIndexOutOfBoundsException: -5325980
> at
> sun.util.calendar.BaseCalendar.getCalendarDateFromFixedDate(BaseCalendar.java:453)
> at
> java.util.GregorianCalendar.computeFields(GregorianCalendar.java:2397)
> {code}
> This issue is to address those thread-safety issues/race conditions.
> cc [~jcamachorodriguez] [~abstractdog] [~omalley]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22853) Beeline should use HS2 server defaults for fetchSize
[ https://issues.apache.org/jira/browse/HIVE-22853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22853: - Status: Open (was: Patch Available) > Beeline should use HS2 server defaults for fetchSize > > > Key: HIVE-22853 > URL: https://issues.apache.org/jira/browse/HIVE-22853 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22853.2.patch, HIVE-22853.3.patch, > HIVE-22853.4.patch, HIVE-22853.patch > > > Currently beeline uses a hard coded default of 1000 rows for fetchSize. This > default value is different from what the server has set. While the beeline > user can reset the value via set command, its cumbersome to change the > workloads. > Rather it should default to the server-side value and set should be used to > override within the session. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22853) Beeline should use HS2 server defaults for fetchSize
[ https://issues.apache.org/jira/browse/HIVE-22853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22853: - Attachment: HIVE-22853.5.patch > Beeline should use HS2 server defaults for fetchSize > > > Key: HIVE-22853 > URL: https://issues.apache.org/jira/browse/HIVE-22853 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22853.2.patch, HIVE-22853.3.patch, > HIVE-22853.4.patch, HIVE-22853.5.patch, HIVE-22853.patch > > > Currently beeline uses a hard coded default of 1000 rows for fetchSize. This > default value is different from what the server has set. While the beeline > user can reset the value via set command, its cumbersome to change the > workloads. > Rather it should default to the server-side value and set should be used to > override within the session. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22853) Beeline should use HS2 server defaults for fetchSize
[ https://issues.apache.org/jira/browse/HIVE-22853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048778#comment-17048778 ] Naveen Gangam commented on HIVE-22853: -- Looks like there are changes on master. I am attaching a rebased version of the patch. > Beeline should use HS2 server defaults for fetchSize > > > Key: HIVE-22853 > URL: https://issues.apache.org/jira/browse/HIVE-22853 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22853.2.patch, HIVE-22853.3.patch, > HIVE-22853.4.patch, HIVE-22853.5.patch, HIVE-22853.patch > > > Currently beeline uses a hard coded default of 1000 rows for fetchSize. This > default value is different from what the server has set. While the beeline > user can reset the value via set command, its cumbersome to change the > workloads. > Rather it should default to the server-side value and set should be used to > override within the session. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22853) Beeline should use HS2 server defaults for fetchSize
[ https://issues.apache.org/jira/browse/HIVE-22853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22853: - Status: Patch Available (was: Open) Rebased patch to master. > Beeline should use HS2 server defaults for fetchSize > > > Key: HIVE-22853 > URL: https://issues.apache.org/jira/browse/HIVE-22853 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22853.2.patch, HIVE-22853.3.patch, > HIVE-22853.4.patch, HIVE-22853.5.patch, HIVE-22853.patch > > > Currently beeline uses a hard coded default of 1000 rows for fetchSize. This > default value is different from what the server has set. While the beeline > user can reset the value via set command, its cumbersome to change the > workloads. > Rather it should default to the server-side value and set should be used to > override within the session. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22953) Update Apache Arrow and flatbuffer versions
[ https://issues.apache.org/jira/browse/HIVE-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22953: --- Description: HIVE-22827 updated flatbuffer version to 1.6.0.1. Current Arrow version consumed by Hive uses 1.2.0 (com.vlkan:flatbuffers version). This issue is to update Arrow and flatbuffers (from official flatbuffers release, same version used by Arrow). was: HIVE-22827 updated flatbuffer version to 1.6.0.1. Current Arrow version consumed by Hive uses 1.2.0 (com.vlkan:flatbuffers version). This issue is to update Arrow to at least 0.15.1 and flatbuffers to 1.11.0 (from official flatbuffers release, same version used by Arrow). > Update Apache Arrow and flatbuffer versions > --- > > Key: HIVE-22953 > URL: https://issues.apache.org/jira/browse/HIVE-22953 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > HIVE-22827 updated flatbuffer version to 1.6.0.1. Current Arrow version > consumed by Hive uses 1.2.0 (com.vlkan:flatbuffers version). > This issue is to update Arrow and flatbuffers (from official flatbuffers > release, same version used by Arrow). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22926) Schedule Repl Dump Task using Hive scheduler
[
https://issues.apache.org/jira/browse/HIVE-22926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048793#comment-17048793
]
Hive QA commented on HIVE-22926:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995083/HIVE-22926.15.patch
{color:green}SUCCESS:{color} +1 due to 22 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 18093 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20903/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20903/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20903/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995083 - PreCommit-HIVE-Build
> Schedule Repl Dump Task using Hive scheduler
>
>
> Key: HIVE-22926
> URL: https://issues.apache.org/jira/browse/HIVE-22926
> Project: Hive
> Issue Type: Bug
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-22926.01.patch, HIVE-22926.02.patch,
> HIVE-22926.03.patch, HIVE-22926.04.patch, HIVE-22926.05.patch,
> HIVE-22926.06.patch, HIVE-22926.07.patch, HIVE-22926.08.patch,
> HIVE-22926.09.patch, HIVE-22926.10.patch, HIVE-22926.11.patch,
> HIVE-22926.12.patch, HIVE-22926.13.patch, HIVE-22926.14.patch,
> HIVE-22926.15.patch, HIVE-22926.patch
>
>
> [https://github.com/apache/hive/pull/927]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22953) Update Apache Arrow and flatbuffer versions
[ https://issues.apache.org/jira/browse/HIVE-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22953: --- Status: Patch Available (was: In Progress) > Update Apache Arrow and flatbuffer versions > --- > > Key: HIVE-22953 > URL: https://issues.apache.org/jira/browse/HIVE-22953 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > HIVE-22827 updated flatbuffer version to 1.6.0.1. Current Arrow version > consumed by Hive uses 1.2.0 (com.vlkan:flatbuffers version). > This issue is to update Arrow and flatbuffers (from official flatbuffers > release, same version used by Arrow). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22953) Update Apache Arrow and flatbuffer versions
[ https://issues.apache.org/jira/browse/HIVE-22953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22953: --- Attachment: HIVE-22953.patch > Update Apache Arrow and flatbuffer versions > --- > > Key: HIVE-22953 > URL: https://issues.apache.org/jira/browse/HIVE-22953 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22953.patch > > > HIVE-22827 updated flatbuffer version to 1.6.0.1. Current Arrow version > consumed by Hive uses 1.2.0 (com.vlkan:flatbuffers version). > This issue is to update Arrow and flatbuffers (from official flatbuffers > release, same version used by Arrow). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22929) Performance: quoted identifier parsing uses throwaway Regex via String.replaceAll()
[
https://issues.apache.org/jira/browse/HIVE-22929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22929:
--
Attachment: HIVE-22929.2.patch
> Performance: quoted identifier parsing uses throwaway Regex via
> String.replaceAll()
> ---
>
> Key: HIVE-22929
> URL: https://issues.apache.org/jira/browse/HIVE-22929
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal Vijayaraghavan
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22929.1.patch, HIVE-22929.2.patch,
> HIVE-22929.2.patch, String.replaceAll.png
>
>
> !String.replaceAll.png!
> https://github.com/apache/hive/blob/master/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g#L530
> {code}
> '`' ( '``' | ~('`') )* '`' { setText(getText().substring(1,
> getText().length() -1 ).replaceAll("``", "`")); }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22929) Performance: quoted identifier parsing uses throwaway Regex via String.replaceAll()
[
https://issues.apache.org/jira/browse/HIVE-22929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22929:
--
Status: Patch Available (was: Open)
> Performance: quoted identifier parsing uses throwaway Regex via
> String.replaceAll()
> ---
>
> Key: HIVE-22929
> URL: https://issues.apache.org/jira/browse/HIVE-22929
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal Vijayaraghavan
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22929.1.patch, HIVE-22929.2.patch,
> HIVE-22929.2.patch, String.replaceAll.png
>
>
> !String.replaceAll.png!
> https://github.com/apache/hive/blob/master/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g#L530
> {code}
> '`' ( '``' | ~('`') )* '`' { setText(getText().substring(1,
> getText().length() -1 ).replaceAll("``", "`")); }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22929) Performance: quoted identifier parsing uses throwaway Regex via String.replaceAll()
[
https://issues.apache.org/jira/browse/HIVE-22929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22929:
--
Status: Open (was: Patch Available)
> Performance: quoted identifier parsing uses throwaway Regex via
> String.replaceAll()
> ---
>
> Key: HIVE-22929
> URL: https://issues.apache.org/jira/browse/HIVE-22929
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal Vijayaraghavan
>Assignee: Krisztian Kasa
>Priority: Major
> Attachments: HIVE-22929.1.patch, HIVE-22929.2.patch,
> HIVE-22929.2.patch, String.replaceAll.png
>
>
> !String.replaceAll.png!
> https://github.com/apache/hive/blob/master/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g#L530
> {code}
> '`' ( '``' | ~('`') )* '`' { setText(getText().substring(1,
> getText().length() -1 ).replaceAll("``", "`")); }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[
https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048798#comment-17048798
]
Hive QA commented on HIVE-22786:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 1 new + 404 unchanged - 0
fixed = 405 total (was 404) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20904/dev-support/hive-personality.sh
|
| git revision | master / de21a98 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20904/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20904/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20904/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Agg with distinct can be optimised in HASH mode
> --
>
> Key: HIVE-22786
> URL: https://issues.apache.org/jira/browse/HIVE-22786
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch,
> HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch,
> HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch, HIVE-22786.9.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22903) Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause
[
https://issues.apache.org/jira/browse/HIVE-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048800#comment-17048800
]
Ashutosh Chauhan commented on HIVE-22903:
-
+1
> Vectorized row_number() resets the row number after one batch in case of
> constant expression in partition clause
>
>
> Key: HIVE-22903
> URL: https://issues.apache.org/jira/browse/HIVE-22903
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22903.01.patch, HIVE-22903.02.patch,
> HIVE-22903.03.patch, HIVE-22903.04.patch, HIVE-22903.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorized row number implementation resets the row number when constant
> expression is passed in partition clause.
> Repro Query
> {code}
> select row_number() over(partition by 1) r1, t from over10k_n8;
> Or
> select row_number() over() r1, t from over10k_n8;
> {code}
> where table over10k_n8 contains more than 1024 records.
> This happens because currently in VectorPTFOperator, we reset evaluators if
> only partition clause is there.
> {code:java}
> // If we are only processing a PARTITION BY, reset our evaluators.
> if (!isPartitionOrderBy) {
> groupBatches.resetEvaluators();
> }
> {code}
> To resolve, we should also check if the entire partition clause is a constant
> expression, if it is so then we should not do
> {{groupBatches.resetEvaluators()}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22903) Vectorized row_number() resets the row number after one batch in case of constant expression in partition clause
[
https://issues.apache.org/jira/browse/HIVE-22903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-22903:
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Shubham!
> Vectorized row_number() resets the row number after one batch in case of
> constant expression in partition clause
>
>
> Key: HIVE-22903
> URL: https://issues.apache.org/jira/browse/HIVE-22903
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 4.0.0
>Reporter: Shubham Chaurasia
>Assignee: Shubham Chaurasia
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22903.01.patch, HIVE-22903.02.patch,
> HIVE-22903.03.patch, HIVE-22903.04.patch, HIVE-22903.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Vectorized row number implementation resets the row number when constant
> expression is passed in partition clause.
> Repro Query
> {code}
> select row_number() over(partition by 1) r1, t from over10k_n8;
> Or
> select row_number() over() r1, t from over10k_n8;
> {code}
> where table over10k_n8 contains more than 1024 records.
> This happens because currently in VectorPTFOperator, we reset evaluators if
> only partition clause is there.
> {code:java}
> // If we are only processing a PARTITION BY, reset our evaluators.
> if (!isPartitionOrderBy) {
> groupBatches.resetEvaluators();
> }
> {code}
> To resolve, we should also check if the entire partition clause is a constant
> expression, if it is so then we should not do
> {{groupBatches.resetEvaluators()}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-12604) Cannot drop a table after creating an index and then renaming to a different database
[
https://issues.apache.org/jira/browse/HIVE-12604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048803#comment-17048803
]
Roja B commented on HIVE-12604:
---
I had faced a similar issue:
My scenario was:
1) create table test.temp_A 2) check if index exists and delete 3) create index
4) check if table test1.A exists and drop 5) Rename table test.temp_A to test1.A
Everysingle time the drop table(step 4) was failing by throwing the error the
schema.Temp_A_index does not exist.
Solution :
I have renamed the table A to B. And have dropped test. B and test1.B and Which
enabled to drop test1. A.
Also changed the sequence of my script to
1) create table temp_A in test2) check if table A exists and drop 3) Rename
test.temp_A to test1.A 4) check if index exists and drop 5) create index
Always check your hdfs location and know look for the the relevant tables and
schema names that got created with index.
Use command : hdfs dfs -rm -r /path to hdfs location/relevant filename.
This resolved my Problem.
> Cannot drop a table after creating an index and then renaming to a different
> database
> -
>
> Key: HIVE-12604
> URL: https://issues.apache.org/jira/browse/HIVE-12604
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
> Environment: Hortonworks Sandbox VM with HDP 2.3
>Reporter: Toby Allsopp
>Priority: Minor
>
> {noformat}
> Connected to: Apache Hive (version 1.2.1.2.3.0.0-2557)
> Driver: Hive JDBC (version 1.2.1.2.3.0.0-2557)
> 0: jdbc:hive2://localhost:1> create database db1;
> No rows affected (0.997 seconds)
> 0: jdbc:hive2://localhost:1> create database db2;
> No rows affected (0.968 seconds)
> 0: jdbc:hive2://localhost:1> create table db1.test (col1 int);
> No rows affected (1.758 seconds)
> 0: jdbc:hive2://localhost:1> create index idx1 on table db1.test(col1) as
> 'compact' with deferred rebuild;
> No rows affected (0.287 seconds)
> 0: jdbc:hive2://localhost:1> alter index idx1 on db1.test rebuild;
> INFO : Tez session hasn't been created yet. Opening session
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1449025977131_0007)
> INFO : Map 1: -/- Reducer 2: 0/1
> INFO : Map 1: -/- Reducer 2: 0(+1)/1
> INFO : Map 1: -/- Reducer 2: 0/1
> INFO : Map 1: -/- Reducer 2: 1/1
> INFO : Loading data to table db1.db1__test_idx1__ from
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/db1.db/b1__test_idx1__/.hive-staging_hive_2015-12-04_02-02-47_278_3621654884902999047-10/-ext-1
> INFO : Table db1.db1__test_idx1__ stats: [numFiles=1, numRows=0,
> totalSize=0, rawDataSize=0]
> No rows affected (7.792 seconds)
> 0: jdbc:hive2://localhost:1> alter table db1.test rename to db2.test;
> No rows affected (0.261 seconds)
> 0: jdbc:hive2://localhost:1> drop table db2.test;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLask.
> MetaException(message:db2.db1__test_idx1__ table not found)
> (state=08S01,code=1)
> {noformat}
> Basically it looks like the rename to a different database left the index
> table in the old database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-12604) Cannot drop a table after creating an index and then renaming to a different database
[
https://issues.apache.org/jira/browse/HIVE-12604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048803#comment-17048803
]
Roja B edited comment on HIVE-12604 at 3/2/20 6:05 AM:
---
I had faced a similar issue:
My scenario was:
1) create table test.temp_A 2) check if index exists and delete 3) create
index 4) check if table test1.A exists and drop if exists 5) Rename table
'test.temp_A' to 'test1.A'
Everysingle time the drop table(step 4) was failing by throwing the error the
schema.Temp_A_index does not exist.
Solution :
Rename the table test1.A to test1.B and drop test1.B . It enabled to drop
test1.A.
Also i have changed the sequence of my script execution to:
1) create table test.temp_A 2) check if table A exists and drop if exists 3)
Rename test.temp_A to test1.A 4) check if index exists and drop 5) create index
.
Always check your hdfs location and look for the relevant tables ,schemas(even
the renamed ones) that got created with the index and drop them using the below
command:
Use command : hdfs dfs -rm -r /path to hdfs location/relevant filename.
This resolved my Problem.
was (Author: roja):
I had faced a similar issue:
My scenario was:
1) create table test.temp_A 2) check if index exists and delete 3) create index
4) check if table test1.A exists and drop 5) Rename table test.temp_A to test1.A
Everysingle time the drop table(step 4) was failing by throwing the error the
schema.Temp_A_index does not exist.
Solution :
I have renamed the table A to B. And have dropped test. B and test1.B and Which
enabled to drop test1. A.
Also changed the sequence of my script to
1) create table temp_A in test2) check if table A exists and drop 3) Rename
test.temp_A to test1.A 4) check if index exists and drop 5) create index
Always check your hdfs location and know look for the the relevant tables and
schema names that got created with index.
Use command : hdfs dfs -rm -r /path to hdfs location/relevant filename.
This resolved my Problem.
> Cannot drop a table after creating an index and then renaming to a different
> database
> -
>
> Key: HIVE-12604
> URL: https://issues.apache.org/jira/browse/HIVE-12604
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.2.1
> Environment: Hortonworks Sandbox VM with HDP 2.3
>Reporter: Toby Allsopp
>Priority: Minor
>
> {noformat}
> Connected to: Apache Hive (version 1.2.1.2.3.0.0-2557)
> Driver: Hive JDBC (version 1.2.1.2.3.0.0-2557)
> 0: jdbc:hive2://localhost:1> create database db1;
> No rows affected (0.997 seconds)
> 0: jdbc:hive2://localhost:1> create database db2;
> No rows affected (0.968 seconds)
> 0: jdbc:hive2://localhost:1> create table db1.test (col1 int);
> No rows affected (1.758 seconds)
> 0: jdbc:hive2://localhost:1> create index idx1 on table db1.test(col1) as
> 'compact' with deferred rebuild;
> No rows affected (0.287 seconds)
> 0: jdbc:hive2://localhost:1> alter index idx1 on db1.test rebuild;
> INFO : Tez session hasn't been created yet. Opening session
> INFO :
> INFO : Status: Running (Executing on YARN cluster with App id
> application_1449025977131_0007)
> INFO : Map 1: -/- Reducer 2: 0/1
> INFO : Map 1: -/- Reducer 2: 0(+1)/1
> INFO : Map 1: -/- Reducer 2: 0/1
> INFO : Map 1: -/- Reducer 2: 1/1
> INFO : Loading data to table db1.db1__test_idx1__ from
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/db1.db/b1__test_idx1__/.hive-staging_hive_2015-12-04_02-02-47_278_3621654884902999047-10/-ext-1
> INFO : Table db1.db1__test_idx1__ stats: [numFiles=1, numRows=0,
> totalSize=0, rawDataSize=0]
> No rows affected (7.792 seconds)
> 0: jdbc:hive2://localhost:1> alter table db1.test rename to db2.test;
> No rows affected (0.261 seconds)
> 0: jdbc:hive2://localhost:1> drop table db2.test;
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLask.
> MetaException(message:db2.db1__test_idx1__ table not found)
> (state=08S01,code=1)
> {noformat}
> Basically it looks like the rename to a different database left the index
> table in the old database.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22786) Vectorization: Agg with distinct can be optimised in HASH mode
[
https://issues.apache.org/jira/browse/HIVE-22786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048820#comment-17048820
]
Hive QA commented on HIVE-22786:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995085/HIVE-22786.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18094 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeader.testHouseKeepingThreadExistence
(batchId=250)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20904/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20904/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20904/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995085 - PreCommit-HIVE-Build
> Vectorization: Agg with distinct can be optimised in HASH mode
> --
>
> Key: HIVE-22786
> URL: https://issues.apache.org/jira/browse/HIVE-22786
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-22786.1.patch, HIVE-22786.2.patch,
> HIVE-22786.3.patch, HIVE-22786.4.wip.patch, HIVE-22786.5.patch,
> HIVE-22786.6.patch, HIVE-22786.7.patch, HIVE-22786.8.patch, HIVE-22786.9.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22948) QueryCache: Treat query cache locations as temporary storage
[
https://issues.apache.org/jira/browse/HIVE-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal Vijayaraghavan updated HIVE-22948:
Attachment: HIVE-22948.1.patch
> QueryCache: Treat query cache locations as temporary storage
>
>
> Key: HIVE-22948
> URL: https://issues.apache.org/jira/browse/HIVE-22948
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Attachments: HIVE-22948.1.patch, HIVE-22948.1.patch
>
>
> The WriteEntity with a query cache query is considered for user authorization
> without having direct access for users.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/command/CommandAuthorizerV2.java#L111
> {code}
> if (privObject instanceof WriteEntity &&
> ((WriteEntity)privObject).isTempURI()) {
> // do not authorize temporary uris
> continue;
> }
> {code}
> is not satisfied by the queries qualifying for the query cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22948) QueryCache: Treat query cache locations as temporary storage
[
https://issues.apache.org/jira/browse/HIVE-22948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048826#comment-17048826
]
Gopal Vijayaraghavan commented on HIVE-22948:
-
Missed a file in the patch.
> QueryCache: Treat query cache locations as temporary storage
>
>
> Key: HIVE-22948
> URL: https://issues.apache.org/jira/browse/HIVE-22948
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 4.0.0, 3.1.2
>Reporter: Gopal Vijayaraghavan
>Assignee: Gopal Vijayaraghavan
>Priority: Major
> Attachments: HIVE-22948.1.patch, HIVE-22948.1.patch
>
>
> The WriteEntity with a query cache query is considered for user authorization
> without having direct access for users.
> https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/security/authorization/command/CommandAuthorizerV2.java#L111
> {code}
> if (privObject instanceof WriteEntity &&
> ((WriteEntity)privObject).isTempURI()) {
> // do not authorize temporary uris
> continue;
> }
> {code}
> is not satisfied by the queries qualifying for the query cache.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048835#comment-17048835
]
Hive QA commented on HIVE-21971:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 1531 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 72 unchanged - 0 fixed
= 73 total (was 72) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-20905/dev-support/hive-personality.sh
|
| git revision | master / f6bae78 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20905/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20905/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-20905/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch, HIVE-21971.4.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'fil
[jira] [Commented] (HIVE-21971) HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with temporary functions + GenericUDF
[
https://issues.apache.org/jira/browse/HIVE-21971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048850#comment-17048850
]
Hive QA commented on HIVE-21971:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995086/HIVE-21971.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 18097 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schq_ingest]
(batchId=185)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/20905/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20905/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20905/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12995086 - PreCommit-HIVE-Build
> HS2 leaks classloader due to `ReflectionUtils::CONSTRUCTOR_CACHE` with
> temporary functions + GenericUDF
> ---
>
> Key: HIVE-21971
> URL: https://issues.apache.org/jira/browse/HIVE-21971
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.3.4
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Critical
> Attachments: HIVE-21971.1.patch, HIVE-21971.2.patch,
> HIVE-21971.3.patch, HIVE-21971.4.patch
>
>
> https://issues.apache.org/jira/browse/HIVE-10329 helped in moving away from
> hadoop's ReflectionUtils constructor cache issue
> (https://issues.apache.org/jira/browse/HADOOP-10513).
> However, there are corner cases where hadoop's {{ReflectionUtils}} is in use
> and this causes gradual build up of memory in HS2.
> I have observed this in Hive 2.3. But the codepath in master for this has not
> changed much.
> Easiest way to repro would be to add a temp function which extends
> {{GenericUDF}}. In {{FunctionRegistry::cloneGenericUDF,}} this would
> end up using {{org.apache.hadoop.util.ReflectionUtils.newInstance}} which in
> turn lands up in COSNTRUCTOR_CACHE of ReflectionUtils.
> {noformat}
> CREATE TEMPORARY FUNCTION dummy AS 'com.hive.test.DummyGenericUDF' USING JAR
> 'file:///home/test/udf/dummy.jar';
> select dummy();
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:107)
> at
> org.apache.hadoop.hive.ql.exec.FunctionRegistry.cloneGenericUDF(FunctionRegistry.java:1353)
> at
> org.apache.hadoop.hive.ql.exec.FunctionInfo.getGenericUDF(FunctionInfo.java:122)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:983)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1359)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> {noformat}
> Note: Reflection based invocation of hadoop's {{ReflectionUtils::clear}} was
> removed in 2.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
