[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214604&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214604
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:18
Start Date: 18/Mar/19 07:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266316079
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -136,7 +143,11 @@ private void doCopyRetry(FileSystem sourceFs,
List s
}
// looks like some network outrage, reset the file system object and
retry.
- FileSystem.closeAllForUGI(Utils.getUGI());
+ if (proxyUser == null) {
+FileSystem.closeAllForUGI(Utils.getUGI());
+ } else {
+FileSystem.closeAllForUGI(proxyUser);
+ }
sourceFs = pathList.get(0).getFileSystem(hiveConf);
destinationFs = destination.getFileSystem(hiveConf);
Review comment:
no need ..dlm or any other system calling repl load should retry in case of
file system error
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214604)
Time Spent: 1h 10m (was: 1h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214603&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214603
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:18
Start Date: 18/Mar/19 07:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266315969
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -72,33 +72,40 @@ public CopyUtils(String distCpDoAsUser, HiveConf hiveConf)
{
public void copyAndVerify(FileSystem destinationFs, Path destRoot,
List srcFiles) throws
IOException, LoginException, HiveFatalException {
Map>> map =
fsToFileMap(srcFiles, destRoot);
-for (Map.Entry>>
entry : map.entrySet()) {
- FileSystem sourceFs = entry.getKey();
- Map> destMap = entry.getValue();
- for (Map.Entry> destMapEntry :
destMap.entrySet()) {
-Path destination = destMapEntry.getKey();
-List fileInfoList =
destMapEntry.getValue();
-boolean useRegularCopy = regularCopy(destinationFs, sourceFs,
fileInfoList);
-
-if (!destinationFs.exists(destination)
-&& !FileUtils.mkdir(destinationFs, destination, hiveConf)) {
- LOG.error("Failed to create destination directory: " + destination);
- throw new IOException("Destination directory creation failed");
-}
+UserGroupInformation proxyUser = getProxyUser();
+try {
+ for (Map.Entry>>
entry : map.entrySet()) {
+FileSystem sourceFs = entry.getKey();
Review comment:
The hive retry logic is for very limited local functionality ..for all other
places ..in case of file system error ..dlm will retry
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214603)
Time Spent: 1h (was: 50m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214608&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214608
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:19
Start Date: 18/Mar/19 07:19
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266316160
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
Review comment:
change to info
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214608)
Time Spent: 1h 40m (was: 1.5h)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214607&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214607
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:19
Start Date: 18/Mar/19 07:19
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266316131
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ExternalTableCopyTaskBuilder.java
##
@@ -99,53 +99,65 @@ private boolean createAndSetPathOwner(Path destPath, Path
sourcePath) throws IOE
return createdDir;
}
-private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
String distCpDoAsUser)
-throws IOException {
- if (distCpDoAsUser == null) {
+private boolean setTargetPathOwner(Path targetPath, Path sourcePath,
UserGroupInformation proxyUser)
+throws IOException, InterruptedException {
+ if (proxyUser == null) {
return createAndSetPathOwner(targetPath, sourcePath);
}
- UserGroupInformation proxyUser = UserGroupInformation.createProxyUser(
- distCpDoAsUser, UserGroupInformation.getLoginUser());
- try {
-Path finalTargetPath = targetPath;
-Path finalSourcePath = sourcePath;
-return proxyUser.doAs((PrivilegedExceptionAction) () ->
-createAndSetPathOwner(finalTargetPath, finalSourcePath));
- } catch (InterruptedException e) {
-throw new IOException(e);
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+createAndSetPathOwner(targetPath, sourcePath));
+}
+
+private boolean checkIfPathExist(Path sourcePath, UserGroupInformation
proxyUser) throws Exception {
+ if (proxyUser == null) {
+return sourcePath.getFileSystem(conf).exists(sourcePath);
}
+ return proxyUser.doAs((PrivilegedExceptionAction) () ->
+ sourcePath.getFileSystem(conf).exists(sourcePath));
}
-private int handleException(Exception e, Path sourcePath, Path targetPath,
int currentRetry) {
+private int handleException(Exception e, Path sourcePath, Path targetPath,
+int currentRetry, UserGroupInformation
proxyUser) {
try {
-if (!sourcePath.getFileSystem(conf).exists(sourcePath)) {
- LOG.warn("Source path missing " + sourcePath, e);
+LOG.warn("Checking if source path " + sourcePath + " is missing for
exception ", e);
+if (!checkIfPathExist(sourcePath, proxyUser)) {
+ LOG.warn("Source path is missing. Ignoring exception.");
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214607)
Time Spent: 1.5h (was: 1h 20m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?focusedWorklogId=214605&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214605
]
ASF GitHub Bot logged work on HIVE-21446:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 07:18
Start Date: 18/Mar/19 07:18
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #569: HIVE-21446 :
Hive Server going OOM during hive external table replications
URL: https://github.com/apache/hive/pull/569#discussion_r266316109
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/repl/CopyUtils.java
##
@@ -238,23 +249,54 @@ private boolean isSourceFileMismatch(FileSystem
sourceFs, ReplChangeManager.File
return false;
}
+ private UserGroupInformation getProxyUser() throws LoginException,
IOException {
+if (copyAsUser == null) {
+ return null;
+}
+UserGroupInformation proxyUser = null;
+int currentRetry = 0;
+while (currentRetry <= MAX_IO_RETRY) {
+ try {
+UserGroupInformation ugi = Utils.getUGI();
+String currentUser = ugi.getShortUserName();
+if (!currentUser.equals(copyAsUser)) {
+ proxyUser = UserGroupInformation.createProxyUser(
+ copyAsUser, UserGroupInformation.getLoginUser());
+}
+return proxyUser;
+ } catch (IOException e) {
+currentRetry++;
+if (currentRetry <= MAX_IO_RETRY) {
+ LOG.warn("Unable to get UGI info", e);
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214605)
Time Spent: 1h 20m (was: 1h 10m)
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21165) ACID: pass query hint to the writers to write hive.acid.key.index
[ https://issues.apache.org/jira/browse/HIVE-21165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794812#comment-16794812 ] Abhishek Somani commented on HIVE-21165: In ACID v2, as every file has only one type of event (insert or delete), isn't hive.acid.key.index not required anymore (as also mentioned in HIVE-20580)? > ACID: pass query hint to the writers to write hive.acid.key.index > - > > Key: HIVE-21165 > URL: https://issues.apache.org/jira/browse/HIVE-21165 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > For the query based compactor from HIVE-20699, the compaction runs as a sql > query. However, this mechanism skips over writing hive.acid.key.index for > each stripe, which is used to skip over stripes that are not supposed to be > read. We need a way to pass a query hint to the writer so that it can write > this index data, when invoked from a sql query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20948) Eliminate file rename in compactor
[
https://issues.apache.org/jira/browse/HIVE-20948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794814#comment-16794814
]
Abhishek Somani commented on HIVE-20948:
Looks like a duplicate of HIVE-21164.
cc [~vgumashta]
> Eliminate file rename in compactor
> --
>
> Key: HIVE-20948
> URL: https://issues.apache.org/jira/browse/HIVE-20948
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Priority: Major
>
> Once HIVE-20823 is committed, we should investigate if it's possible to have
> compactor write directly to base_x_cZ or delta_x_y_cZ.
> For query based compaction: can we control location of temp table dir? We
> support external temp tables so this may work but we'd need to have non-acid
> insert create files with {{bucket_x}} names.
>
> For MR/Tez/LLAP based (should this be done at all?), need to figure out how
> retries of tasks will work. Just like we currently generate an MR job to
> compact, we should be able to generate a Tez job.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794815#comment-16794815
]
Hive QA commented on HIVE-21001:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962772/HIVE-21001.44.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ambiguitycheck]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_3]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_ppd_char]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcr] (batchId=65)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[rand_partitionpruner3]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_date_1]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[current_date_timestamp]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_hint]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_1]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_ppd_varchar]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_2]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin6]
(batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_null_agg]
(batchId=178)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_date_1]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_interval_2]
(batchId=177)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in]
(batchId=140)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_null_agg]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar]
(batchId=129)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query45]
(batchId=277)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query14]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query16]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query23]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query28]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query32]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query61]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query92]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query94]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query95]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query97]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query16]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query23]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query28]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query32]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query39]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query61]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query92]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query94]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query95]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query97]
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16553/testReport
Cons
[jira] [Commented] (HIVE-21001) Upgrade to calcite-1.19
[
https://issues.apache.org/jira/browse/HIVE-21001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794819#comment-16794819
]
Hive QA commented on HIVE-21001:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} accumulo-handler in master has 21 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 5 new + 282 unchanged - 29
fixed = 287 total (was 311) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
3s{color} | {color:red} root: The patch generated 5 new + 282 unchanged - 29
fixed = 287 total (was 311) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16553/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16553/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16553/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16553/yetus/whitespace-eol.txt
|
| modules | C: ql accumulo-handler hbase-handler . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16553/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade to calcite-1.19
> ---
>
> Key: HIVE-21001
> URL: https://issues.apache.org/jira/browse/HIVE-21001
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan
[jira] [Updated] (HIVE-21452) Loss of query condition when exist exists
[
https://issues.apache.org/jira/browse/HIVE-21452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zengxl updated HIVE-21452:
--
Description:
In our production environment, there are four tables to do association queries.
There are exists in the conditions, and we found that the first two conditions
were lost.
The following two conditions are missing：
{color:#f79232}t2.cust_no is null and t4.cust_level not in ('4','5'){color}
In the test environment, I prepared the data of the following four tables, and
a condition was lost in the simulation.
tables:
test_table1
cust_no,name
60001,lisa
60002,tina
60003,kylin
60004,jeny
60005,john
60006,jamse
test_table2
cust_no,acct_type
60001,1
60001,1
60001,2
60002,1
60003,2
60003,3
test_table3
cust_no
60001
60002
60003
60004
60005
60007
test_table4
cust_no,cust_level
60001,1
60002,2
60003,3
60004,4
60005,5
create table tmp.test_table1(cust_no string,name string);
create table tmp.test_table2(cust_no string,acct_type string);
create table tmp.test_table3(cust_no string);
create table tmp.test_table4(cust_no string,cust_level string);
insert into tmp.test_table1 select '60001','lisa';
insert into tmp.test_table1 select '60002','tina';
insert into tmp.test_table1 select '60003','kylin';
insert into tmp.test_table1 select '60004','jeny';
insert into tmp.test_table1 select '60005','john';
insert into tmp.test_table1 select '60006','jamse';
insert into tmp.test_table2 select '60001','1';
insert into tmp.test_table2 select '60001','1';
insert into tmp.test_table2 select '60001','2';
insert into tmp.test_table2 select '60002','1';
insert into tmp.test_table2 select '60003','2';
insert into tmp.test_table2 select '60002','3';
insert into tmp.test_table3 select '60001';
insert into tmp.test_table3 select '60002';
insert into tmp.test_table3 select '60003';
insert into tmp.test_table3 select '60004';
insert into tmp.test_table3 select '60005';
insert into tmp.test_table3 select '60007';
insert into tmp.test_table4 select '60001','1';
insert into tmp.test_table4 select '60002','2';
insert into tmp.test_table4 select '60003','3';
insert into tmp.test_table4 select '60004','4';
insert into tmp.test_table4 select '60005','5';
Here is my query SQL And shut down mapjoin:
set hive.auto.convert.join=false;
select t1.cust_no as cust_no,t2.cust_no as custNO,t1.name from tmp.test_table1
t1
left join tmp.test_table2 t2 on t1.cust_no=t2.cust_no
and t2.acct_type='1'
left join tmp.test_table4 t4 on t1.cust_no=t4.cust_no
where t2.cust_no is null and t4.cust_level not in ('4','5') and exists (select
1 from tmp.test_table3 t3 where t1.cust_no=t3.cust_no)
All I want is to include cust_no for 6003，But the result is inclusive 6004 and
6005，this wrong 。{color:#f79232}In my production environment, 6001 came
out。Loss of condition because cust_no is null。{color}
{color:#f6c342}View the execution plan, t4.cust_level not in ('4','5')
condition missing{color}
*{color:#f6c342}Explain:{color}*
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-2 depends on stages: Stage-1
Stage-0 depends on stages: Stage-2
STAGE PLANS:
Stage: Stage-1
Map Reduce
Map Operator Tree:
TableScan
alias: t1
Statistics: Num rows: 12 Data size: 2128 Basic stats: COMPLETE Column stats:
NONE
Filter Operator
predicate: cust_no is not null (type: boolean)
Statistics: Num rows: 6 Data size: 1064 Basic stats: COMPLETE Column stats:
NONE
Reduce Output Operator
key expressions: cust_no (type: string)
sort order: +
Map-reduce partition columns: cust_no (type: string)
Statistics: Num rows: 6 Data size: 1064 Basic stats: COMPLETE Column stats:
NONE
value expressions: name (type: string)
TableScan
alias: t2
Statistics: Num rows: 12 Data size: 2088 Basic stats: COMPLETE Column stats:
NONE
Filter Operator
predicate: ((acct_type = '1') and cust_no is not null) (type: boolean)
Statistics: Num rows: 3 Data size: 522 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: cust_no (type: string)
sort order: +
Map-reduce partition columns: cust_no (type: string)
Statistics: Num rows: 3 Data size: 522 Basic stats: COMPLETE Column stats: NONE
TableScan
alias: t4
Statistics: Num rows: 10 Data size: 1740 Basic stats: COMPLETE Column stats:
NONE
Filter Operator
predicate: cust_no is not null (type: boolean)
Statistics: Num rows: 5 Data size: 870 Basic stats: COMPLETE Column stats: NONE
Reduce Output Operator
key expressions: cust_no (type: string)
sort order: +
Map-reduce partition columns: cust_no (type: string)
Statistics: Num rows: 5 Data size: 870 Basic stats: COMPLETE Column stats: NONE
Reduce Operator Tree:
Join Operator
condition map:
Left Outer Join0 to 1
Left Outer Join0 to 2
keys:
0 cust_no (type: string)
1 cust_no (type: string)
2 cust_no (type: string)
outputColumnNames: _col0, _col1, _col5
Statistics: Num rows: 13 Data
[jira] [Updated] (HIVE-21213) Acid table bootstrap replication needs to handle directory created by compaction with txn id
[ https://issues.apache.org/jira/browse/HIVE-21213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21213: --- Description: The current implementation of compaction uses the txn id in the directory name. This is used to isolate the queries from reading the directory until compaction has finished and to avoid the compactor marking used earlier. In case of replication, during bootstrap , directory is copied as it is with the same name from source to destination cluster. But the directory created by compaction with txn id can not be copied as the txn list at target may be different from source. The txn id which is valid at source may be an aborted txn at target. So conversion logic is required to create a new directory with valid txn at target and dump the data to the newly created directory. (was: The current implementation of compaction uses the txn id in the directory name. This is used to isolate the queries from reading the directory until compaction has finished and to avoid the compactor marking used earlier. In case of replication, the directory can not be copied as the txn list at target may be different from source. So conversion logic is required to create a new directory with valid txn at target and dump the data to the newly created directory.) > Acid table bootstrap replication needs to handle directory created by > compaction with txn id > > > Key: HIVE-21213 > URL: https://issues.apache.org/jira/browse/HIVE-21213 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2, repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > > The current implementation of compaction uses the txn id in the directory > name. This is used to isolate the queries from reading the directory until > compaction has finished and to avoid the compactor marking used earlier. In > case of replication, during bootstrap , directory is copied as it is with the > same name from source to destination cluster. But the directory created by > compaction with txn id can not be copied as the txn list at target may be > different from source. The txn id which is valid at source may be an aborted > txn at target. So conversion logic is required to create a new directory with > valid txn at target and dump the data to the newly created directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21454) Tez default configs get overwritten by MR default configs
[ https://issues.apache.org/jira/browse/HIVE-21454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-21454: - Affects Version/s: 3.0.0 3.1.0 3.1.1 > Tez default configs get overwritten by MR default configs > - > > Key: HIVE-21454 > URL: https://issues.apache.org/jira/browse/HIVE-21454 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: Syed Shameerur Rahman >Priority: Major > > Due to changes done in HIVE-17781 Tez default configs such as > tez.counters.max which has a default value of 1200 gets overwritten by > mapreduce.job.counters.max which has a default value of 120. > So one of the obvious solution is to initialize TezConfig with its default > value before the method in HIVE-17781 is called and hence overwritten won't > happen. > cc [~mithun] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[ https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21462: -- Labels: pull-request-available (was: ) > Upgrading SQL server backed metastore when changing data type of a column > with constraints > -- > > Key: HIVE-21462 > URL: https://issues.apache.org/jira/browse/HIVE-21462 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Original Estimate: 24h > Remaining Estimate: 24h > > SQL server does not allow changing data type of a column which has a > constraint or an index on it. The constraint or the index needs to be dropped > before changing the data type and needs to be recreated after that. Metastore > upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[ https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794870#comment-16794870 ] Ashutosh Bapat commented on HIVE-21462: --- Since dropping an unnamed constraint requires some SQL code in SQL server, it's better to name all the constraints explicitly to keep the scripts simple. So, there are additional changes in the scripts to name the new constraints created. > Upgrading SQL server backed metastore when changing data type of a column > with constraints > -- > > Key: HIVE-21462 > URL: https://issues.apache.org/jira/browse/HIVE-21462 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > SQL server does not allow changing data type of a column which has a > constraint or an index on it. The constraint or the index needs to be dropped > before changing the data type and needs to be recreated after that. Metastore > upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[ https://issues.apache.org/jira/browse/HIVE-21462?focusedWorklogId=214659&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214659 ] ASF GitHub Bot logged work on HIVE-21462: - Author: ASF GitHub Bot Created on: 18/Mar/19 09:40 Start Date: 18/Mar/19 09:40 Worklog Time Spent: 10m Work Description: ashutosh-bapat commented on pull request #573: HIVE-21462 : Upgrading SQL server backed metastore when changing datatype of a column with constraints URL: https://github.com/apache/hive/pull/573 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 214659) Time Spent: 10m Remaining Estimate: 23h 50m (was: 24h) > Upgrading SQL server backed metastore when changing data type of a column > with constraints > -- > > Key: HIVE-21462 > URL: https://issues.apache.org/jira/browse/HIVE-21462 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > SQL server does not allow changing data type of a column which has a > constraint or an index on it. The constraint or the index needs to be dropped > before changing the data type and needs to be recreated after that. Metastore > upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21454) Tez default configs get overwritten by MR default configs
[ https://issues.apache.org/jira/browse/HIVE-21454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syed Shameerur Rahman updated HIVE-21454: - Description: Due to changes done in HIVE-17781 Tez default configs such as tez.counters.max which has a default value of 1200 gets overwritten by mapreduce.job.counters.max which has a default value of 120. This happens because tez.counters.max gets initialized in the query run time. So one of the obvious solution is to initialize TezConfig with its default value before the method in HIVE-17781 is called and hence overwritten won't happen. cc [~mithun] was: Due to changes done in HIVE-17781 Tez default configs such as tez.counters.max which has a default value of 1200 gets overwritten by mapreduce.job.counters.max which has a default value of 120. So one of the obvious solution is to initialize TezConfig with its default value before the method in HIVE-17781 is called and hence overwritten won't happen. cc [~mithun] > Tez default configs get overwritten by MR default configs > - > > Key: HIVE-21454 > URL: https://issues.apache.org/jira/browse/HIVE-21454 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: Syed Shameerur Rahman >Priority: Major > > Due to changes done in HIVE-17781 Tez default configs such as > tez.counters.max which has a default value of 1200 gets overwritten by > mapreduce.job.counters.max which has a default value of 120. This happens > because tez.counters.max gets initialized in the query run time. > So one of the obvious solution is to initialize TezConfig with its default > value before the method in HIVE-17781 is called and hence overwritten won't > happen. > cc [~mithun] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20580) OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
[
https://issues.apache.org/jira/browse/HIVE-20580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794916#comment-16794916
]
Peter Vary commented on HIVE-20580:
---
[~ekoifman]: Could you please check my approach? I would like to have the
confirmation that the original intention was for the {{isOriginal}} to return
true for Non-ACID ORC files, and false for ACID files. The {{isOriginal(Reader
file)}} reflects this, but the {{isOriginal(Footer footer)}} returns the
opposite results without the patch. Is my assumption correct and both methods
should behave in the same way and return false for the ACID files?
Thanks,
Peter
> OrcInputFormat.isOriginal() should not rely on hive.acid.key.index
> --
>
> Key: HIVE-20580
> URL: https://issues.apache.org/jira/browse/HIVE-20580
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.1.0
>Reporter: Eugene Koifman
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-20580.2.patch, HIVE-20580.3.patch,
> HIVE-20580.4.patch, HIVE-20580.5.patch, HIVE-20580.6.patch, HIVE-20580.patch
>
>
> {{org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isOriginal()}} is checking
> for presence of {{hive.acid.key.index}} in the footer. This is only created
> when the file is written by {{OrcRecordUpdater}}. It should instead check
> for presence of Acid metadata columns so that a file can be produced by
> something other than {{OrcRecordUpater}}.
> Also, {{hive.acid.key.index}} counts number of different type of events which
> is not really useful for Acid V2 (as of Hive 3) since each file only has 1
> type of event.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21446: --- Status: Patch Available (was: Open) > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21446: --- Status: Open (was: Patch Available) > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21446: --- Attachment: HIVE-21446.02.patch > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794941#comment-16794941
]
Peter Vary commented on HIVE-21455:
---
[~belugabehr], [~zchovan]: Would you be interested in this?
I would be happy to review.
Thanks,
Peter
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Priority: Minor
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-17404) Orc split generation cache does not handle files without file tail
[
https://issues.apache.org/jira/browse/HIVE-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794953#comment-16794953
]
Aditya Shah commented on HIVE-17404:
Unrelated failures.
> Orc split generation cache does not handle files without file tail
> --
>
> Key: HIVE-17404
> URL: https://issues.apache.org/jira/browse/HIVE-17404
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Prasanth Jayachandran
>Assignee: Aditya Shah
>Priority: Critical
> Attachments: HIVE-17404.patch
>
>
> Some old files do not have Orc FileTail. If file tail does not exist, split
> generation should fallback to old way of storing footers.
> This can result in exceptions like below
> {code}
> ORC split generation failed with exception: Malformed ORC file. Invalid
> postscript length 9
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1735)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1822)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:450)
> at
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:569)
> at
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.orc.FileFormatException: Malformed ORC file. Invalid
> postscript length 9
> at org.apache.orc.impl.ReaderImpl.ensureOrcFooter(ReaderImpl.java:297)
> at org.apache.orc.impl.ReaderImpl.extractFileTail(ReaderImpl.java:470)
> at
> org.apache.hadoop.hive.ql.io.orc.LocalCache.getAndValidate(LocalCache.java:103)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.getSplits(OrcInputFormat.java:804)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.runGetSplitsSync(OrcInputFormat.java:922)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$ETLSplitStrategy.generateSplitWork(OrcInputFormat.java:891)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.scheduleSplits(OrcInputFormat.java:1763)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1707)
> ... 15 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794963#comment-16794963
]
Hive QA commented on HIVE-21446:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
9s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} shims/common: The patch generated 0 new + 94
unchanged - 1 fixed = 94 total (was 95) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} The patch 0.23 passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 19 unchanged - 2 fixed
= 21 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16554/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16554/yetus/diff-checkstyle-ql.txt
|
| modules | C: shims/common shims/0.23 common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16554/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16794976#comment-16794976
]
Hive QA commented on HIVE-21446:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962798/HIVE-21446.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16554/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16554/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16554/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962798 - PreCommit-HIVE-Build
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?focusedWorklogId=214744&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214744
]
ASF GitHub Bot logged work on HIVE-21430:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 12:55
Start Date: 18/Mar/19 12:55
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #572: HIVE-21430 :
INSERT into a dynamically partitioned table with autogather stats disabled
throws a MetaException
URL: https://github.com/apache/hive/pull/572#discussion_r266400761
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2626,6 +2637,7 @@ private void constructOneLBLocationMap(FileStatus fSta,
}
boolean isTxnTable = AcidUtils.isTransactionalTable(tbl);
+AcidUtils.TableSnapshot tableSnapshot = getTableSnapshot(tbl, writeId);
Review comment:
can be done only for txn table
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214744)
Time Spent: 0.5h (was: 20m)
Remaining Estimate: 47.5h (was: 47h 40m)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 0.5h
> Remaining Estimate: 47.5h
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?focusedWorklogId=214743&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214743
]
ASF GitHub Bot logged work on HIVE-21430:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 12:55
Start Date: 18/Mar/19 12:55
Worklog Time Spent: 10m
Work Description: maheshk114 commented on pull request #572: HIVE-21430 :
INSERT into a dynamically partitioned table with autogather stats disabled
throws a MetaException
URL: https://github.com/apache/hive/pull/572#discussion_r266423668
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2377,9 +2382,15 @@ private void setStatsPropAndAlterPartitions(boolean
resetStatistics, Table tbl,
LOG.debug(sb.toString());
}
+String validWriteIdList = null;
+Long writeId = 0L;
Review comment:
use primitive long type
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214743)
Time Spent: 20m (was: 10m)
Remaining Estimate: 47h 40m (was: 47h 50m)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 20m
> Remaining Estimate: 47h 40m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21446: --- Attachment: (was: HIVE-21446.02.patch) > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21446: --- Status: Patch Available (was: Open) > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21446: --- Status: Open (was: Patch Available) > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21446) Hive Server going OOM during hive external table replications
[ https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-21446: --- Attachment: HIVE-21446.02.patch > Hive Server going OOM during hive external table replications > - > > Key: HIVE-21446 > URL: https://issues.apache.org/jira/browse/HIVE-21446 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > > The file system objects opened using proxy users are not closed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Attachment: (was: HIVE-21264.3.patch)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Attachment: (was: HIVE-21264.3.patch)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21371) Make NonSyncByteArrayOutputStream Overflow Conscious
[
https://issues.apache.org/jira/browse/HIVE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21371:
--
Attachment: (was: HIVE-21371.2.patch)
> Make NonSyncByteArrayOutputStream Overflow Conscious
> -
>
> Key: HIVE-21371
> URL: https://issues.apache.org/jira/browse/HIVE-21371
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21371.1.patch, HIVE-21371.2.patch
>
>
> {code:java|title=NonSyncByteArrayOutputStream}
> private int enLargeBuffer(int increment) {
> int temp = count + increment;
> int newLen = temp;
> if (temp > buf.length) {
> if ((buf.length << 1) > temp) {
> newLen = buf.length << 1;
> }
> byte newbuf[] = new byte[newLen];
> System.arraycopy(buf, 0, newbuf, 0, count);
> buf = newbuf;
> }
> return newLen;
> }
> {code}
> This will fail if the array is 2GB or larger because it will double the size
> every time without consideration for the 4GB limit on arrays.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Attachment: (was: HIVE-21264.3.patch)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21371) Make NonSyncByteArrayOutputStream Overflow Conscious
[
https://issues.apache.org/jira/browse/HIVE-21371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21371:
--
Attachment: (was: HIVE-21371.2.patch)
> Make NonSyncByteArrayOutputStream Overflow Conscious
> -
>
> Key: HIVE-21371
> URL: https://issues.apache.org/jira/browse/HIVE-21371
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-21371.1.patch, HIVE-21371.2.patch
>
>
> {code:java|title=NonSyncByteArrayOutputStream}
> private int enLargeBuffer(int increment) {
> int temp = count + increment;
> int newLen = temp;
> if (temp > buf.length) {
> if ((buf.length << 1) > temp) {
> newLen = buf.length << 1;
> }
> byte newbuf[] = new byte[newLen];
> System.arraycopy(buf, 0, newbuf, 0, count);
> buf = newbuf;
> }
> return newLen;
> }
> {code}
> This will fail if the array is 2GB or larger because it will double the size
> every time without consideration for the 4GB limit on arrays.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21073) Remove Extra String Object
[
https://issues.apache.org/jira/browse/HIVE-21073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21073:
--
Attachment: (was: HIVE-21073.1.patch)
> Remove Extra String Object
> --
>
> Key: HIVE-21073
> URL: https://issues.apache.org/jira/browse/HIVE-21073
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.1.1
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Fix For: 4.0.0
>
> Attachments: HIVE-21073.1.patch
>
>
> {code}
> public static String generatePath(Path baseURI, String filename) {
> String path = new String(baseURI + Path.SEPARATOR + filename);
> return path;
> }
> public static String generateFileName(Byte tag, String bigBucketFileName) {
> String fileName = new String("MapJoin-" + tag + "-" + bigBucketFileName +
> suffix);
> return fileName;
> }
> {code}
> It's a bit odd to be performing string concatenation and then wrapping the
> results in a new string. This is creating superfluous String objects.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21264) Improvements Around CharTypeInfo
[
https://issues.apache.org/jira/browse/HIVE-21264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21264:
--
Attachment: (was: HIVE-21264.3.patch)
> Improvements Around CharTypeInfo
>
>
> Key: HIVE-21264
> URL: https://issues.apache.org/jira/browse/HIVE-21264
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21264.1.patch, HIVE-21264.2.patch,
> HIVE-21264.3.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The {{CharTypeInfo}} stores the type name of the data type (char/varchar) and
> the length (1-255). {{CharTypeInfo}} objects are often getting cached once
> they are created.
> The {{hashcode()}} and {{equals()}} of its sub-classes varchar and char are
> inconsistent.
> * Make hashcode and equals consistent (and fast)
> * Simplify the {{getQualifiedName}} implementation and reduce the scope to
> protected
> * Other related nits
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795028#comment-16795028
]
Hive QA commented on HIVE-21446:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
21s{color} | {color:blue} shims/common in master has 6 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} shims/common: The patch generated 0 new + 94
unchanged - 1 fixed = 94 total (was 95) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} The patch 0.23 passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 19 unchanged - 2 fixed
= 21 total (was 21) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16555/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16555/yetus/diff-checkstyle-ql.txt
|
| modules | C: shims/common shims/0.23 common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16555/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21
[jira] [Updated] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[ https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21462: -- Attachment: HIVE-21462.01.patch Status: Patch Available (was: Open) > Upgrading SQL server backed metastore when changing data type of a column > with constraints > -- > > Key: HIVE-21462 > URL: https://issues.apache.org/jira/browse/HIVE-21462 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21462.01.patch > > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > SQL server does not allow changing data type of a column which has a > constraint or an index on it. The constraint or the index needs to be dropped > before changing the data type and needs to be recreated after that. Metastore > upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21372) Use Apache Commons IO To Read Stream To String
[ https://issues.apache.org/jira/browse/HIVE-21372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795040#comment-16795040 ] Ashutosh Chauhan commented on HIVE-21372: - +1 > Use Apache Commons IO To Read Stream To String > -- > > Key: HIVE-21372 > URL: https://issues.apache.org/jira/browse/HIVE-21372 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-21372.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21446) Hive Server going OOM during hive external table replications
[
https://issues.apache.org/jira/browse/HIVE-21446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795056#comment-16795056
]
Hive QA commented on HIVE-21446:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962808/HIVE-21446.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_groupby_reduce]
(batchId=61)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16555/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16555/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16555/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962808 - PreCommit-HIVE-Build
> Hive Server going OOM during hive external table replications
> -
>
> Key: HIVE-21446
> URL: https://issues.apache.org/jira/browse/HIVE-21446
> Project: Hive
> Issue Type: Bug
> Components: repl
>Affects Versions: 4.0.0
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21446.01.patch, HIVE-21446.02.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The file system objects opened using proxy users are not closed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21406: Attachment: (was: HIVE-21406.01.patch) > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21406: Attachment: HIVE-21406.01.patch > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[
https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795086#comment-16795086
]
Hive QA commented on HIVE-21462:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 21 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16556/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16556/yetus/whitespace-tabs.txt
|
| modules | C: standalone-metastore/metastore-server . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16556/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrading SQL server backed metastore when changing data type of a column
> with constraints
> --
>
> Key: HIVE-21462
> URL: https://issues.apache.org/jira/browse/HIVE-21462
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21462.01.patch
>
> Original Estimate: 24h
> Time Spent: 10m
> Remaining Estimate: 23h 50m
>
> SQL server does not allow changing data type of a column which has a
> constraint or an index on it. The constraint or the index needs to be dropped
> before changing the data type and needs to be recreated after that. Metastore
> upgrade scripts aren't doing this and thus upgrade fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[ https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-21465: --- Assignee: Laszlo Bodor > Introduce a base abstract VectorExpression to eliminate further code > duplications > - > > Key: HIVE-21465 > URL: https://issues.apache.org/jira/browse/HIVE-21465 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[
https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795110#comment-16795110
]
Hive QA commented on HIVE-21462:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962818/HIVE-21462.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16556/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16556/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16556/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962818 - PreCommit-HIVE-Build
> Upgrading SQL server backed metastore when changing data type of a column
> with constraints
> --
>
> Key: HIVE-21462
> URL: https://issues.apache.org/jira/browse/HIVE-21462
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21462.01.patch
>
> Original Estimate: 24h
> Time Spent: 10m
> Remaining Estimate: 23h 50m
>
> SQL server does not allow changing data type of a column which has a
> constraint or an index on it. The constraint or the index needs to be dropped
> before changing the data type and needs to be recreated after that. Metastore
> upgrade scripts aren't doing this and thus upgrade fails.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Description:
>From time to time I see a similar/same logic to be repeated, which always
>consists of some stuff like:
1. some inner loops (even the comments are the same)
{code}
grep -iRH "Set isNull before call in case it changes it mind" | grep
"ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
{code}
2. call to an actual "business logic" function ==> 1 line which could be
abstracted... and it has been abstracted in hive codebase in n different cases.
However, an abstract BaseVectorExpression would not force anybody to extend it
instead of VectorExpression and not copy the code again, but it could be a good
start and an example.
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szurap reassigned HIVE-21455:
Assignee: Miklos Szurap
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795123#comment-16795123
]
Miklos Szurap commented on HIVE-21455:
--
Hi [~pvary], I've created a patch, could you review it? Thx.
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szurap updated HIVE-21455:
-
Attachment: HIVE-21455.patch
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21455) Too verbose logging in AvroGenericRecordReader
[
https://issues.apache.org/jira/browse/HIVE-21455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795125#comment-16795125
]
Miklos Szurap commented on HIVE-21455:
--
To add to that, the fix is trivial so I've not included tests.
I suggest to:
* keep the "Found the Avro schema in the job" messages there (so we can see
how many files/splits are processed)
* in case DEBUG level logging is enabled, we are logging the schema too - if
that's needed for troubleshooting.
> Too verbose logging in AvroGenericRecordReader
> --
>
> Key: HIVE-21455
> URL: https://issues.apache.org/jira/browse/HIVE-21455
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Miklos Szurap
>Assignee: Miklos Szurap
>Priority: Minor
> Attachments: HIVE-21455.patch
>
>
> {{AvroGenericRecordReader}} logs the Avro schema for each datafile. It is too
> verbose, likely we don't need to log that on INFO level.
> For example a table:
> {noformat}
> create table avro_tbl (c1 string, c2 int, c3 float) stored as avro;
> {noformat}
> and querying it with a select star - with 3 datafiles HiveServer2 logs the
> following:
> {noformat}
> 2019-03-15 09:18:35,999 INFO org.apache.hadoop.mapred.FileInputFormat:
> [HiveServer2-Handler-Pool: Thread-64]: Total input paths to process : 3
> 2019-03-15 09:18:35,999 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,004 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> 2019-03-15 09:18:36,010 INFO
> org.apache.hadoop.hive.ql.io.avro.AvroGenericRecordReader:
> [HiveServer2-Handler-Pool: Thread-64]: Found the avro schema in the job:
> {"type":"record","name":"avro_tbl","namespace":"test","fields":[{"name":"c1","type":["null","string"],"default":null},{"name":"c2","type":["null","int"],"default":null},{"name":"c3","type":["null","float"],"default":null}]}
> {noformat}
> This has a huge performance and storage penalty on a table with big schema
> and thousands of datafiles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?focusedWorklogId=214855&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214855
]
ASF GitHub Bot logged work on HIVE-21430:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 16:18
Start Date: 18/Mar/19 16:18
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #572:
HIVE-21430 : INSERT into a dynamically partitioned table with autogather stats
disabled throws a MetaException
URL: https://github.com/apache/hive/pull/572#discussion_r266524070
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2626,6 +2637,7 @@ private void constructOneLBLocationMap(FileStatus fSta,
}
boolean isTxnTable = AcidUtils.isTransactionalTable(tbl);
+AcidUtils.TableSnapshot tableSnapshot = getTableSnapshot(tbl, writeId);
Review comment:
Thanks for catching this. Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214855)
Time Spent: 40m (was: 0.5h)
Remaining Estimate: 47h 20m (was: 47.5h)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 40m
> Remaining Estimate: 47h 20m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work logged] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?focusedWorklogId=214856&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-214856
]
ASF GitHub Bot logged work on HIVE-21430:
-
Author: ASF GitHub Bot
Created on: 18/Mar/19 16:18
Start Date: 18/Mar/19 16:18
Worklog Time Spent: 10m
Work Description: ashutosh-bapat commented on pull request #572:
HIVE-21430 : INSERT into a dynamically partitioned table with autogather stats
disabled throws a MetaException
URL: https://github.com/apache/hive/pull/572#discussion_r266524121
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -2377,9 +2382,15 @@ private void setStatsPropAndAlterPartitions(boolean
resetStatistics, Table tbl,
LOG.debug(sb.toString());
}
+String validWriteIdList = null;
+Long writeId = 0L;
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 214856)
Time Spent: 50m (was: 40m)
Remaining Estimate: 47h 10m (was: 47h 20m)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21430:
--
Status: In Progress (was: Patch Available)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Bapat updated HIVE-21430:
--
Attachment: HIVE-21430.02.patch
Status: Patch Available (was: In Progress)
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795181#comment-16795181
]
Ashutosh Bapat commented on HIVE-21430:
---
[~maheshk114], I have updated PR addressing your comments in a separate commit.
Please review.
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Attachment: HIVE-21465.01.patch
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Attachment: (was: HIVE-21465.01.patch)
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Status: Patch Available (was: Open)
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Description:
In vectorized expressions, from time to time, I see a similar/same logic to be
repeated, which always consists of some stuff like:
1. some inner loops (even the comments are the same)
{code}
grep -iRH "Set isNull before call in case it changes it mind" | grep
"ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
{code}
2. call to an actual "business logic" function ==> 1 line which could be
abstracted... and it has been abstracted in hive codebase in n different cases.
However, an abstract BaseVectorExpression would not force anybody to extend it
instead of VectorExpression and not copy the code again, but it could be a good
start and an example.
was:
>From time to time I see a similar/same logic to be repeated, which always
>consists of some stuff like:
1. some inner loops (even the comments are the same)
{code}
grep -iRH "Set isNull before call in case it changes it mind" | grep
"ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
{code}
2. call to an actual "business logic" function ==> 1 line which could be
abstracted... and it has been abstracted in hive codebase in n different cases.
However, an abstract BaseVectorExpression would not force anybody to extend it
instead of VectorExpression and not copy the code again, but it could be a good
start and an example.
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795201#comment-16795201
]
Laszlo Bodor commented on HIVE-21465:
-
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laszlo Bodor updated HIVE-21465:
Attachment: HIVE-21465.01.patch
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> From time to time I see a similar/same logic to be repeated, which always
> consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21406) Add .factorypath files to .gitignore
[
https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795205#comment-16795205
]
Hive QA commented on HIVE-21406:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962823/HIVE-21406.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=86)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16557/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16557/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16557/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962823 - PreCommit-HIVE-Build
> Add .factorypath files to .gitignore
>
>
> Key: HIVE-21406
> URL: https://issues.apache.org/jira/browse/HIVE-21406
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10
> PM.png
>
>
> .factorypath files are generated by eclipse and should be ignored
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795201#comment-16795201
]
Laszlo Bodor edited comment on HIVE-21465 at 3/18/19 4:56 PM:
--
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time + a some func refactors, which could
show how this base class could prevent code repetition
beforeLoop is an example for an optionally extendable method (which should be
optimized by JIT in case it's a noop I guess)
was (Author: abstractdog):
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time
beforeLoop is an example for an optionally extendable method (which should be
optimized by JIT in case it's a noop I guess)
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21462) Upgrading SQL server backed metastore when changing data type of a column with constraints
[ https://issues.apache.org/jira/browse/HIVE-21462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795214#comment-16795214 ] Daniel Dai commented on HIVE-21462: --- +1 > Upgrading SQL server backed metastore when changing data type of a column > with constraints > -- > > Key: HIVE-21462 > URL: https://issues.apache.org/jira/browse/HIVE-21462 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21462.01.patch > > Original Estimate: 24h > Time Spent: 10m > Remaining Estimate: 23h 50m > > SQL server does not allow changing data type of a column which has a > constraint or an index on it. The constraint or the index needs to be dropped > before changing the data type and needs to be recreated after that. Metastore > upgrade scripts aren't doing this and thus upgrade fails. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795201#comment-16795201
]
Laszlo Bodor edited comment on HIVE-21465 at 3/18/19 4:51 PM:
--
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time
beforeLoop is an example for an optionally extendable method (which should be
optimized by JIT in case it's a noop I guess)
was (Author: abstractdog):
uploaded an initial version into review board
https://reviews.apache.org/r/70231/
this introduces generic BaseVectorExpression, which can ensure correct input /
output columnvector types at compile time
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21406: Attachment: HIVE-21406.01.patch > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21406) Add .factorypath files to .gitignore
[ https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21406: Attachment: (was: HIVE-21406.01.patch) > Add .factorypath files to .gitignore > > > Key: HIVE-21406 > URL: https://issues.apache.org/jira/browse/HIVE-21406 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10 > PM.png > > > .factorypath files are generated by eclipse and should be ignored -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795236#comment-16795236
]
Hive QA commented on HIVE-21430:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
18s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16558/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16558/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21466:
--
Attachment: HIVE-21466.1.patch
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-21466:
--
Status: Patch Available (was: Open)
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-21466:
-
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21430) INSERT into a dynamically partitioned table with hive.stats.autogather = false throws a MetaException
[
https://issues.apache.org/jira/browse/HIVE-21430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795264#comment-16795264
]
Hive QA commented on HIVE-21430:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962843/HIVE-21430.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15834 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16558/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16558/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16558/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962843 - PreCommit-HIVE-Build
> INSERT into a dynamically partitioned table with hive.stats.autogather =
> false throws a MetaException
> -
>
> Key: HIVE-21430
> URL: https://issues.apache.org/jira/browse/HIVE-21430
> Project: Hive
> Issue Type: Bug
>Reporter: Ashutosh Bapat
>Assignee: Ashutosh Bapat
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21430.01.patch, HIVE-21430.02.patch,
> metaexception_repro.patch,
> org.apache.hadoop.hive.ql.stats.TestStatsUpdaterThread-output.txt
>
> Original Estimate: 48h
> Time Spent: 50m
> Remaining Estimate: 47h 10m
>
> When the test TestStatsUpdaterThread#testTxnDynamicPartitions added in the
> attached patch is run it throws exception (full logs attached.)
> org.apache.hadoop.hive.metastore.api.MetaException: Cannot change stats state
> for a transactional table default.simple_stats without providing the
> transactional write state for verification (new write ID 5, valid write IDs
> null; current state \{"BASIC_STATS":"true","COLUMN_STATS":{"s":"true"}}; new
> state null
> at
> org.apache.hadoop.hive.metastore.ObjectStore.alterPartitionNoTxn(ObjectStore.java:4328)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795303#comment-16795303
]
Hive QA commented on HIVE-21465:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 3 new + 9 unchanged - 23 fixed
= 12 total (was 32) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
17s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16559/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16559/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16559/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16559/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Introduce a base abstract VectorExpression to eliminate further code
> duplications
> -
>
> Key: HIVE-21465
> URL: https://issues.apache.org/jira/browse/HIVE-21465
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-21465.01.patch
>
>
> In vectorized expressions, from time to time, I see a similar/same logic to
> be repeated, which always consists of some stuff like:
> 1. some inner loops (even the comments are the same)
> {code}
> grep -iRH "Set isNull before call in case it changes it mind" | grep
> "ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/"
> {code}
> 2. call to an actual "business logic" function ==> 1 line which could be
> abstracted... and it has been abstracted in hive codebase in n different
> cases.
> However, an abstract BaseVectorExpression would not force anybody to extend
> it instead of VectorExpression and not copy the code again, but it could be a
> good start and an example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-13517) Hive logs in Spark Executor and Driver should show thread-id.
[ https://issues.apache.org/jira/browse/HIVE-13517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-13517: --- Assignee: (was: Bharathkrishna Guruvayoor Murali) > Hive logs in Spark Executor and Driver should show thread-id. > - > > Key: HIVE-13517 > URL: https://issues.apache.org/jira/browse/HIVE-13517 > Project: Hive > Issue Type: Sub-task > Components: Spark >Affects Versions: 1.2.1, 2.0.0 >Reporter: Szehon Ho >Priority: Major > Attachments: HIVE-13517.1.patch, HIVE-13517.2.patch, > executor-driver-log.PNG > > > In Spark, there might be more than one task running in one executor. > Similarly, there may be more than one thread running in Driver. > This makes debugging through the logs a nightmare. It would be great if there > could be thread-ids in the logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Labels: newbie (was: ) > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Priority: Minor > Labels: newbie > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Priority: Minor (was: Major) > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor reassigned HIVE-21467: --- Assignee: Laszlo Bodor > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warning in ide, which could be annoying, and it > can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Attachment: HIVE-21467.01.patch > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warning in ide, which could be annoying, and it > can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Description: These imports trigger lots of warnings in ide, which could be annoying, and it can be replaced easily to org.junit.Assert, the signature and behavior are the same, so the tests should pass. was: These imports trigger lots of warning in ide, which could be annoying, and it can be replaced easily to org.junit.Assert, the signature and behavior are the same, so the tests should pass. > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warnings in ide, which could be annoying, and > it can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Description: These imports trigger lots of warning in ide, which could be annoying, and it can be replaced easily to org.junit.Assert, the signature and behavior are the same, so the tests should pass. > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warning in ide, which could be annoying, and it > can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[ https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-21467: Status: Patch Available (was: Open) > Remove deprecated junit.framework.Assert imports > > > Key: HIVE-21467 > URL: https://issues.apache.org/jira/browse/HIVE-21467 > Project: Hive > Issue Type: Bug >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Minor > Labels: newbie > Attachments: HIVE-21467.01.patch > > > These imports trigger lots of warnings in ide, which could be annoying, and > it can be replaced easily to org.junit.Assert, the signature and behavior are > the same, so the tests should pass. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21465) Introduce a base abstract VectorExpression to eliminate further code duplications
[
https://issues.apache.org/jira/browse/HIVE-21465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795327#comment-16795327
]
Hive QA commented on HIVE-21465:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962847/HIVE-21465.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 125 failed/errored test(s), 15807 tests
executed
*Failed tests:*
{noformat}
TestCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=24)
[authorization_cli_stdconfigauth.q,vector_fullouter_mapjoin_1_fast.q,timestamp_comparison.q,parquet_vectorization_10.q,vector_left_outer_join.q,vectorization_parquet_ppd_decimal.q,stats10.q,vector_join_part_col_char.q,nonblock_op_deduplicate.q,udf_testlength2.q,column_pruner_multiple_children.q,udf_stddev.q,select_dummy_source.q,alter_varchar2.q,tez_multi_union.q,orc_diff_part_cols.q,partition_boolean.q,default_file_format.q,udf_to_utc_timestamp.q,windowing_ntile.q,autoColumnStats_1.q,alter_rename_partition.q,semijoin.q,authorization_create_func1.q,smb_mapjoin_3.q,udf_notop.q,input_testxpath3.q,pointlookup5.q,vector_null_map.q,array_map_access_nonconstant.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ansi_sql_arithmetic]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cast_tinyint_to_double]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_5] (batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_udf] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_const_type]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[outer_reference_windowed]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=18)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_13]
(batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_17]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_3]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_div0]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_not]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[timestamptz_2]
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[update_all_types]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_case_when_1]
(batchId=92)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_1]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_5]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_aggregate]
(batchId=19)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_cast]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_mapjoin]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_math_funcs]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_round]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_udf2]
(batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_10]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_13]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_17]
(batchId=95)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_3]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_not]
(batchId=46)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case]
(batchId=63)
org.apache.hadoop.hive.cli.TestCompareCliDriver.testCliDriver[llap_0]
(batchId=271)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[convert_decimal64_to_decimal]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_1]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_decimal64_reader]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_rebuild_dummy]
(batchId=166)
org.apache.hadoop.hive.cli.TestMi
[jira] [Commented] (HIVE-21406) Add .factorypath files to .gitignore
[
https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795341#comment-16795341
]
Hive QA commented on HIVE-21406:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
52s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16560/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16560/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add .factorypath files to .gitignore
>
>
> Key: HIVE-21406
> URL: https://issues.apache.org/jira/browse/HIVE-21406
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10
> PM.png
>
>
> .factorypath files are generated by eclipse and should be ignored
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21406) Add .factorypath files to .gitignore
[
https://issues.apache.org/jira/browse/HIVE-21406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795373#comment-16795373
]
Hive QA commented on HIVE-21406:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962850/HIVE-21406.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16560/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16560/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16560/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962850 - PreCommit-HIVE-Build
> Add .factorypath files to .gitignore
>
>
> Key: HIVE-21406
> URL: https://issues.apache.org/jira/browse/HIVE-21406
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Attachments: HIVE-21406.01.patch, Screen Shot 2019-03-07 at 2.02.10
> PM.png
>
>
> .factorypath files are generated by eclipse and should be ignored
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Attachment: HIVE-21422.1.patch > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795387#comment-16795387 ] Oliver Draese commented on HIVE-21422: -- Added new counter output also to LLAP server's iomem web endpoint > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795389#comment-16795389
]
Hive QA commented on HIVE-21466:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
36s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16561/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16561/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting
[jira] [Issue Comment Deleted] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21422: - Comment: was deleted (was: Added pull request for this patch: https://github.com/apache/hive/pull/563/files) > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795399#comment-16795399 ] Oliver Draese commented on HIVE-21422: -- Added pull request: https://github.com/apache/hive/pull/574 > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[ https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795410#comment-16795410 ] Gopal V commented on HIVE-21422: LGTM - +1 tests pending > Add metrics to LRFU cache policy > > > Key: HIVE-21422 > URL: https://issues.apache.org/jira/browse/HIVE-21422 > Project: Hive > Issue Type: Improvement > Components: llap >Affects Versions: 4.0.0 >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Major > Labels: llap > Fix For: 4.0.0 > > Attachments: HIVE-21422.1.patch, HIVE-21422.patch > > > The LRFU cache policy for the LLAP data cache doesn't provide enough insight > to figure out, what is cached and why something might get evicted. This > ticket is used to add Hadoop metrics 2 information (accessible via JMX) to > the LRFU policy, providing following information: > * How much memory is cached for data buffers > * How much memory is cached for meta data buffers > * How large is the min-heap of the cache policy > * How long is the eviction short list (linked list) > * How much memory is currently "locked" (buffers with positive reference > count) and therefore in use by a query > These new counters are found in the MX bean, following this path: > Hadoop/LlapDaemon/LowLevelLrfuCachePolicy- > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21466) Increase Default Size of SPLIT_MAXSIZE
[
https://issues.apache.org/jira/browse/HIVE-21466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795423#comment-16795423
]
Hive QA commented on HIVE-21466:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962854/HIVE-21466.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query28]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query8]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[mv_query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query28]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query44]
(batchId=275)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query8]
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16561/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16561/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16561/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962854 - PreCommit-HIVE-Build
> Increase Default Size of SPLIT_MAXSIZE
> --
>
> Key: HIVE-21466
> URL: https://issues.apache.org/jira/browse/HIVE-21466
> Project: Hive
> Issue Type: Improvement
> Components: Configuration
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-21466.1.patch
>
>
> {code:java}
> MAPREDMAXSPLITSIZE(FileInputFormat.SPLIT_MAXSIZE, 25600L, "", true),
> {code}
> [https://github.com/apache/hive/blob/8d4300a02691777fc96f33861ed27e64fed72f2c/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java#L682]
> This field specifies a maximum size for each MR (maybe other?) splits.
> This number should be a multiple of the HDFS Block size. The way that this
> maximum is implemented, is that each block is added to the split, and if the
> split grows to be larger than the maximum allowed, the split is submitted to
> the cluster and a new split is opened.
> So, imagine the following scenario:
> * HDFS block size of 16 bytes
> * Maximum size of 40 bytes
> This will produce a split with 3 blocks. (2x16) = 32; another block will be
> inserted, (3x16) = 48 bytes in the split. So, while many operators would
> assume a split of 2 blocks, the actual is 3 blocks. Setting the maximum split
> size to a multiple of the HDFS block size will make this behavior less
> confusing.
> The current setting is ~256MB and when this was introduced, the default HDFS
> block size was 64MB. That is a factor of 4x. However, now HDFS block sizes
> are 128MB by default, so I propose setting this to 4x128MB. The larger
> splits (fewer tasks) should give a nice performance boost for modern hardware.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[
https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795467#comment-16795467
]
Hive QA commented on HIVE-21467:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12962867/HIVE-21467.01.patch
{color:green}SUCCESS:{color} +1 due to 96 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 15833 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udaf_invalid_place]
(batchId=99)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testRoundToDecimalPlaces
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorACos
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorASin
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorATan
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorAbs
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorCos
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorDegrees
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorExp
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorLn
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorLog10
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorLog2
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorRadians
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorRound
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorSign
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorSin
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorSqrt
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorMathFunctions.testVectorTan
(batchId=314)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testCastDoubleToTimestamp
(batchId=313)
org.apache.hadoop.hive.ql.exec.vector.expressions.TestVectorTypeCasts.testVectorCastLongToDouble
(batchId=313)
org.apache.hive.hcatalog.mapreduce.TestHCatPartitioned.testHCatPartitionedTable[1]
(batchId=209)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/16562/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/16562/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-16562/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12962867 - PreCommit-HIVE-Build
> Remove deprecated junit.framework.Assert imports
>
>
> Key: HIVE-21467
> URL: https://issues.apache.org/jira/browse/HIVE-21467
> Project: Hive
> Issue Type: Bug
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Minor
> Labels: newbie
> Attachments: HIVE-21467.01.patch
>
>
> These imports trigger lots of warnings in ide, which could be annoying, and
> it can be replaced easily to org.junit.Assert, the signature and behavior are
> the same, so the tests should pass.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-21467) Remove deprecated junit.framework.Assert imports
[
https://issues.apache.org/jira/browse/HIVE-21467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795475#comment-16795475
]
Hive QA commented on HIVE-21467:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
7s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} serde in master has 197 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
21s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
43s{color} | {color:blue} service in master has 48 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} beeline in master has 44 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} hcatalog/core in master has 28 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} hcatalog/hcatalog-pig-adapter in master has 2 extant
Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} hcatalog/webhcat/svr in master has 96 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} hcatalog/streaming in master has 11 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
51s{color} | {color:blue} itests/util in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
27s{color} | {color:blue} testutils/ptest2 in master has 24 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 11s{color}
| {color:red} storage-api generated 2 new + 0 unchanged - 2 fixed = 2 total
(was 2) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} patch/storage-api cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
36s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
42s{color} | {color:red} patc
[jira] [Assigned] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-21468: -- > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21468 started by Jesus Camacho Rodriguez. -- > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21468: --- Attachment: HIVE-21468.patch > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21468.patch > > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21422) Add metrics to LRFU cache policy
[
https://issues.apache.org/jira/browse/HIVE-21422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795484#comment-16795484
]
Hive QA commented on HIVE-21422:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 79 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 21 new + 21 unchanged
- 3 fixed = 42 total (was 24) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16563/dev-support/hive-personality.sh
|
| git revision | master / 36bd89d |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16563/yetus/diff-checkstyle-llap-server.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16563/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add metrics to LRFU cache policy
>
>
> Key: HIVE-21422
> URL: https://issues.apache.org/jira/browse/HIVE-21422
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Affects Versions: 4.0.0
>Reporter: Oliver Draese
>Assignee: Oliver Draese
>Priority: Major
> Labels: llap
> Fix For: 4.0.0
>
> Attachments: HIVE-21422.1.patch, HIVE-21422.patch
>
>
> The LRFU cache policy for the LLAP data cache doesn't provide enough insight
> to figure out, what is cached and why something might get evicted. This
> ticket is used to add Hadoop metrics 2 information (accessible via JMX) to
> the LRFU policy, providing following information:
> * How much memory is cached for data buffers
> * How much memory is cached for meta data buffers
> * How large is the min-heap of the cache policy
> * How long is the eviction short list (linked list)
> * How much memory is currently "locked" (buffers with positive reference
> count) and therefore in use by a query
> These new counters are found in the MX bean, following this path:
> Hadoop/LlapDaemon/LowLevelLrfuCachePolicy-
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-21468) Case sensitivity in identifier names for JDBC storage handler
[ https://issues.apache.org/jira/browse/HIVE-21468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21468: --- Status: Patch Available (was: In Progress) > Case sensitivity in identifier names for JDBC storage handler > - > > Key: HIVE-21468 > URL: https://issues.apache.org/jira/browse/HIVE-21468 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Currently, when Calcite generates the SQL query for the JDBC storage handler, > it will ignore capitalization for the identifiers names, which can lead to > errors at execution time (though the query is properly generated). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21456) Hive Metastore HTTP Thrift
[ https://issues.apache.org/jira/browse/HIVE-21456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amit Khanna updated HIVE-21456: --- Attachment: HIVE-21456.2.patch > Hive Metastore HTTP Thrift > -- > > Key: HIVE-21456 > URL: https://issues.apache.org/jira/browse/HIVE-21456 > Project: Hive > Issue Type: Bug > Components: Metastore, Standalone Metastore >Reporter: Amit Khanna >Assignee: Amit Khanna >Priority: Major > Attachments: HIVE-21456.2.patch, HIVE-21456.patch > > > Hive Metastore currently doesn't have support for HTTP transport because of > which it is not possible to access it via Knox. Adding support for Thrift > over HTTP transport will allow the clients to access via Knox -- This message was sent by Atlassian JIRA (v7.6.3#76005)
