[jira] [Commented] (HIVE-18863) trunc() calls itself trunk() in an error message

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394078#comment-16394078
 ] 

Hive QA commented on HIVE-18863:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m  6s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9584/dev-support/hive-personality.sh
 |
| git revision | master / 41afa4e |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9584/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9584/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> trunc() calls itself trunk() in an error message
> 
>
> Key: HIVE-18863
> URL: https://issues.apache.org/jira/browse/HIVE-18863
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Reporter: Tim Armstrong
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
>  Labels: newbie
> Attachments: HIVE-18863.1.patch
>
>
> {noformat}
> > select  trunc('millennium', cast('2001-02-16 20:38:40' as timestamp))
> FAILED: SemanticException Line 0:-1 Argument type mismatch ''2001-02-16 
> 20:38:40'': trunk() only takes STRING/CHAR/VARCHAR types as second argument, 
> got TIMESTAMP
> {noformat}
> I saw this on a derivative of Hive 1.1.0 (cdh5.15.0), but the string still 
> seems to be present on master:
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L262



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394072#comment-16394072
 ] 

Hive QA commented on HIVE-18875:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913808/HIVE-18875.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 53 failed/errored test(s), 12954 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18910:
--
Attachment: HIVE-18910.3.patch

> Migrate to Murmur hash for shuffle and bucketing
> 
>
> Key: HIVE-18910
> URL: https://issues.apache.org/jira/browse/HIVE-18910
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18910.1.patch, HIVE-18910.2.patch, 
> HIVE-18910.3.patch
>
>
> Hive uses JAVA hash which is not as good as murmur for better distribution 
> and efficiency in bucketing a table.
> Migrate to murmur hash but still keep backward compatibility for existing 
> users so that they dont have to reload the existing tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18927) Hive "insert overwrite" doesn't replace the destination files if no partition in metastore for the files

[wangzhihao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

wangzhihao updated HIVE-18927:
--
Description: 
[This 
post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
 describes a way to reproduce this issue:
{noformat}
# Add some files into file system but no partition in metastore to track it.
hdfs dfs -put test.txt test/p=p1

# Insert overwrite the partition(p = p1)
DROP TABLE IF EXISTS partition_test;
CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;

# verify the text.txt is not removed.
hdfs dfs -ls test/p=p1
Found 2 items
-rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
-rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
{noformat}
The reason is that 
[Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
 will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has 
no partition for the files, the {{oldPath}} is null and thus the files get no 
chance to be cleaned. We should also clean {{destf}} in method 
[Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
 to fix the issue.

  was:
[This 
post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
 describes a way to produce this issue:
{noformat}
# Add some files into file system but no partition in metastore to track it.
hdfs dfs -put test.txt test/p=p1

# Insert overwrite the partition(p = p1)
DROP TABLE IF EXISTS partition_test;
CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;

# verify the text.txt is not removed.
hdfs dfs -ls test/p=p1
Found 2 items
-rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
-rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
{noformat}
The reason is that 
[Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
 will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has 
no partition for the files, the {{oldPath}} is null and thus the files get no 
chance to be cleaned. We should also clean {{destf}} in method 
[Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
 to fix the issue.


> Hive "insert overwrite" doesn't replace the destination files if no partition 
> in metastore for the files
> 
>
> Key: HIVE-18927
> URL: https://issues.apache.org/jira/browse/HIVE-18927
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: wangzhihao
>Priority: Major
>
> [This 
> post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
>  describes a way to reproduce this issue:
> {noformat}
> # Add some files into file system but no partition in metastore to track it.
> hdfs dfs -put test.txt test/p=p1
> # Insert overwrite the partition(p = p1)
> DROP TABLE IF EXISTS partition_test;
> CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
> INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;
> # verify the text.txt is not removed.
> hdfs dfs -ls test/p=p1
> Found 2 items
> -rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
> -rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
> {noformat}
> The reason is that 
> [Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
>  will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has 
> no partition for the files, the {{oldPath}} is null and thus the files get no 
> chance to be cleaned. We should also clean {{destf}} in method 
> [Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
>  to fix the issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18927) Hive "insert overwrite" doesn't replace the destination files if no partition in metastore for the files

[wangzhihao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

wangzhihao updated HIVE-18927:
--
Description: 
[This 
post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
 describes a way to produce this issue:
{noformat}
# Add some files into file system but no partition in metastore to track it.
hdfs dfs -put test.txt test/p=p1

# Insert overwrite the partition(p = p1)
DROP TABLE IF EXISTS partition_test;
CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;

# verify the text.txt is not removed.
hdfs dfs -ls test/p=p1
Found 2 items
-rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
-rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
{noformat}
The reason is that 
[Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
 will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has 
no partition for the files, the {{oldPath}} is null and thus the files get no 
chance to be cleaned. We should also clean {{destf}} in method 
[Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
 to fix the issue.

  was:
[This 
post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
 describe a way to produce this issue:
{noformat}
# Add some files into file system but no partition in metastore to track it.
hdfs dfs -put test.txt test/p=p1

# Insert overwrite the partition(p = p1)
DROP TABLE IF EXISTS partition_test;
CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;

# verify the text.txt is not removed.
hdfs dfs -ls test/p=p1
Found 2 items
-rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
-rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
{noformat}
The reason is that 
[Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
 will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has 
no partition for the files, the {{oldPath}} is null and thus the files get no 
chance to be cleaned. We should also clean {{destf}} in method 
[Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
 to fix the issue.


> Hive "insert overwrite" doesn't replace the destination files if no partition 
> in metastore for the files
> 
>
> Key: HIVE-18927
> URL: https://issues.apache.org/jira/browse/HIVE-18927
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: wangzhihao
>Priority: Major
>
> [This 
> post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
>  describes a way to produce this issue:
> {noformat}
> # Add some files into file system but no partition in metastore to track it.
> hdfs dfs -put test.txt test/p=p1
> # Insert overwrite the partition(p = p1)
> DROP TABLE IF EXISTS partition_test;
> CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
> INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;
> # verify the text.txt is not removed.
> hdfs dfs -ls test/p=p1
> Found 2 items
> -rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
> -rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
> {noformat}
> The reason is that 
> [Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
>  will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has 
> no partition for the files, the {{oldPath}} is null and thus the files get no 
> chance to be cleaned. We should also clean {{destf}} in method 
> [Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
>  to fix the issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18875) Enable SMB Join by default in Tez

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394057#comment-16394057
 ] 

Hive QA commented on HIVE-18875:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m  1s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9583/dev-support/hive-personality.sh
 |
| git revision | master / 41afa4e |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9583/yetus/patch-asflicense-problems.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9583/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Enable SMB Join by default in Tez
> -
>
> Key: HIVE-18875
> URL: https://issues.apache.org/jira/browse/HIVE-18875
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18875.1.patch, HIVE-18875.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394041#comment-16394041
 ] 

Hive QA commented on HIVE-18877:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913806/HIVE-18877.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 108 failed/errored test(s), 13747 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats] 
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] 
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_1]
 (batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[update_access_time_non_current_db]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_div0]
 (batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.org.apache.hadoop.hive.cli.TestNegativeCliDriver
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_convert] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[mm_truncate_cols]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_bucketmapjoin]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[smb_mapjoin_14] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[sortmerge_mapjoin_mismatch_1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[special_character_in_tabnames_1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_out_of_range]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[split_sample_wrong_format]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_join_2] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_orderby_2]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_grandparent]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_groupby]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_multiple_cols_in_select]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_aggregate]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_select_distinct]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_subquery_chain_exists]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[temp_table_rename]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[touch2] 
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_bucketed_column]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_list_bucketing]
 (batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_column_seqfile]
 (batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_nonexistant_column]
 (batchId=94)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[truncate_partition_column]
 (batchId=94)

[jira] [Commented] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394019#comment-16394019
 ] 

Hive QA commented on HIVE-18877:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} The patch beeline passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 189 
unchanged - 1 fixed = 189 total (was 190) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
11s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 58s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9582/dev-support/hive-personality.sh
 |
| git revision | master / 41afa4e |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9582/yetus/patch-asflicense-problems.txt
 |
| modules | C: beeline itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9582/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> HiveSchemaTool.validateSchemaTables() should wrap a SQLException when 
> rethrowing
> 
>
> Key: HIVE-18877
> URL: https://issues.apache.org/jira/browse/HIVE-18877
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
> Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, 
> HIVE-18877.3.patch
>
>
> If schematool is run with the -verbose flag then it will print a stack trace 
> for an exception that occurs. If a SQLException is caught during 
> HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown 
> containing the text of the SQLException. If we instead throw  a 
> HiveMetaException that wraps the SQLException then the stacktrace will help 
> with diagnosis of issues where the SQLException contains a generic error 
> text. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394004#comment-16394004
 ] 

Hive QA commented on HIVE-18862:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913789/HIVE-18862.01.patch

{color:green}SUCCESS:{color} +1 due to 1759 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 638 failed/errored test(s), 12954 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Updated] (HIVE-18928) HS2: Perflogger has a race condition

[Gopal V (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gopal V updated HIVE-18928:
---
Description: 
{code}

Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
~[?:1.8.0_112]
at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
~[?:1.8.0_112]
at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
~[?:1.8.0_112]
at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
~[?:1.8.0_112]
at com.google.common.collect.Iterables.toArray(Iterables.java:316) 
~[guava-19.0.jar:?]
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342) 
~[guava-19.0.jar:?]
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327) 
~[guava-19.0.jar:?]
at 
org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218) 
~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
 ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
{code}


  was:
{code}
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
~[?:1.8.0_112]
at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
~[?:1.8.0_112]
at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
~[?:1.8.0_112]
at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
~[?:1.8.0_112]
at com.google.common.collect.Iterables.toArray(Iterables.java:316) 
~[guava-19.0.jar:?]
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342) 
~[guava-19.0.jar:?]
at com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327) 
~[guava-19.0.jar:?]
at 
org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218) 
~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498) 
~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
at 
org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
 ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
{code



> HS2: Perflogger has a race condition
> 
>
> Key: HIVE-18928
> URL: https://issues.apache.org/jira/browse/HIVE-18928
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Priority: Major
>
> {code}
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) 
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471) 
> ~[?:1.8.0_112]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469) 
> ~[?:1.8.0_112]
> at java.util.AbstractCollection.toArray(AbstractCollection.java:196) 
> ~[?:1.8.0_112]
> at com.google.common.collect.Iterables.toArray(Iterables.java:316) 
> ~[guava-19.0.jar:?]
> at 
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:342) 
> ~[guava-19.0.jar:?]
> at 
> com.google.common.collect.ImmutableMap.copyOf(ImmutableMap.java:327) 
> ~[guava-19.0.jar:?]
> at 
> org.apache.hadoop.hive.ql.log.PerfLogger.getEndTimes(PerfLogger.java:218) 
> ~[hive-common-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1561) 
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1498) 
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at 
> org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:198)
>  ~[hive-service-3.0.0.3.0.0.2-132.jar:3.0.0.3.0.0.2-132]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394001#comment-16394001
 ] 

Hive QA commented on HIVE-18862:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
11s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 58s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9581/dev-support/hive-personality.sh
 |
| git revision | master / 41afa4e |
| Default Java | 1.8.0_111 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9581/yetus/whitespace-eol.txt 
|
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9581/yetus/patch-asflicense-problems.txt
 |
| modules | C: . itests/util ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9581/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch
>
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18915) Better client logging when a HoS session can't be opened

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18915:

Description: Users just get a {{FAILED: Execution Error, return code 30041 
from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark 
client for Spark session [id]}} when a HoS session can't be opened, would be 
better   (was: Users just get a {{FAILED: Execution Error, return code 30041 
from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark 
client for Spark session [id]} when a HoS session can't be opened, would be 
better )

> Better client logging when a HoS session can't be opened
> 
>
> Key: HIVE-18915
> URL: https://issues.apache.org/jira/browse/HIVE-18915
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Priority: Major
>
> Users just get a {{FAILED: Execution Error, return code 30041 from 
> org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client 
> for Spark session [id]}} when a HoS session can't be opened, would be better 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18436) Upgrade to Spark 2.3.0

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18436:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master. Thanks Rui for the review.

> Upgrade to Spark 2.3.0
> --
>
> Key: HIVE-18436
> URL: https://issues.apache.org/jira/browse/HIVE-18436
> Project: Hive
>  Issue Type: Task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18436.1.patch, HIVE-18436.2.patch, 
> HIVE-18436.3.patch
>
>
> Branching has been completed. Release candidates should be published soon. 
> Might be a while before the actual release, but at least we get to identify 
> any issues early.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18926) Imporve operator-tree matching

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393986#comment-16393986
 ] 

Hive QA commented on HIVE-18926:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913787/HIVE-18926.01.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 13351 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Updated] (HIVE-18831) Differentiate errors that are thrown by Spark tasks

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18831:

Attachment: HIVE-18831.4.patch

> Differentiate errors that are thrown by Spark tasks
> ---
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, 
> HIVE-18831.3.patch, HIVE-18831.4.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we 
> don't differentiate between errors from HS2 / RSC vs. errors thrown by 
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its 
> difficult to know what part of the excution threw the exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-16992) LLAP: better default lambda for LRFU policy

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393968#comment-16393968
 ] 

Sergey Shelukhin commented on HIVE-16992:
-

Looks like lambda needs to be reduced massively, right now it is indeed a 
glorified LRU.
Overall it seems like due to the clock being the counter of cache ops, the 
LRU/LFU balance is also dependent to some extent on cache size.

I will post a patch next week after clean up and looking at some query runs.

> LLAP: better default lambda for LRFU policy
> ---
>
> Key: HIVE-16992
> URL: https://issues.apache.org/jira/browse/HIVE-16992
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> LRFU is currently skewed heavily towards LRU; there are 10k-s or 100k-s of 
> buffers tracked during a typical workload, but the heap size is around 700. 
> We should see if making it closer to LFU (by tweaking the lambda) will 
> improve hit rate with small queries infrequently interleaved with large 
> scans; and whether it will have negative effects due to perf overhead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18926) Imporve operator-tree matching

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393958#comment-16393958
 ] 

Hive QA commented on HIVE-18926:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
36s{color} | {color:red} ql: The patch generated 2 new + 134 unchanged - 8 
fixed = 136 total (was 142) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
14s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 51s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9580/dev-support/hive-personality.sh
 |
| git revision | master / b362de3 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9580/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9580/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9580/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Imporve operator-tree matching
> --
>
> Key: HIVE-18926
> URL: https://issues.apache.org/jira/browse/HIVE-18926
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18926.01.patch
>
>
> currently joins are not matched



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18927) Hive "insert overwrite" doesn't replace the destination files if no partition in metastore for the files

[wangzhihao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

wangzhihao updated HIVE-18927:
--
Description: 
[This 
post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
 describe a way to produce this issue:
{noformat}
# Add some files into file system but no partition in metastore to track it.
hdfs dfs -put test.txt test/p=p1

# Insert overwrite the partition(p = p1)
DROP TABLE IF EXISTS partition_test;
CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;

# verify the text.txt is not removed.
hdfs dfs -ls test/p=p1
Found 2 items
-rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
-rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
{noformat}
The reason is that 
[Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
 will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has 
no partition for the files, the {{oldPath}} is null and thus the files get no 
chance to be cleaned. We should also clean {{destf}} in method 
[Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
 to fix the issue.

  was:
[This 
post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
 describe a way to produce this issue:
{noformat}
# Add some files into file system but no partition in metastore to track it.
hdfs dfs -put test.txt test/p=p1

# Insert overwrite the partition(p = p1)
DROP TABLE IF EXISTS partition_test;
CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;

# verify the text.txt is not removed.
hdfs dfs -ls test/p=p1
Found 2 items
-rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
-rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
{noformat}
The reason is that 
[Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
 will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore have 
no partition for the files, the {{oldPath}} is null and thus the files get no 
chance to be cleaned. We should also clean {{destf}} in method 
[Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
 to fix the issue.


> Hive "insert overwrite" doesn't replace the destination files if no partition 
> in metastore for the files
> 
>
> Key: HIVE-18927
> URL: https://issues.apache.org/jira/browse/HIVE-18927
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: wangzhihao
>Priority: Major
>
> [This 
> post|http://www.ericlin.me/2015/05/hive-insert-overwrite-does-not-remove-existing-data/]
>  describe a way to produce this issue:
> {noformat}
> # Add some files into file system but no partition in metastore to track it.
> hdfs dfs -put test.txt test/p=p1
> # Insert overwrite the partition(p = p1)
> DROP TABLE IF EXISTS partition_test;
> CREATE EXTERNAL TABLE partition_test (a int) PARTITIONED BY (p string);
> INSERT OVERWRITE TABLE partition_test PARTITION (p = 'p1') SELECT 123;
> # verify the text.txt is not removed.
> hdfs dfs -ls test/p=p1
> Found 2 items
> -rwxr-xr-x   3 hdfs supergroup 194965 2015-05-05 00:15 test/p=p1/00_0
> -rw-r--r--   3 hdfs supergroup  8 2015-05-05 00:10 test/p=p1/test.txt
> {noformat}
> The reason is that 
> [Hive.loadPartition|https://github.com/apache/hive/blob/9b36ffa92cc4e0f47ea03d8d167debe743342f5b/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L1652]
>  will try to {{replaceFiles}} only if {{oldPath}} exists. Since metastore has 
> no partition for the files, the {{oldPath}} is null and thus the files get no 
> chance to be cleaned. We should also clean {{destf}} in method 
> [Hive.replaceFiles|https://github.com/apache/hive/blob/b362de3871764731d8371657b07140e37a3c5105/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3817]
>  to fix the issue.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-18662) hive.acid.key.index is missing entries

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman resolved HIVE-18662.
---
   Resolution: Fixed
 Assignee: Eugene Koifman
Fix Version/s: 3.0.0

fixed in HIVE-18817

> hive.acid.key.index is missing entries
> --
>
> Key: HIVE-18662
> URL: https://issues.apache.org/jira/browse/HIVE-18662
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
>
> OrcRecordUpdater.KeyIndexBuilder stores an index in ORC footer where each 
> entry is the last ROW__ID of each stripe.  In acid1 this is used to filter 
> the events from delta file when merging with part of the base.
>  
> as can be seen in {{TestTxnCommands.testVersioning()}} (added in HIVE-18659) 
> the {{hive.acid.key.index}} is empty.  
>  
> This is because very little data is written and WriterImpl.flushStripe() is 
> not called except when {
> {WriterImpl.close()}
> is called.  In the later, {{WriterCallback.preFooterWrite()}} is called 
> before {{preStripeWrite}} and so KeyIndexBuilder.preFooterWriter() records 
> nothing in \{{hive.acid.key.index}}
>  
> need to investigate if this is an issue, in particular acid 2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18863) trunc() calls itself trunk() in an error message

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393908#comment-16393908
 ] 

Vihang Karajgaonkar commented on HIVE-18863:


+1

> trunc() calls itself trunk() in an error message
> 
>
> Key: HIVE-18863
> URL: https://issues.apache.org/jira/browse/HIVE-18863
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Reporter: Tim Armstrong
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
>  Labels: newbie
> Attachments: HIVE-18863.1.patch
>
>
> {noformat}
> > select  trunc('millennium', cast('2001-02-16 20:38:40' as timestamp))
> FAILED: SemanticException Line 0:-1 Argument type mismatch ''2001-02-16 
> 20:38:40'': trunk() only takes STRING/CHAR/VARCHAR types as second argument, 
> got TIMESTAMP
> {noformat}
> I saw this on a derivative of Hive 1.1.0 (cdh5.15.0), but the string still 
> seems to be present on master:
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L262



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18739) Add support for Export from unpartitioned Acid table

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18739:
--
Attachment: HIVE-18739.06.patch

> Add support for Export from unpartitioned Acid table
> 
>
> Key: HIVE-18739
> URL: https://issues.apache.org/jira/browse/HIVE-18739
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18739.01.patch, HIVE-18739.04.patch, 
> HIVE-18739.04.patch, HIVE-18739.06.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393906#comment-16393906
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


Thanks for the review [~alangates] Attached the latest patch from the PR to run 
the precommit tests to make sure there is nothing fundamentally broken. If 
there are no unexpected failures, I will merge the PR into standalone-metastore 
on Monday.

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch, 
> HIVE-17580.093-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-17580:
---
Attachment: HIVE-17580.093-standalone-metastore.patch

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
>  Labels: pull-request-available
> Attachments: HIVE-17580.003-standalone-metastore.patch, 
> HIVE-17580.04-standalone-metastore.patch, 
> HIVE-17580.05-standalone-metastore.patch, 
> HIVE-17580.06-standalone-metastore.patch, 
> HIVE-17580.07-standalone-metastore.patch, 
> HIVE-17580.08-standalone-metastore.patch, 
> HIVE-17580.09-standalone-metastore.patch, 
> HIVE-17580.092-standalone-metastore.patch, 
> HIVE-17580.093-standalone-metastore.patch
>
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18832) Support change management for trashing data files from ACID tables.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393872#comment-16393872
 ] 

Hive QA commented on HIVE-18832:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913725/HIVE-18832.1.patch

{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12958 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Updated] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18560:

Status: Patch Available  (was: Open)

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18560.01.patch
>
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18560:

Attachment: HIVE-18560.01.patch

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18560.01.patch
>
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18560:

Component/s: Tests

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18560:

Attachment: (was: HIVE-18560.01.patch)

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>  Components: Tests
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18560:

Summary: qtests: QTestUtil refactor/split - QOutProcessor  (was: qtests: 
QTestUtil refactor/split)

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18560.01.patch
>
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18560) qtests: QTestUtil refactor/split - QOutProcessor

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18560:

Attachment: HIVE-18560.01.patch

> qtests: QTestUtil refactor/split - QOutProcessor
> 
>
> Key: HIVE-18560
> URL: https://issues.apache.org/jira/browse/HIVE-18560
> Project: Hive
>  Issue Type: Task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18560.01.patch
>
>
> QTestUtil needs some refactoring, as it has become a large, hard-to-maintain, 
> hard-to-test class with a bunch of responsibilities:
>  * running qtests
>  * parsing/evaluating results
>  * masking/processing results
> First, I would extract a processing class from it (e.g. QOutProcessor or 
> something like that), which should be capable of being tested easily, and 
> contains all the logic what's needed to make the final q.out results from the 
> original q.out file. 
> This processing class shall remain untouched in case the testing 
> infrastructure (QTestUtil) is replaced or changed in some ways.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18832) Support change management for trashing data files from ACID tables.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393810#comment-16393810
 ] 

Hive QA commented on HIVE-18832:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
36s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
49s{color} | {color:red} ql in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red}  0m 
32s{color} | {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  0m 32s{color} 
| {color:red} hive-unit in the patch failed. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
15s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 2 
unchanged - 0 fixed = 6 total (was 2) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} ql: The patch generated 0 new + 85 unchanged - 4 
fixed = 85 total (was 89) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
16s{color} | {color:red} standalone-metastore: The patch generated 3 new + 32 
unchanged - 18 fixed = 35 total (was 50) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 37s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9579/dev-support/hive-personality.sh
 |
| git revision | master / b362de3 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9579/yetus/patch-mvninstall-itests_hive-unit.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9579/yetus/patch-mvninstall-ql.txt
 |
| compile | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9579/yetus/patch-compile-itests_hive-unit.txt
 |
| javac | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9579/yetus/patch-compile-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9579/yetus/diff-checkstyle-itests_hive-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9579/yetus/diff-checkstyle-standalone-metastore.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9579/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests/hive-unit ql standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9579/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Support change management for trashing data files from ACID tables.
> ---
>
> Key: 

[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393745#comment-16393745
 ] 

Hive QA commented on HIVE-18864:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913723/HIVE-18864.02.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 13351 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone

[Piotr Findeisen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393721#comment-16393721
 ] 

Piotr Findeisen commented on HIVE-18925:


i created a PR with this change. https://github.com/apache/hive/pull/319

> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
>  Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Priority: Major
>  Labels: pull-request-available
>
> Hive Server2 doesn't  work if started with 
> {{-Duser.timezone=America/Bahia_Banderas}}
>  
> Steps to reproduce
>  # use [https://github.com/big-data-europe/docker-hive]
>  # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to 
> {{hive-server}} docker container environment configuration
>  # {{docker-compose up}}
>  # 
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1 
> --verbose=true
> ...
> jdbc:hive2://localhost:1> select 1;{code}
> The above fails and prints
> {noformat}
> Error: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas) (state=08S01,code=0)
> java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause 
> with org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at java.lang.Throwable.initCause(Throwable.java:457)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
> at 
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
> ... 15 more
> Caused by: java.lang.ExceptionInInitializerError: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
> ... 21 more{noformat}
> From the above stacktrace it's not visible what is the cause, but i think 
> it's initialization of 
> {{org.apache.hive.common.util.TimestampParser#startingDateValue}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Issue Comment Deleted] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone

[Piotr Findeisen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Findeisen updated HIVE-18925:
---
Comment: was deleted

(was: i created a PR with this change. https://github.com/apache/hive/pull/319)

> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
>  Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Priority: Major
>  Labels: pull-request-available
>
> Hive Server2 doesn't  work if started with 
> {{-Duser.timezone=America/Bahia_Banderas}}
>  
> Steps to reproduce
>  # use [https://github.com/big-data-europe/docker-hive]
>  # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to 
> {{hive-server}} docker container environment configuration
>  # {{docker-compose up}}
>  # 
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1 
> --verbose=true
> ...
> jdbc:hive2://localhost:1> select 1;{code}
> The above fails and prints
> {noformat}
> Error: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas) (state=08S01,code=0)
> java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause 
> with org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at java.lang.Throwable.initCause(Throwable.java:457)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
> at 
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
> ... 15 more
> Caused by: java.lang.ExceptionInInitializerError: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
> ... 21 more{noformat}
> From the above stacktrace it's not visible what is the cause, but i think 
> it's initialization of 
> {{org.apache.hive.common.util.TimestampParser#startingDateValue}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-18925:
--
Labels: pull-request-available  (was: )

> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
>  Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Priority: Major
>  Labels: pull-request-available
>
> Hive Server2 doesn't  work if started with 
> {{-Duser.timezone=America/Bahia_Banderas}}
>  
> Steps to reproduce
>  # use [https://github.com/big-data-europe/docker-hive]
>  # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to 
> {{hive-server}} docker container environment configuration
>  # {{docker-compose up}}
>  # 
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1 
> --verbose=true
> ...
> jdbc:hive2://localhost:1> select 1;{code}
> The above fails and prints
> {noformat}
> Error: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas) (state=08S01,code=0)
> java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause 
> with org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at java.lang.Throwable.initCause(Throwable.java:457)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
> at 
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
> ... 15 more
> Caused by: java.lang.ExceptionInInitializerError: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
> ... 21 more{noformat}
> From the above stacktrace it's not visible what is the cause, but i think 
> it's initialization of 
> {{org.apache.hive.common.util.TimestampParser#startingDateValue}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393720#comment-16393720
 ] 

ASF GitHub Bot commented on HIVE-18925:
---

GitHub user findepi opened a pull request:

https://github.com/apache/hive/pull/319

HIVE-18925 Prevent failure when JVM zone had change on 1970-01-01

This prevents static initializer failure if JVM zone did not observe
1970-01-01 00:00:00, while retaining original behavior for all other
zones.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/findepi/hive fix-startup-banderas

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/319.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #319


commit 98b8becb0571da00bf90e073f13f70b4a7698472
Author: Piotr Findeisen 
Date:   2018-03-09T22:52:40Z

HIVE-18925 Prevent failure when JVM zone had change on 1970-01-01

This prevents static initializer failure if JVM zone did not observe
1970-01-01 00:00:00, while retaining original behavior for all other
zones.




> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
>  Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Priority: Major
>  Labels: pull-request-available
>
> Hive Server2 doesn't  work if started with 
> {{-Duser.timezone=America/Bahia_Banderas}}
>  
> Steps to reproduce
>  # use [https://github.com/big-data-europe/docker-hive]
>  # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to 
> {{hive-server}} docker container environment configuration
>  # {{docker-compose up}}
>  # 
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1 
> --verbose=true
> ...
> jdbc:hive2://localhost:1> select 1;{code}
> The above fails and prints
> {noformat}
> Error: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas) (state=08S01,code=0)
> java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause 
> with org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at java.lang.Throwable.initCause(Throwable.java:457)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
> at 
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
> ... 15 more
> Caused by: java.lang.ExceptionInInitializerError: null
> at 

[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393713#comment-16393713
 ] 

Aihua Xu commented on HIVE-14792:
-

[~mithun] Uploaded the patch-5: the only change I did is to make a copy of 
original properties {{Map originalTableParameters = new 
HashMap<>(table.getParameters());}}, so when there is any change to the 
property, we will add it to the table parameter list. Before, we were adding 
everything from Serde to table parameter list as well.

> AvroSerde reads the remote schema-file at least once per mapper, per table 
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Aihua Xu
>Priority: Major
>  Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch, 
> HIVE-14792.4.patch, HIVE-14792.5.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause 
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn 
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties) 
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
>   schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
>   // Get column names and sort order
>   columnNames = Arrays.asList(columnNameProperty.split(","));
>   columnTypes = 
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
>   schema = getSchemaFromCols(properties, columnNames, columnTypes, 
> columnCommentProperty);
>  
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
>  schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized 
> (i.e. at least once per mapper), the schema file is read remotely. For 
> queries with thousands of mappers, this leads to a stampede to the handful 
> (3?) datanodes that host the schema-file. In the best case, this causes 
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part 
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive 
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the 
> size-limit on table-parameters. The typical size of the Avro-schema file is 
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter 
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made 
> available as part of table-properties (but not serialized into the 
> metastore), the downstream logic will remain largely intact. I have a patch 
> that does this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Aihua Xu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aihua Xu updated HIVE-14792:

Attachment: HIVE-14792.5.patch

> AvroSerde reads the remote schema-file at least once per mapper, per table 
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Aihua Xu
>Priority: Major
>  Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch, 
> HIVE-14792.4.patch, HIVE-14792.5.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause 
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn 
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties) 
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
>   schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
>   // Get column names and sort order
>   columnNames = Arrays.asList(columnNameProperty.split(","));
>   columnTypes = 
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
>   schema = getSchemaFromCols(properties, columnNames, columnTypes, 
> columnCommentProperty);
>  
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
>  schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized 
> (i.e. at least once per mapper), the schema file is read remotely. For 
> queries with thousands of mappers, this leads to a stampede to the handful 
> (3?) datanodes that host the schema-file. In the best case, this causes 
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part 
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive 
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the 
> size-limit on table-parameters. The typical size of the Avro-schema file is 
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter 
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made 
> available as part of table-properties (but not serialized into the 
> metastore), the downstream logic will remain largely intact. I have a patch 
> that does this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone

[Piotr Findeisen (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393711#comment-16393711
 ] 

Piotr Findeisen commented on HIVE-18925:


I think it should be sufficient to replace
{code:java}
protected final static DateTime startingDateValue = new DateTime(1970, 1, 1, 0, 
0, 0, 0);{code}
with
{code:java}
protected final static DateTime startingDateValue = new DateTime(1970, 1, 1, 0, 
0, 0, 0, DateTimeZone.UTC)
.withZoneRetainFields(DateTimeZone.getDefault());{code}
 

This is equivalent to the former for any normal zone, but doesn't fail for 
"odd" zone that skipped 1970-01-01 00:00:00 in its local time.

 

 

> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
>  Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Priority: Major
>
> Hive Server2 doesn't  work if started with 
> {{-Duser.timezone=America/Bahia_Banderas}}
>  
> Steps to reproduce
>  # use [https://github.com/big-data-europe/docker-hive]
>  # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to 
> {{hive-server}} docker container environment configuration
>  # {{docker-compose up}}
>  # 
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1 
> --verbose=true
> ...
> jdbc:hive2://localhost:1> select 1;{code}
> The above fails and prints
> {noformat}
> Error: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas) (state=08S01,code=0)
> java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause 
> with org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
> at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
> at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
> at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
> at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
> at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
> at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
> at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
> at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas)
> at java.lang.Throwable.initCause(Throwable.java:457)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
> at 
> org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
> at 
> org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
> at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
> ... 15 more
> Caused by: java.lang.ExceptionInInitializerError: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
> at 
> org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
> ... 21 more{noformat}
> From the above stacktrace it's not visible what is the cause, but i think 
> it's initialization of 
> 

[jira] [Updated] (HIVE-15218) Kyro Exception on subsequent run of a query in LLAP mode

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-15218:

Resolution: Not A Problem
Status: Resolved  (was: Patch Available)

> Kyro Exception on subsequent run of a query in LLAP mode
> 
>
> Key: HIVE-15218
> URL: https://issues.apache.org/jira/browse/HIVE-15218
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-15218.1.patch, HIVE-15218.2.patch, 
> HIVE-15218.branch-1.patch
>
>
> Following exception is observed when running TPCDS query19 during concurrency 
> test
> {code}
> Vertex failed, vertexName=Map 3, vertexId=vertex_1477340478603_0610_9_05, 
> diagnostics=[Task failed, taskId=task_1477340478603_0610_9_05_06, 
> diagnostics=[TaskAttempt 0 killed, TaskAttempt 1 failed, info=[Error: Error 
> while running task ( failure ) : 
> attempt_1477340478603_0610_9_05_06_1:java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Failed to load plan: 
> hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/ndembla/0559ce24-663e-482a-a0ea-106d220b53be/hi...
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
>   at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>   at 
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Failed to load plan: 
> hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/ndembla/0559ce24-663e-482a-a0ea-106d220b53be/hi...
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:57)
>   at 
> org.apache.hadoop.hive.ql.exec.ObjectCacheWrapper.retrieve(ObjectCacheWrapper.java:40)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:129)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
>   ... 15 more
> Caused by: java.lang.RuntimeException: Failed to load plan: 
> hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/ndembla/0559ce24-663e-482a-a0ea-106d220b53be/hi...
>   at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:469)
>   at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:305)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor$1.call(MapRecordProcessor.java:132)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:55)
>   ... 18 more
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: 
> Encountered unregistered class ID: 63
> Serialization trace:
> statistics (org.apache.hadoop.hive.ql.plan.TableScanDesc)
> conf (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
>   at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:182)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>   at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708)
>   at 
> 

[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393703#comment-16393703
 ] 

Jason Dere commented on HIVE-18846:
---

RB at https://reviews.apache.org/r/66005/

> Query results cache: Allow queries to refer to the pending results of a query 
> that has not finished yet
> ---
>
> Key: HIVE-18846
> URL: https://issues.apache.org/jira/browse/HIVE-18846
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch
>
>
> Currently, a query's results can only be looked up in the cache if the query 
> has completely finished execution. Allow new queries to use the results cache 
> to find queries that are still executing so they can re-use the results when 
> the query has finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393700#comment-16393700
 ] 

Jason Dere commented on HIVE-18846:
---

Update patch to fix empty results case which came up during testing.

> Query results cache: Allow queries to refer to the pending results of a query 
> that has not finished yet
> ---
>
> Key: HIVE-18846
> URL: https://issues.apache.org/jira/browse/HIVE-18846
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch
>
>
> Currently, a query's results can only be looked up in the cache if the query 
> has completely finished execution. Allow new queries to use the results cache 
> to find queries that are still executing so they can re-use the results when 
> the query has finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-18846:
--
Status: Patch Available  (was: Open)

> Query results cache: Allow queries to refer to the pending results of a query 
> that has not finished yet
> ---
>
> Key: HIVE-18846
> URL: https://issues.apache.org/jira/browse/HIVE-18846
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch
>
>
> Currently, a query's results can only be looked up in the cache if the query 
> has completely finished execution. Allow new queries to use the results cache 
> to find queries that are still executing so they can re-use the results when 
> the query has finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18846) Query results cache: Allow queries to refer to the pending results of a query that has not finished yet

[Jason Dere (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Dere updated HIVE-18846:
--
Attachment: HIVE-18846.2.patch

> Query results cache: Allow queries to refer to the pending results of a query 
> that has not finished yet
> ---
>
> Key: HIVE-18846
> URL: https://issues.apache.org/jira/browse/HIVE-18846
> Project: Hive
>  Issue Type: Sub-task
>  Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18846.1.patch, HIVE-18846.2.patch
>
>
> Currently, a query's results can only be looked up in the cache if the query 
> has completely finished execution. Allow new queries to use the results cache 
> to find queries that are still executing so they can re-use the results when 
> the query has finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18864) ValidWriteIdList snapshot seems incorrect if obtained after allocating writeId by current transaction.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393692#comment-16393692
 ] 

Hive QA commented on HIVE-18864:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 54s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9578/dev-support/hive-personality.sh
 |
| git revision | master / b362de3 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9578/yetus/patch-asflicense-problems.txt
 |
| modules | C: metastore ql standalone-metastore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9578/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ValidWriteIdList snapshot seems incorrect if obtained after allocating 
> writeId by current transaction.
> --
>
> Key: HIVE-18864
> URL: https://issues.apache.org/jira/browse/HIVE-18864
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18864.01.patch, HIVE-18864.02.patch
>
>
> For multi-statement txns, it is possible that write on a table happens after 
> a read. Let's see the below scenario.
>  # Committed txn=9 writes on table T1 with writeId=5.
>  # Open txn=10. ValidTxnList(open:null, txn_HWM=10),
>  # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
>  # Open txn=11, writes on table T1 with writeid=6.
>  # Read table T1 from txn=10. ValidWriteIdList(open:null, write_HWM=5).
>  # Write table T1 from txn=10 with writeId=7.
>  # Read table T1 from txn=10. {color:#d04437}*ValidWriteIdList(open:null, 
> write_HWM=7)*. – This read will able to see rows added by txn=11 which is 
> still open.{color}
> {color:#d04437}So, it is needed to rebuild the open/aborted list of 
> ValidWriteIdList based on txn_HWM. Any 

[jira] [Updated] (HIVE-17206) make a version of Compactor specific to unbucketed tables

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-17206:
--
Description: 
current Compactor will work but is not optimized/flexible enough

The current compactor is designed to generate the number of splits equal to the 
number of buckets in the table.   That is the degree of parallelism.

For unbucketed tables, the same is used but the "number of buckets" is derived 
from the files found in the deltas.  For small writes, there will likely be 
just 1 bucket_0 file.  For large writes, the parallelism of the write 
determines the number of output files.

Need to make sure Compactor can control parallelism for unbucketed tables as it 
wishes.  For example, hash partition all records (by ROW__ID?) into N disjoint 
sets.


  was:current Compactor will work but is not optimized/flexible enough


> make a version of Compactor specific to unbucketed tables
> -
>
> Key: HIVE-17206
> URL: https://issues.apache.org/jira/browse/HIVE-17206
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> current Compactor will work but is not optimized/flexible enough
> The current compactor is designed to generate the number of splits equal to 
> the number of buckets in the table.   That is the degree of parallelism.
> For unbucketed tables, the same is used but the "number of buckets" is 
> derived from the files found in the deltas.  For small writes, there will 
> likely be just 1 bucket_0 file.  For large writes, the parallelism of the 
> write determines the number of output files.
> Need to make sure Compactor can control parallelism for unbucketed tables as 
> it wishes.  For example, hash partition all records (by ROW__ID?) into N 
> disjoint sets.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18751) ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393645#comment-16393645
 ] 

Hive QA commented on HIVE-18751:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913724/HIVE-18751.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 13350 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-18905) HS2: SASL auth loads HiveConf for every JDBC call

[Kryvenko Igor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393546#comment-16393546
 ] 

Kryvenko Igor commented on HIVE-18905:
--

[~gopalv] Please, could you review?

> HS2: SASL auth loads HiveConf for every JDBC call
> -
>
> Key: HIVE-18905
> URL: https://issues.apache.org/jira/browse/HIVE-18905
> Project: Hive
>  Issue Type: Bug
>Reporter: Gopal V
>Assignee: Kryvenko Igor
>Priority: Minor
> Attachments: HIVE-18905.patch
>
>
> SASL authentication filter does a new HiveConf() for no good reason.
> {code}
>   public static PasswdAuthenticationProvider 
> getAuthenticationProvider(AuthMethods authMethod)
> throws AuthenticationException {
> return getAuthenticationProvider(authMethod, new HiveConf());
>   }
> {code}
> The session HiveConf is not needed to do this operation & it can't be changed 
> after the HS2 starts up (today).
> {code}
> org.apache.hadoop.hive.conf.HiveConf.() HiveConf.java:4404
> org.apache.hive.service.auth.AuthenticationProviderFactory.getAuthenticationProvider(AuthenticationProviderFactory$AuthMethods)
>  AuthenticationProviderFactory.java:61
> org.apache.hive.service.auth.PlainSaslHelper$PlainServerCallbackHandler.handle(Callback[])
>  PlainSaslHelper.java:106
> org.apache.hive.service.auth.PlainSaslServer.evaluateResponse(byte[]) 
> PlainSaslServer.java:103
> org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(byte[])
>  TSaslTransport.java:539
> org.apache.thrift.transport.TSaslTransport.open() TSaslTransport.java:283
> org.apache.thrift.transport.TSaslServerTransport.open() 
> TSaslServerTransport.java:41
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TTransport)
>  TSaslServerTransport.java:216
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run() 
> TThreadPoolServer.java:269
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) 
> ThreadPoolExecutor.java:1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run() 
> ThreadPoolExecutor.java:617
> java.lang.Thread.run() Thread.java:745
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18264) CachedStore: Store cached partitions/col stats within the table cache and make prewarm non-blocking

[Vaibhav Gumashta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vaibhav Gumashta updated HIVE-18264:

Attachment: HIVE-18264.6.patch

> CachedStore: Store cached partitions/col stats within the table cache and 
> make prewarm non-blocking
> ---
>
> Key: HIVE-18264
> URL: https://issues.apache.org/jira/browse/HIVE-18264
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-18264.1.patch, HIVE-18264.2.patch, 
> HIVE-18264.3.patch, HIVE-18264.4.patch, HIVE-18264.5.patch, HIVE-18264.6.patch
>
>
> Currently we have a separate cache for partitions and partition col stats 
> which results in some calls iterating through each of these for 
> retrieving/updating. For example, to modify a partition col stat, currently 
> we need to lock table, partition and partition col stats caches which are all 
> separate hashmaps. We can get better performance by organizing 
> hierarchically. For example, we can have a partition, partition col stats and 
> table col stats cache per table to improve on the previous mechanisms. This 
> will also result in better concurrency, since now instead of locking the 
> whole cache, we can selectively lock the table cache and modify multiple 
> tables in parallel. 
> In addition, currently, the prewarm mechanism populates all the caches 
> initially (it skips tables that do not pass whitelist/blacklist filter) and 
> it is a blocking call. This patch also makes prewarm non-blocking so that the 
> calls for tables that are already cached can be served from the memory and 
> the ones that are not can be served from the rdbms. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18727) Update GenericUDFEnforceNotNullConstraint to throw an ERROR instead of Exception on failure

[Kryvenko Igor (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393540#comment-16393540
 ] 

Kryvenko Igor commented on HIVE-18727:
--

[~vgarg] Have you some proposition of package name in which to move 
DataConstraintViolationError?

> Update GenericUDFEnforceNotNullConstraint to throw an ERROR instead of 
> Exception on failure
> ---
>
> Key: HIVE-18727
> URL: https://issues.apache.org/jira/browse/HIVE-18727
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Kryvenko Igor
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18727.patch
>
>
> Throwing an exception makes TezProcessor stop retrying the task. Since this 
> is NOT NULL constraint violation we don't want TezProcessor to keep retrying 
> on failure.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18668) Druid tests fail with shaded guava(?)

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18668:

Summary: Druid tests fail with shaded guava(?)  (was: Really shade guava in 
ql)

> Druid tests fail with shaded guava(?)
> -
>
> Key: HIVE-18668
> URL: https://issues.apache.org/jira/browse/HIVE-18668
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18668.01.patch
>
>
> After HIVE-15393 a test started to fail in druid; after some investigation it 
> turned out that ql doesn't shade it's guava artifact at all...because it 
> shades 'com.google.guava' instead 'com.google.common'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (HIVE-18668) Really shade guava in ql

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reopened HIVE-18668:
-

> Really shade guava in ql
> 
>
> Key: HIVE-18668
> URL: https://issues.apache.org/jira/browse/HIVE-18668
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18668.01.patch
>
>
> After HIVE-15393 a test started to fail in druid; after some investigation it 
> turned out that ql doesn't shade it's guava artifact at all...because it 
> shades 'com.google.guava' instead 'com.google.common'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18668) Really shade guava in ql

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393532#comment-16393532
 ] 

Sergey Shelukhin commented on HIVE-18668:
-

Hmm... I think this actually broke the shading.
The string is group ID and artifact name, which in ql pom is the way it was.
I can name it foo:bar and it will still work if it matches the dependency IDs 
in the pom.

With this patch:
{noformat}
$ jar tf ./ql/target/hive-exec-3.0.0-SNAPSHOT.jar | grep -i stopwatch | grep -v 
apache
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
jodd/datetime/JStopWatch.class
{noformat}

Without this patch:
{noformat}
$ jar tf ./ql/target/hive-exec-3.0.0-SNAPSHOT.jar | grep -i stopwatch | grep -v 
apache
Picked up JAVA_TOOL_OPTIONS: -Djava.awt.headless=true
jodd/datetime/JStopWatch.class
com/google/common/base/Stopwatch$1.class
com/google/common/base/Stopwatch.class
com/google/common/util/concurrent/RateLimiter$SleepingStopwatch$1.class
com/google/common/util/concurrent/RateLimiter$SleepingStopwatch.class
{noformat}


Looks like presence of guava actually breaks Druid tests.
I am going to revert this for now because it makes Hive pick random guava jars 
(version across Hadoop, Tez etc is a big mess), so it cannot be a long term fix 
for Druid anyway. Druid tests need to be fixed in some other manner.

cc [~bslim]

> Really shade guava in ql
> 
>
> Key: HIVE-18668
> URL: https://issues.apache.org/jira/browse/HIVE-18668
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18668.01.patch
>
>
> After HIVE-15393 a test started to fail in druid; after some investigation it 
> turned out that ql doesn't shade it's guava artifact at all...because it 
> shades 'com.google.guava' instead 'com.google.common'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18751) ACID table scan through get_splits UDF doesn't receive ValidWriteIdList configuration.

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393521#comment-16393521
 ] 

Hive QA commented on HIVE-18751:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 21s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9577/dev-support/hive-personality.sh
 |
| git revision | master / 34eb9ed |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9577/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9577/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ACID table scan through get_splits UDF doesn't receive ValidWriteIdList 
> configuration.
> --
>
> Key: HIVE-18751
> URL: https://issues.apache.org/jira/browse/HIVE-18751
> Project: Hive
>  Issue Type: Sub-task
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
>  Labels: ACID, UDF, pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18751.01.patch
>
>
> Per table write ID (HIVE-18192) have replaced global transaction ID with 
> write ID to version data files in ACID/MM tables,
> To ensure snapshot isolation, need to generate ValidWriteIdList for the given 
> txn/table and use it when scan the ACID/MM tables.
> In case of get_splits UDF which runs on ACID table scan query won't receive 
> it properly through configuration (hive.txn.tables.valid.writeids) and hence 
> throws exception. 
> TestAcidOnTez.testGetSplitsLocks is the test failing for the same. Need to 
> fix it.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18804) StatsUtils.getColStatisticsFromExprMap may only provide info for a column once

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393488#comment-16393488
 ] 

Hive QA commented on HIVE-18804:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913580/HIVE-18804.02.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 205 failed/errored test(s), 13351 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Updated] (HIVE-18863) trunc() calls itself trunk() in an error message

[Bharathkrishna Guruvayoor Murali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharathkrishna Guruvayoor Murali updated HIVE-18863:

Status: Patch Available  (was: Open)

Corrected the typo trunk -> trunc

> trunc() calls itself trunk() in an error message
> 
>
> Key: HIVE-18863
> URL: https://issues.apache.org/jira/browse/HIVE-18863
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Reporter: Tim Armstrong
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
>  Labels: newbie
> Attachments: HIVE-18863.1.patch
>
>
> {noformat}
> > select  trunc('millennium', cast('2001-02-16 20:38:40' as timestamp))
> FAILED: SemanticException Line 0:-1 Argument type mismatch ''2001-02-16 
> 20:38:40'': trunk() only takes STRING/CHAR/VARCHAR types as second argument, 
> got TIMESTAMP
> {noformat}
> I saw this on a derivative of Hive 1.1.0 (cdh5.15.0), but the string still 
> seems to be present on master:
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L262



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18863) trunc() calls itself trunk() in an error message

[Bharathkrishna Guruvayoor Murali (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharathkrishna Guruvayoor Murali updated HIVE-18863:

Attachment: HIVE-18863.1.patch

> trunc() calls itself trunk() in an error message
> 
>
> Key: HIVE-18863
> URL: https://issues.apache.org/jira/browse/HIVE-18863
> Project: Hive
>  Issue Type: Bug
>  Components: UDF
>Reporter: Tim Armstrong
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
>  Labels: newbie
> Attachments: HIVE-18863.1.patch
>
>
> {noformat}
> > select  trunc('millennium', cast('2001-02-16 20:38:40' as timestamp))
> FAILED: SemanticException Line 0:-1 Argument type mismatch ''2001-02-16 
> 20:38:40'': trunk() only takes STRING/CHAR/VARCHAR types as second argument, 
> got TIMESTAMP
> {noformat}
> I saw this on a derivative of Hive 1.1.0 (cdh5.15.0), but the string still 
> seems to be present on master:
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTrunc.java#L262



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18917) Add spark.home to hive.conf.restricted.list

[Aihua Xu (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393460#comment-16393460
 ] 

Aihua Xu commented on HIVE-18917:
-

+1. 

> Add spark.home to hive.conf.restricted.list
> ---
>
> Key: HIVE-18917
> URL: https://issues.apache.org/jira/browse/HIVE-18917
> Project: Hive
>  Issue Type: Task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18917.1.patch
>
>
> Add spark.home to hive.conf.restricted.list so its not settable by users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18883) Add findbugs to yetus pre-commit checks

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393427#comment-16393427
 ] 

Sahil Takiar commented on HIVE-18883:
-

[~pvary] is there a good way to test this on Hive QA? I think the patch won't 
get applied until after it checks for {{FINDBUGS_HOME}}

{code}


 Determining needed tests



(Depending upon input size and number of plug-ins, this may take a while)
...


 Applying patch to master



Applying the patch:
Fri Mar  9 06:08:08 UTC 2018
cd /data/hiveptest/working/yetus_PreCommit-HIVE-Build-9564
git apply --binary -v --stat --apply -p1 
/data/hiveptest/logs/PreCommit-HIVE-Build-9564/yetus/patch
{code}

I've tested this locally and it works.

> Add findbugs to yetus pre-commit checks
> ---
>
> Key: HIVE-18883
> URL: https://issues.apache.org/jira/browse/HIVE-18883
> Project: Hive
>  Issue Type: Sub-task
>  Components: Testing Infrastructure
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18883.1.patch, HIVE-18883.2.patch
>
>
> We should enable FindBugs for our YETUS pre-commit checks, this will help 
> overall code quality and should decrease the overall number of bugs in Hive.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18917) Add spark.home to hive.conf.restricted.list

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393413#comment-16393413
 ] 

Sahil Takiar commented on HIVE-18917:
-

[~aihuaxu], [~vihangk1] could one of you take a look? Should be a quick one. 
Change prevents Hive users from setting {{spark.home}} at the session level 
(e.g. via {{set spark.home=[something]}}). Test failures are unrelated. 
subquery_scalar.q and ppd_join5.q failures are known flaky tests.

> Add spark.home to hive.conf.restricted.list
> ---
>
> Key: HIVE-18917
> URL: https://issues.apache.org/jira/browse/HIVE-18917
> Project: Hive
>  Issue Type: Task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18917.1.patch
>
>
> Add spark.home to hive.conf.restricted.list so its not settable by users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18804) StatsUtils.getColStatisticsFromExprMap may only provide info for a column once

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393409#comment-16393409
 ] 

Hive QA commented on HIVE-18804:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m  6s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9576/dev-support/hive-personality.sh
 |
| git revision | master / 34eb9ed |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9576/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9576/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> StatsUtils.getColStatisticsFromExprMap may only provide info for a column once
> --
>
> Key: HIVE-18804
> URL: https://issues.apache.org/jira/browse/HIVE-18804
> Project: Hive
>  Issue Type: Sub-task
>  Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18804.01.patch, HIVE-18804.02.patch
>
>
> currently {{StatsUtils.getColStatisticsFromExprMap}} may duplicate the 
> datasize by passing the info about the same column more than once
> https://github.com/apache/hive/blob/e8e5ab24616aa834f4966efe3a5f437f6bee4d1d/ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java#L1529



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18773) Support multiple instances of Cleaner

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-18773:
-

Assignee: Eugene Koifman

> Support multiple instances of Cleaner
> -
>
> Key: HIVE-18773
> URL: https://issues.apache.org/jira/browse/HIVE-18773
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
>
> We support multiple Workers by making each Worker update the status of the 
> entry in COMPACTION_QUEUE to make sure only 1 worker grabs it.  Once we have 
> HIVE-18772, Cleaner should not need any state we can easily have  > 1 Cleaner 
> instance by introducing 1 more status type "being cleaned".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393381#comment-16393381
 ] 

Eugene Koifman commented on HIVE-18693:
---

[~steveyeom2017], I think the code changes are fine in general but the test is 
missing a few checks.  Left some comments in RB.

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch, HIVE-18693.02.patch, 
> HIVE-18693.03.patch, HIVE-18693.04.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14792) AvroSerde reads the remote schema-file at least once per mapper, per table reference.

[Mithun Radhakrishnan (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393380#comment-16393380
 ] 

Mithun Radhakrishnan commented on HIVE-14792:
-

Ok, this patch does not look ready to go in. :/ I was hoping these were 
transient failures, but it looks like [^HIVE-14792.3.patch] is still failing 
hard, based on the test-results above.

[^HIVE-14792.4.patch] happens to run, because the optimization is not enabled.

This is back to where it was when I ran out of time. :[ I'll update if I can 
take a crack at this.

> AvroSerde reads the remote schema-file at least once per mapper, per table 
> reference.
> -
>
> Key: HIVE-14792
> URL: https://issues.apache.org/jira/browse/HIVE-14792
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Mithun Radhakrishnan
>Assignee: Aihua Xu
>Priority: Major
>  Labels: TODOC2.2, TODOC2.4
> Fix For: 3.0.0, 2.4.0, 2.2.1
>
> Attachments: HIVE-14792.1.patch, HIVE-14792.3.patch, 
> HIVE-14792.4.patch
>
>
> Avro tables that use "external" schema files stored on HDFS can cause 
> excessive calls to {{FileSystem::open()}}, especially for queries that spawn 
> large numbers of mappers.
> This is because of the following code in {{AvroSerDe::initialize()}}:
> {code:title=AvroSerDe.java|borderStyle=solid}
> public void initialize(Configuration configuration, Properties properties) 
> throws SerDeException {
> // ...
> if (hasExternalSchema(properties)
> || columnNameProperty == null || columnNameProperty.isEmpty()
> || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
>   schema = determineSchemaOrReturnErrorSchema(configuration, properties);
> } else {
>   // Get column names and sort order
>   columnNames = Arrays.asList(columnNameProperty.split(","));
>   columnTypes = 
> TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);
>   schema = getSchemaFromCols(properties, columnNames, columnTypes, 
> columnCommentProperty);
>  
> properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(),
>  schema.toString());
> }
> // ...
> }
> {code}
> For tables using {{avro.schema.url}}, every time the SerDe is initialized 
> (i.e. at least once per mapper), the schema file is read remotely. For 
> queries with thousands of mappers, this leads to a stampede to the handful 
> (3?) datanodes that host the schema-file. In the best case, this causes 
> slowdowns.
> It would be preferable to distribute the Avro-schema to all mappers as part 
> of the job-conf. The alternatives aren't exactly appealing:
> # One can't rely solely on the {{column.list.types}} stored in the Hive 
> metastore. (HIVE-14789).
> # {{avro.schema.literal}} might not always be usable, because of the 
> size-limit on table-parameters. The typical size of the Avro-schema file is 
> between 0.5-3MB, in my limited experience. Bumping the max table-parameter 
> size isn't a great solution.
> If the {{avro.schema.file}} were read during query-planning, and made 
> available as part of table-properties (but not serialized into the 
> metastore), the downstream logic will remain largely intact. I have a patch 
> that does this.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393369#comment-16393369
 ] 

Hive QA commented on HIVE-17178:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913541/HIVE-17178.6.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 13351 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Updated] (HIVE-18875) Enable SMB Join by default in Tez

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18875:
--
Attachment: HIVE-18875.2.patch

> Enable SMB Join by default in Tez
> -
>
> Key: HIVE-18875
> URL: https://issues.apache.org/jira/browse/HIVE-18875
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18875.1.patch, HIVE-18875.2.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18877) HiveSchemaTool.validateSchemaTables() should wrap a SQLException when rethrowing

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-18877:
--
Attachment: HIVE-18877.3.patch

> HiveSchemaTool.validateSchemaTables() should wrap a SQLException when 
> rethrowing
> 
>
> Key: HIVE-18877
> URL: https://issues.apache.org/jira/browse/HIVE-18877
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>Priority: Major
> Attachments: HIVE-18877.1.patch, HIVE-18877.2.patch, 
> HIVE-18877.3.patch
>
>
> If schematool is run with the -verbose flag then it will print a stack trace 
> for an exception that occurs. If a SQLException is caught during 
> HiveSchemaTool.validateSchemaTables() then a HiveMetaException is rethrown 
> containing the text of the SQLException. If we instead throw  a 
> HiveMetaException that wraps the SQLException then the stacktrace will help 
> with diagnosis of issues where the SQLException contains a generic error 
> text. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18922) Hive is not cleaning up staging directories

[Anant Mittal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anant Mittal reassigned HIVE-18922:
---

Assignee: Sergio Peña

> Hive is not cleaning up  staging directories
> 
>
> Key: HIVE-18922
> URL: https://issues.apache.org/jira/browse/HIVE-18922
> Project: Hive
>  Issue Type: Bug
>Reporter: Anant Mittal
>Assignee: Sergio Peña
>Priority: Major
>
> Hive is creating hdfs folders with format 
> /.hive-staging_hive__-xx/-ext-x
> These are not being cleaned up even after long duration. The folder is used 
> to load to the table. Example:
> Loading data to table default.tablename from 
> hdfs://clustermachine/apps/hive/warehouse/tablename/.hive-staging_hive_2018-01-31_11-45-14_005_1129336997995057804-51/-ext-1
>  
> This might be covered to some extent by HIVE-11940 but, want to make sure all 
> cases are addressed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393291#comment-16393291
 ] 

Hive QA commented on HIVE-17178:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
39s{color} | {color:red} ql: The patch generated 3 new + 69 unchanged - 3 fixed 
= 72 total (was 72) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 39s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9575/dev-support/hive-personality.sh
 |
| git revision | master / 34eb9ed |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9575/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9575/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9575/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Spark Partition Pruning Sink Operator can't target multiple Works
> -
>
> Key: HIVE-17178
> URL: https://issues.apache.org/jira/browse/HIVE-17178
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17178.1.patch, HIVE-17178.2.patch, 
> HIVE-17178.3.patch, HIVE-17178.4.patch, HIVE-17178.5.patch, HIVE-17178.6.patch
>
>
> A Spark Partition Pruning Sink Operator cannot be used to target multiple Map 
> Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated 
> if a single table needs to be used to target multiple Map Works.
> The following query shows the issue:
> {code}
> set hive.spark.dynamic.partition.pruning=true;
> set hive.auto.convert.join=true;
> create table part_table_1 (col int) partitioned by (part_col int);
> create table part_table_2 (col int) partitioned by (part_col int);
> create table regular_table (col int);
> insert into table regular_table values (1);
> alter table part_table_1 add partition (part_col=1);
> insert into table part_table_1 partition (part_col=1) values (1), (2), (3), 
> (4);
> alter table part_table_1 add partition 

[jira] [Commented] (HIVE-15218) Kyro Exception on subsequent run of a query in LLAP mode

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393249#comment-16393249
 ] 

Hive QA commented on HIVE-15218:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12839478/HIVE-15218.branch-1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/9574/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/9574/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-9574/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-03-09 17:36:59.309
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-9574/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-03-09 17:36:59.312
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   73ccc44..34eb9ed  master -> origin/master
+ git reset --hard HEAD
HEAD is now at 73ccc44 HIVE-18889 : update all parts of Hive to use the same 
Guava version (Sergey Shelukhin, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 34eb9ed HIVE-18918 - Bad error message in 
CompactorMR.lanuchCompactionJob() (Eugene Koifman, reviewed by Jason Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-03-09 17:37:00.900
+ rm -rf ../yetus_PreCommit-HIVE-Build-9574
+ mkdir ../yetus_PreCommit-HIVE-Build-9574
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-9574
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-9574/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not 
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java: does not exist in 
index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java: does not 
exist in index
error: patch failed: 
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:1167
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' 
with conflicts.
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:291
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java' with 
conflicts.
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:1266
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java' 
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed: 
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:1167
Falling back to three-way merge...
Applied patch to 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java' 
with conflicts.
error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/Driver.java:291
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/Driver.java' with 
conflicts.
error: patch failed: 
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:1266
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java' 
with conflicts.
U common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
U ql/src/java/org/apache/hadoop/hive/ql/Driver.java
U ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 

[jira] [Commented] (HIVE-18093) Improve logging when HoS application is killed

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393243#comment-16393243
 ] 

Hive QA commented on HIVE-18093:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913696/HIVE-18093.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 13350 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Assigned] (HIVE-13375) Cannot get the right count(*) through 'group by'

[Teddy Choi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Teddy Choi reassigned HIVE-13375:
-

Assignee: Teddy Choi

> Cannot get the right count(*) through 'group by'
> 
>
> Key: HIVE-13375
> URL: https://issues.apache.org/jira/browse/HIVE-13375
> Project: Hive
>  Issue Type: Bug
>  Components: Beeline
>Affects Versions: 1.1.0
> Environment: hive 1.1 under cloudera 5.4.0
>Reporter: Harper Koo
>Assignee: Teddy Choi
>Priority: Major
> Attachments: explain.txt
>
>
> I have a query looks like:
> select
> b.rd
> ,b.type
> ,null as pv
> ,count(*) as uv
> from(
> select rd
>,type
>,user
> from (select rd
>  ,case when url like '%abc' then 'A'
>when url like '%abc1' then 'B'
>when url like '%abc2' then 'C'
>   end as type
>  ,nvl(id,guid) as user
>from tableA pq
>where rd='2016-03-08'
>and url like '%abc%'
>   )t
> group by rd,type,user 
> order by rd
> )b
> group by b.rd,b.type;
> I want to get the unique 'user' from tableA without using count(distinct()), 
> but the result surprised me!
> the result looks like:
> 2016-03-08 A NULL 245
> 2016-03-08 B NULL 53
> 2016-03-08 C NULL 6
> 2016-03-08 A NULL 543
> 2016-03-08 B NULL 456
> 2016-03-08 C NULL 25
> 2016-03-08 A NULL 544
> 2016-03-08 B NULL 65
> 2016-03-08 C NULL 34
> ……
> It is quite long and does not group by rd and type at all.
> I run the same query in impala and it works well there and gives me the right 
> answer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18693) Snapshot Isolation does not work for Micromanaged table when a insert transaction is aborted

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393189#comment-16393189
 ] 

Eugene Koifman commented on HIVE-18693:
---

[~steveyeom2017] could you update RB with latest patch

> Snapshot Isolation does not work for Micromanaged table when a insert 
> transaction is aborted
> 
>
> Key: HIVE-18693
> URL: https://issues.apache.org/jira/browse/HIVE-18693
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Steve Yeom
>Assignee: Steve Yeom
>Priority: Major
> Attachments: HIVE-18693.01.patch, HIVE-18693.02.patch, 
> HIVE-18693.03.patch, HIVE-18693.04.patch
>
>
> TestTxnCommands2#writeBetweenWorkerAndCleaner with minor 
> changes (changing delete command to insert command) fails on MM table.
> Specifically the last SELECT commands returns wrong results. 
> But this test works fine with full ACID table. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18918) Bad error message in CompactorMR.lanuchCompactionJob()

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18918:
--
  Resolution: Fixed
   Fix Version/s: 3.0.0
Target Version/s: 3.0.0
  Status: Resolved  (was: Patch Available)

committed to master
thanks Jason for the review

> Bad error message in CompactorMR.lanuchCompactionJob()
> --
>
> Key: HIVE-18918
> URL: https://issues.apache.org/jira/browse/HIVE-18918
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.2.2
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18918.01.patch
>
>
> {noformat}
>   rj.waitForCompletion();
>   if (!rj.isSuccessful()) {
> throw new IOException(compactionType == CompactionType.MAJOR ? 
> "Major" : "Minor" +
>" compactor job failed for " + jobName + "! Hadoop JobId: " + 
> rj.getID());
>   }
> {noformat}
> produces no useful info in case of Major compaction
> {noformat}
> 2018-02-28 00:59:16,416 ERROR [gdpr1-61]: compactor.Worker 
> (Worker.java:run(191)) - Caught exception while trying to compact 
> id:38602,dbname:audit,tableName:COMP_ENTRY_AF_A,partN\
> ame:partition_dt=2017-04-11,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0.
>   Marking failed to avoid repeated failures, java.io.IOException: Ma\
> jor
> at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.launchCompactionJob(CompactorMR.java:314)
> at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:269)
> at 
> org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.hive.ql.txn.compactor.Worker.run(Worker.java:172)
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18723) CompactorOutputCommitter.commitJob() - check rename() ret val

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393134#comment-16393134
 ] 

Eugene Koifman commented on HIVE-18723:
---

[~vbeshka], I don't understand the logic.  If B exists, rename will create B/A. 
 The delete you added will delete A (in B/A) so B will not have the results of 
the compaction.

> CompactorOutputCommitter.commitJob() - check rename() ret val
> -
>
> Key: HIVE-18723
> URL: https://issues.apache.org/jira/browse/HIVE-18723
> Project: Hive
>  Issue Type: Improvement
>  Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Kryvenko Igor
>Priority: Major
> Attachments: HIVE-18723.1.patch, HIVE-18723.2.patch, 
> HIVE-18723.3.patch, HIVE-18723.patch
>
>
> right now ret val is ignored {{fs.rename(fileStatus.getPath(), newPath); }}
> Should this use {{FileUtils.ename(FileSystem fs, Path sourcePath, Path 
> destPath, Configuration conf) }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18093) Improve logging when HoS application is killed

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393125#comment-16393125
 ] 

Hive QA commented on HIVE-18093:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}  8m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9573/dev-support/hive-personality.sh
 |
| git revision | master / 73ccc44 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9573/yetus/patch-asflicense-problems.txt
 |
| modules | C: spark-client U: spark-client |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9573/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Improve logging when HoS application is killed
> --
>
> Key: HIVE-18093
> URL: https://issues.apache.org/jira/browse/HIVE-18093
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18093.1.patch, HIVE-18093.2.patch
>
>
> When a HoS jobs is explicitly killed via a user (via a yarn command), the 
> logs just say "RPC channel closed"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18918) Bad error message in CompactorMR.lanuchCompactionJob()

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393071#comment-16393071
 ] 

Hive QA commented on HIVE-18918:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913684/HIVE-18918.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 100 failed/errored test(s), 13351 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18901) Lower ResourceDownloader Logging to Debug

[BELUGA BEHR (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392977#comment-16392977
 ] 

BELUGA BEHR commented on HIVE-18901:


[~stakiar]

The following code is the primary method for adding resources to a job.  It 
should be the primary source of logging information and it already includes 
logging which provides a list of resources that were successfully downloaded on 
line 1474.  It also logs any error that occurred during the process.  This 
information is more helpful that "I'm about to do something."  The admin only 
needs to know that it successfully did something or that it failed to do 
something. The steps required to do that something, especially buried down in a 
private method of a utility class from a different package, should be 'debug' 
level logging. 

https://github.com/apache/hive/blob/111ed096496c37674601173cfeaa40cbf948f878/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L1423-L1477

We also have this somewhat duplicate logging as well:

{code}
converting to local hdfs:///user/hive/hiveauxjars/simplejava.jar
converting to local hdfs:///user/hive/hiveauxjars/sample_udf.jar
Added [/tmp/2a023cae-1e02-4cce-ade6-35d04f8f4f3d_resources/simplejava.jar, 
/tmp/2a023cae-1e02-4cce-ade6-35d04f8f4f3d_resources/sample_udf.jar] to class 
path
{code}

https://github.com/apache/hive/blob/111ed096496c37674601173cfeaa40cbf948f878/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java#L1328

{code}
INFO : converting to local hdfs://nameservice1/user/hive/udf/exhibit.jar 
INFO : Added [/tmp/6b16398c-854b-40a5-8439-b16d123e8f1e_resources/exhibit.jar] 
to class path 
INFO : Added resources: [hdfs://nameservice1/user/hive/udf/exhibit.jar] 
{code}

The first line of "converting to local..." is just not that helpful/useful here.

Thanks!

> Lower ResourceDownloader Logging to Debug
> -
>
> Key: HIVE-18901
> URL: https://issues.apache.org/jira/browse/HIVE-18901
> Project: Hive
>  Issue Type: Improvement
>  Components: HiveServer2
>Affects Versions: 3.0.0, 2.4.0
>Reporter: BELUGA BEHR
>Assignee: Kryvenko Igor
>Priority: Trivial
>  Labels: noob
> Attachments: HIVE-18901.patch
>
>
> https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/util/ResourceDownloader.java#L98
> {code}
>   LOG.info("converting to local {}", srcUri);
> {code}
> # Set to _debug_ level
> # Uppercase the "C" to "Converting" to match all other Hive logging



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18918) Bad error message in CompactorMR.lanuchCompactionJob()

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392981#comment-16392981
 ] 

Hive QA commented on HIVE-18918:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9572/dev-support/hive-personality.sh
 |
| git revision | master / 73ccc44 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9572/yetus/patch-asflicense-problems.txt
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9572/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Bad error message in CompactorMR.lanuchCompactionJob()
> --
>
> Key: HIVE-18918
> URL: https://issues.apache.org/jira/browse/HIVE-18918
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 1.2.2
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-18918.01.patch
>
>
> {noformat}
>   rj.waitForCompletion();
>   if (!rj.isSuccessful()) {
> throw new IOException(compactionType == CompactionType.MAJOR ? 
> "Major" : "Minor" +
>" compactor job failed for " + jobName + "! Hadoop JobId: " + 
> rj.getID());
>   }
> {noformat}
> produces no useful info in case of Major compaction
> {noformat}
> 2018-02-28 00:59:16,416 ERROR [gdpr1-61]: compactor.Worker 
> (Worker.java:run(191)) - Caught exception while trying to compact 
> id:38602,dbname:audit,tableName:COMP_ENTRY_AF_A,partN\
> ame:partition_dt=2017-04-11,state:^@,type:MAJOR,properties:null,runAs:null,tooManyAborts:false,highestTxnId:0.
>   Marking failed to avoid repeated failures, java.io.IOException: Ma\
> jor
> at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.launchCompactionJob(CompactorMR.java:314)
> at 
> org.apache.hadoop.hive.ql.txn.compactor.CompactorMR.run(CompactorMR.java:269)
> at 
> org.apache.hadoop.hive.ql.txn.compactor.Worker$1.run(Worker.java:175)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at 

[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18862:

Status: Patch Available  (was: Open)

> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch
>
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18862:

Attachment: HIVE-18862.01.patch

> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
> Attachments: HIVE-18862.01.patch
>
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392947#comment-16392947
 ] 

Hive QA commented on HIVE-18831:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913680/HIVE-18831.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 12957 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Updated] (HIVE-18926) Imporve operator-tree matching

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-18926:

Status: Patch Available  (was: Open)

> Imporve operator-tree matching
> --
>
> Key: HIVE-18926
> URL: https://issues.apache.org/jira/browse/HIVE-18926
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18926.01.patch
>
>
> currently joins are not matched



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18926) Imporve operator-tree matching

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich updated HIVE-18926:

Attachment: HIVE-18926.01.patch

> Imporve operator-tree matching
> --
>
> Key: HIVE-18926
> URL: https://issues.apache.org/jira/browse/HIVE-18926
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-18926.01.patch
>
>
> currently joins are not matched



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-18926) Imporve operator-tree matching

[Zoltan Haindrich (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltan Haindrich reassigned HIVE-18926:
---


> Imporve operator-tree matching
> --
>
> Key: HIVE-18926
> URL: https://issues.apache.org/jira/browse/HIVE-18926
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
>
> currently joins are not matched



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392904#comment-16392904
 ] 

Hive QA commented on HIVE-18831:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 1 new + 44 unchanged - 4 fixed 
= 45 total (was 48) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 45s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9571/dev-support/hive-personality.sh
 |
| git revision | master / 73ccc44 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9571/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9571/yetus/patch-asflicense-problems.txt
 |
| modules | C: itests ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9571/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Differentiate errors that are thrown by Spark tasks
> ---
>
> Key: HIVE-18831
> URL: https://issues.apache.org/jira/browse/HIVE-18831
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18831.1.patch, HIVE-18831.2.patch, 
> HIVE-18831.3.patch
>
>
> We propagate exceptions from Spark task failures to the client well, but we 
> don't differentiate between errors from HS2 / RSC vs. errors thrown by 
> individual tasks.
> Main motivation is that when the client sees a propagated Spark exception its 
> difficult to know what part of the excution threw the exception.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18907) Create utility to fix acid key index issue from HIVE-18817

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392890#comment-16392890
 ] 

Hive QA commented on HIVE-18907:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913674/HIVE-18907.3.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 12957 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-18907) Create utility to fix acid key index issue from HIVE-18817

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392856#comment-16392856
 ] 

Hive QA commented on HIVE-18907:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
 6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
6s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
34s{color} | {color:red} root: The patch generated 28 new + 0 unchanged - 0 
fixed = 28 total (was 0) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
32s{color} | {color:red} ql: The patch generated 28 new + 0 unchanged - 0 fixed 
= 28 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9570/dev-support/hive-personality.sh
 |
| git revision | master / 73ccc44 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9570/yetus/diff-checkstyle-root.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9570/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9570/yetus/patch-asflicense-problems.txt
 |
| modules | C: . ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9570/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Create utility to fix acid key index issue from HIVE-18817
> --
>
> Key: HIVE-18907
> URL: https://issues.apache.org/jira/browse/HIVE-18907
> Project: Hive
>  Issue Type: Bug
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18907.1.patch, HIVE-18907.2.patch, 
> HIVE-18907.3.patch
>
>
> While HIVE-18817 will create new ORC Acid files from hitting the 
> ArrayIndexOutOfBounds issue, existing files created before HIVE-18817 will 
> still cause this issue. If there are delta directories then one way to 
> generate new files is to perform a major compaction. But this does not work 
> if there are no delta directories for the table/partition.
> Add a tool to fix the Acid ORC files directly in the case that a compaction 
> cannot be performed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18862) qfiles: prepare .q files for using datasets

[Laszlo Bodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laszlo Bodor updated HIVE-18862:

Summary: qfiles: prepare .q files for using datasets  (was: qfiles: prepare 
.q files for using datasets - clientpositive)

> qfiles: prepare .q files for using datasets
> ---
>
> Key: HIVE-18862
> URL: https://issues.apache.org/jira/browse/HIVE-18862
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Laszlo Bodor
>Assignee: Laszlo Bodor
>Priority: Major
>
> # Parse .q files for source table usage
>  # Add needed dataset annotations
>  # Remove setInitScript("q_test_init.sql"); from CliConfig



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone

[Piotr Findeisen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Findeisen updated HIVE-18925:
---
Description: 
Hive Server2 doesn't  work if started with 
{{-Duser.timezone=America/Bahia_Banderas}}

 

Steps to reproduce
 # use [https://github.com/big-data-europe/docker-hive]
 # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to 
{{hive-server}} docker container environment configuration
 # {{docker-compose up}}
 # 
{code:java}
host# docker-compose exec hive-server bash
container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1 --verbose=true
...
jdbc:hive2://localhost:1> select 1;{code}

The above fails and prints
{noformat}
Error: java.lang.IllegalStateException: Can't overwrite cause with 
org.joda.time.IllegalInstantException: Illegal instant due to time zone offset 
transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
(America/Bahia_Banderas) (state=08S01,code=0)
java.sql.SQLException: java.lang.IllegalStateException: Can't overwrite cause 
with org.joda.time.IllegalInstantException: Illegal instant due to time zone 
offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
(America/Bahia_Banderas)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:323)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:253)
at org.apache.hive.beeline.Commands.executeInternal(Commands.java:997)
at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalStateException: Can't overwrite cause with 
org.joda.time.IllegalInstantException: Illegal instant due to time zone offset 
transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
(America/Bahia_Banderas)
at java.lang.Throwable.initCause(Throwable.java:457)
at 
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
at 
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:237)
at 
org.apache.hive.service.cli.HiveSQLException.toCause(HiveSQLException.java:198)
at 
org.apache.hive.service.cli.HiveSQLException.(HiveSQLException.java:108)
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
at org.apache.hive.jdbc.HiveStatement.runAsyncOnServer(HiveStatement.java:313)
... 15 more
Caused by: java.lang.ExceptionInInitializerError: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at 
org.apache.hive.service.cli.HiveSQLException.newInstance(HiveSQLException.java:245)
at 
org.apache.hive.service.cli.HiveSQLException.toStackTrace(HiveSQLException.java:211)
... 21 more{noformat}
>From the above stacktrace it's not visible what is the cause, but i think it's 
>initialization of 
>{{org.apache.hive.common.util.TimestampParser#startingDateValue}}

 

  was:
Hive Server2 doesn't  work if started with 
{{-Duser.timezone=America/Bahia_Banderas}}

 

Steps to reproduce
 # use [https://github.com/big-data-europe/docker-hive]
 # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to 
{{hive-server}} docker container environment configuration
 # {{docker-compose up}}
 # 
{code:java}
host# docker-compose exec hive-server bash
container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1
...
jdbc:hive2://localhost:1> select 1;
Error: java.lang.IllegalStateException: Can't overwrite cause with 
org.joda.time.IllegalInstantException: Illegal instant due to time zone offset 
transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
(America/Bahia_Banderas) (state=08S01,code=0){code}


> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> 

[jira] [Updated] (HIVE-18925) Hive doesn't work when JVM is America/Bahia_Banderas time zone

[Piotr Findeisen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Piotr Findeisen updated HIVE-18925:
---
Summary: Hive doesn't work when JVM is America/Bahia_Banderas time zone  
(was: Hive doesn't start when JVM is America/Bahia_Banderas time zone)

> Hive doesn't work when JVM is America/Bahia_Banderas time zone
> --
>
> Key: HIVE-18925
> URL: https://issues.apache.org/jira/browse/HIVE-18925
> Project: Hive
>  Issue Type: Bug
> Environment: JVM in America/Bahia_Banderas zone
>Reporter: Piotr Findeisen
>Priority: Major
>
> Hive Server2 doesn't  work if started with 
> {{-Duser.timezone=America/Bahia_Banderas}}
>  
> Steps to reproduce
>  # use [https://github.com/big-data-europe/docker-hive]
>  # Add {{HADOOP_CLIENT_OPTS: '-Duser.timezone=America/Bahia_Banderas'}} to 
> {{hive-server}} docker container environment configuration
>  # {{docker-compose up}}
>  # 
> {code:java}
> host# docker-compose exec hive-server bash
> container# /opt/hive/bin/beeline -u jdbc:hive2://localhost:1
> ...
> jdbc:hive2://localhost:1> select 1;
> Error: java.lang.IllegalStateException: Can't overwrite cause with 
> org.joda.time.IllegalInstantException: Illegal instant due to time zone 
> offset transition (daylight savings time 'gap'): 1970-01-01T00:00:00.000 
> (America/Bahia_Banderas) (state=08S01,code=0){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392780#comment-16392780
 ] 

Hive QA commented on HIVE-18910:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913672/HIVE-18910.2.patch

{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 721 failed/errored test(s), 13350 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)

[jira] [Commented] (HIVE-18861) druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating classpath problems on hadoop 3.x

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392751#comment-16392751
 ] 

Steve Loughran commented on HIVE-18861:
---

thx for your help nurturing this in.

> druid-hdfs-storage is pulling in hadoop-aws-2.7.x and aws SDK, creating 
> classpath problems on hadoop 3.x
> 
>
> Key: HIVE-18861
> URL: https://issues.apache.org/jira/browse/HIVE-18861
> Project: Hive
>  Issue Type: Sub-task
>  Components: Druid integration
>Affects Versions: 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18861-001.patch, HIVE-18861-001.patch, 
> HIVE-18861.patch, HIVE-18861.patch
>
>
> druid-hdfs-storage JAR is transitively pulling in hadoop-aws JAR 2.7.3, which 
> creates classpath problems as a set of aws-sdk 1.10.77 JARs get on the CP, 
> even with Hadoop 3 & its move to a full aws-sdk-bundle JAR.
> Two options
> # exclude the dependency
> # force it up to whatever ${hadoop.version} is, so make it consistent



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392733#comment-16392733
 ] 

Hive QA commented on HIVE-18910:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
33s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
15s{color} | {color:red} streaming in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
40s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} storage-api: The patch generated 3 new + 88 unchanged 
- 3 fixed = 91 total (was 91) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
13s{color} | {color:red} serde: The patch generated 5 new + 215 unchanged - 2 
fixed = 220 total (was 217) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
38s{color} | {color:red} ql: The patch generated 13 new + 604 unchanged - 3 
fixed = 617 total (was 607) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
13s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 30s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9569/dev-support/hive-personality.sh
 |
| git revision | master / 73ccc44 |
| Default Java | 1.8.0_111 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9569/yetus/patch-mvninstall-hcatalog_streaming.txt
 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9569/yetus/patch-mvninstall-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9569/yetus/diff-checkstyle-storage-api.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9569/yetus/diff-checkstyle-serde.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9569/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9569/yetus/patch-asflicense-problems.txt
 |
| modules | C: storage-api serde hcatalog/streaming ql standalone-metastore U: 
. |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9569/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Migrate to Murmur hash for shuffle and bucketing
> 
>
> Key: HIVE-18910
> URL: https://issues.apache.org/jira/browse/HIVE-18910
> Project: Hive
>  Issue Type: Task
>Reporter: Deepak Jaiswal
>

[jira] [Commented] (HIVE-18917) Add spark.home to hive.conf.restricted.list

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392704#comment-16392704
 ] 

Hive QA commented on HIVE-18917:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913670/HIVE-18917.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 99 failed/errored test(s), 13351 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=94)

[jira] [Commented] (HIVE-15218) Kyro Exception on subsequent run of a query in LLAP mode

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392677#comment-16392677
 ] 

Prasanth Jayachandran commented on HIVE-15218:
--

Initially thought this was the issue but later figured out that this only 
happened when there was a stranded llap daemon which was not killed by yarn. 
After proper cleanup of all llap daemons this issue never occured.

> Kyro Exception on subsequent run of a query in LLAP mode
> 
>
> Key: HIVE-15218
> URL: https://issues.apache.org/jira/browse/HIVE-15218
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-15218.1.patch, HIVE-15218.2.patch, 
> HIVE-15218.branch-1.patch
>
>
> Following exception is observed when running TPCDS query19 during concurrency 
> test
> {code}
> Vertex failed, vertexName=Map 3, vertexId=vertex_1477340478603_0610_9_05, 
> diagnostics=[Task failed, taskId=task_1477340478603_0610_9_05_06, 
> diagnostics=[TaskAttempt 0 killed, TaskAttempt 1 failed, info=[Error: Error 
> while running task ( failure ) : 
> attempt_1477340478603_0610_9_05_06_1:java.lang.RuntimeException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Failed to load plan: 
> hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/ndembla/0559ce24-663e-482a-a0ea-106d220b53be/hi...
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
>   at 
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
>   at 
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
>   at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
>   at 
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Failed to load plan: 
> hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/ndembla/0559ce24-663e-482a-a0ea-106d220b53be/hi...
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:57)
>   at 
> org.apache.hadoop.hive.ql.exec.ObjectCacheWrapper.retrieve(ObjectCacheWrapper.java:40)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:129)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
>   ... 15 more
> Caused by: java.lang.RuntimeException: Failed to load plan: 
> hdfs://cn105-10.l42scl.hortonworks.com:8020/tmp/hive/ndembla/0559ce24-663e-482a-a0ea-106d220b53be/hi...
>   at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:469)
>   at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:305)
>   at 
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor$1.call(MapRecordProcessor.java:132)
>   at 
> org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:55)
>   ... 18 more
> Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: 
> Encountered unregistered class ID: 63
> Serialization trace:
> statistics (org.apache.hadoop.hive.ql.plan.TableScanDesc)
> conf (org.apache.hadoop.hive.ql.exec.TableScanOperator)
> aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
>   at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
>   at 
> org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:182)
>   at 
> org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:118)
>   at 
> 

[jira] [Commented] (HIVE-18917) Add spark.home to hive.conf.restricted.list

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392660#comment-16392660
 ] 

Hive QA commented on HIVE-18917:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
 6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
12s{color} | {color:red} The patch generated 49 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 41s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-9568/dev-support/hive-personality.sh
 |
| git revision | master / 73ccc44 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9568/yetus/patch-asflicense-problems.txt
 |
| modules | C: common itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-9568/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Add spark.home to hive.conf.restricted.list
> ---
>
> Key: HIVE-18917
> URL: https://issues.apache.org/jira/browse/HIVE-18917
> Project: Hive
>  Issue Type: Task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18917.1.patch
>
>
> Add spark.home to hive.conf.restricted.list so its not settable by users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[Ankit Singhal (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392656#comment-16392656
 ] 

Ankit Singhal commented on HIVE-18873:
--

bq. IIRC, storage handlers use hive.optimize.ppd.storage to push filters down 
to storage handlers. Maybe we can restrict this only when 
hive.optimize.index.filters is used so that storage handlers don't get impacted 
by HIVE-15680.
[~prasanth_j], How about skipping this check when storage handlers implement 
HiveStoragePredicateHandler? 

 AFAICS, Actually, the storage handlers were receiving the right predicate for 
respective alias while creating the input splits. So once the split is created, 
we don't need to push down filter again during recordReader as InputFormat 
should be carrying all required info through their custom InputSplit. (though 
it seems not true for HiveHBaseTableInputFormat only).
 

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-18873) Skipping predicate pushdown for MR silently at HiveInputFormat can cause storage handlers to produce erroneous result

[Ankit Singhal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankit Singhal updated HIVE-18873:
-
Attachment: HIVE-18873.patch

> Skipping predicate pushdown for MR silently at HiveInputFormat can cause 
> storage handlers to produce erroneous result
> -
>
> Key: HIVE-18873
> URL: https://issues.apache.org/jira/browse/HIVE-18873
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18873.patch
>
>
> {code:java}
> // disable filter pushdown for mapreduce when there are more than one table 
> aliases,
>     // since we don't clone jobConf per alias
>     if (mrwork != null && mrwork.getAliases() != null && 
> mrwork.getAliases().size() > 1 &&
>       jobConf.get(ConfVars.HIVE_EXECUTION_ENGINE.varname).equals("mr")) {
>       return;
>     }
> {code}
> I believe this needs to be handled at OpProcFactory so that hive doesn't 
> believe that predicate is handled by storage handler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-18034) Improving logging with HoS executors spend lots of time in GC

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392649#comment-16392649
 ] 

Hive QA commented on HIVE-18034:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12913664/HIVE-18034.6.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 73 failed/errored test(s), 12644 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=95)