[jira] [Commented] (HIVE-18214) Flaky test: TestSparkClient

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314434#comment-16314434
 ] 

Hive QA commented on HIVE-18214:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
9s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m  
9s{color} | {color:red} spark-client: The patch generated 13 new + 17 unchanged 
- 14 fixed = 30 total (was 31) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m  
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  8m  2s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a6b88d9 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8477/yetus/diff-checkstyle-spark-client.txt
 |
| modules | C: spark-client U: spark-client |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8477/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Flaky test: TestSparkClient
> ---
>
> Key: HIVE-18214
> URL: https://issues.apache.org/jira/browse/HIVE-18214
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-18214.1.patch, HIVE-18214.2.patch
>
>
> Looks like there is a race condition in {{TestSparkClient#runTest}}. The test 
> creates a {{RemoteDriver}} in memory, which creates a {{JavaSparkContext}}. A 
> new {{JavaSparkContext}} is created for each test that is run. There is a 
> race condition where the {{RemoteDriver}} isn't given enough time to 
> shutdown, so when the next test starts running it creates another 
> {{JavaSparkContext}} which causes an exception like 
> {{org.apache.spark.SparkException: Only one SparkContext may be running in 
> this JVM (see SPARK-2243)}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18384) ConcurrentModificationException in log4j2.x library

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314429#comment-16314429
 ] 

Hive QA commented on HIVE-18384:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904850/HIVE-18384.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 11549 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testProcessSelectDatabase 
(batchId=184)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testQuit (batchId=184)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testRun (batchId=184)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testThatCliDriverPrintsHeaderForCommandsWithSchema
 (batchId=184)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testThatCliDriverPrintsNoHeaderForCommandsWithNoSchema
 (batchId=184)
org.apache.hadoop.hive.cli.TestCliDriverMethods.testprocessInitFiles 
(batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.conf.TestHiveAsyncLogging.testAsyncLoggingInitialization 
(batchId=255)
org.apache.hadoop.hive.conf.TestHiveLogging.testHiveLogging (batchId=255)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan1 (batchId=278)
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapPlan2 (batchId=278)
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan1 (batchId=278)
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan2 (batchId=278)
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan3 (batchId=278)
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan4 (batchId=278)
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan5 (batchId=278)
org.apache.hadoop.hive.ql.exec.TestExecDriver.testMapRedPlan6 (batchId=278)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking6 
(batchId=290)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.hcatalog.cli.TestPermsGrp.testCustomPerms (batchId=192)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8476/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8476/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8476/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904850 - PreCommit-HIVE-Build

> ConcurrentModificationException in log4j2.x library
> ---
>
> Key: HIVE-18384
> URL: https://issues.apache.org/jira/browse/HIVE-18384
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18384.1.patch, HIVE-18384.2.patch
>
>
> In one of the internal testing, observed the following exception
> {code}
> java.util.ConcurrentModificationException
>   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) 
> ~[?:1.8.0_152]
>   at 

[jira] [Commented] (HIVE-18384) ConcurrentModificationException in log4j2.x library

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314416#comment-16314416
 ] 

Hive QA commented on HIVE-18384:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m  
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
11s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 39s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  xml  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a6b88d9 |
| Default Java | 1.8.0_111 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8476/yetus/patch-asflicense-problems.txt
 |
| modules | C: . testutils/ptest2 U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8476/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> ConcurrentModificationException in log4j2.x library
> ---
>
> Key: HIVE-18384
> URL: https://issues.apache.org/jira/browse/HIVE-18384
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18384.1.patch, HIVE-18384.2.patch
>
>
> In one of the internal testing, observed the following exception
> {code}
> java.util.ConcurrentModificationException
>   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) 
> ~[?:1.8.0_152]
>   at java.util.ArrayList$Itr.next(ArrayList.java:859) ~[?:1.8.0_152]
>   at 
> java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042) 
> ~[?:1.8.0_152]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.appendCollection(ParameterFormatter.java:596)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.appendPotentiallyRecursiveValue(ParameterFormatter.java:504)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.recursiveDeepToString(ParameterFormatter.java:429)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.formatMessage2(ParameterFormatter.java:189)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterizedMessage.formatTo(ParameterizedMessage.java:224)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterizedMessage.getFormattedMessage(ParameterizedMessage.java:200)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> 

[jira] [Commented] (HIVE-18366) Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead of hbase.table.name as the table name property

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314396#comment-16314396
 ] 

Hive QA commented on HIVE-18366:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904836/HIVE-18366.2.patch

{color:green}SUCCESS:{color} +1 due to 18 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 11536 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver
 (batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8475/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8475/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8475/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904836 - PreCommit-HIVE-Build

> Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead 
> of hbase.table.name as the table name property
> 
>
> Key: HIVE-18366
> URL: https://issues.apache.org/jira/browse/HIVE-18366
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-18366.1.patch, HIVE-18366.2.patch
>
>
> HBase 2.0 changes the table name property to 
> hbase.mapreduce.hfileoutputformat.table.name. HiveHFileOutputFormat is using 
> the new property name while HiveHBaseTableOutputFormat is not. If we create 
> the table as follows, HiveHBaseTableOutputFormat is used which still uses the 
> old property hbase.table.name.
> {noformat}
> create table hbase_table2(key int, val string) stored by 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties 
> ('hbase.columns.mapping' = ':key,cf:val') tblproperties 
> ('hbase.mapreduce.hfileoutputformat.table.name' = 
> 'positive_hbase_handler_bulk')
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18366) Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead of hbase.table.name as the table name property

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314382#comment-16314382
 ] 

Hive QA commented on HIVE-18366:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
11s{color} | {color:red} itests/util: The patch generated 1 new + 11 unchanged 
- 0 fixed = 12 total (was 11) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 22s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a6b88d9 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8475/yetus/diff-checkstyle-itests_util.txt
 |
| modules | C: hbase-handler hcatalog/webhcat/svr itests/hcatalog-unit 
itests/util U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8475/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Update HBaseSerDe to use hbase.mapreduce.hfileoutputformat.table.name instead 
> of hbase.table.name as the table name property
> 
>
> Key: HIVE-18366
> URL: https://issues.apache.org/jira/browse/HIVE-18366
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-18366.1.patch, HIVE-18366.2.patch
>
>
> HBase 2.0 changes the table name property to 
> hbase.mapreduce.hfileoutputformat.table.name. HiveHFileOutputFormat is using 
> the new property name while HiveHBaseTableOutputFormat is not. If we create 
> the table as follows, HiveHBaseTableOutputFormat is used which still uses the 
> old property hbase.table.name.
> {noformat}
> create table hbase_table2(key int, val string) stored by 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties 
> ('hbase.columns.mapping' = ':key,cf:val') tblproperties 
> ('hbase.mapreduce.hfileoutputformat.table.name' = 
> 'positive_hbase_handler_bulk')
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18052) Run p-tests on mm tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314373#comment-16314373
 ] 

Hive QA commented on HIVE-18052:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  5m 
 1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 
33s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
47s{color} | {color:red} ql: The patch generated 12 new + 1698 unchanged - 3 
fixed = 1710 total (was 1701) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  1m 
56s{color} | {color:red} root: The patch generated 12 new + 2851 unchanged - 3 
fixed = 2863 total (was 2854) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  8m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
11s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 27s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  
xml  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a6b88d9 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8474/yetus/diff-checkstyle-ql.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8474/yetus/diff-checkstyle-root.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8474/yetus/whitespace-eol.txt 
|
| modules | C: common standalone-metastore ql service hcatalog/core 
hcatalog/hcatalog-pig-adapter hcatalog/server-extensions 
hcatalog/webhcat/java-client hcatalog/streaming . itests/hcatalog-unit 
itests/hive-minikdc itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8474/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Run p-tests on mm tables
> 
>
> Key: HIVE-18052
> URL: https://issues.apache.org/jira/browse/HIVE-18052
> Project: Hive
>  Issue Type: Task
>Reporter: Steve Yeom
>Assignee: Steve Yeom
> Attachments: HIVE-18052.1.patch, HIVE-18052.10.patch, 
> HIVE-18052.11.patch, HIVE-18052.12.patch, HIVE-18052.13.patch, 
> HIVE-18052.14.patch, HIVE-18052.15.patch, HIVE-18052.16.patch, 
> HIVE-18052.17.patch, HIVE-18052.2.patch, HIVE-18052.3.patch, 
> HIVE-18052.4.patch, HIVE-18052.5.patch, HIVE-18052.6.patch, 
> 

[jira] [Commented] (HIVE-18052) Run p-tests on mm tables

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314372#comment-16314372
 ] 

Hive QA commented on HIVE-18052:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904835/HIVE-18052.17.patch

{color:green}SUCCESS:{color} +1 due to 98 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1081 failed/errored test(s), 10211 tests 
executed
*Failed tests:*
{noformat}
TestNegativeCliDriver - did not produce a TEST-*.xml file (likely timed out) 
(batchId=93)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: HIVE-18386.patch

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18386.patch
>
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: (was: HIVE-18386.patch)

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18386.patch
>
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: HIVE-18386.patch

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18386.patch
>
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: (was: HIVE-18386.patch)

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: (was: HIVE-18386.patch)

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18386.patch
>
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: HIVE-18386.patch

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18386.patch
>
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: HIVE-18386.patch

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18386.patch
>
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: (was: HIVE-18386.patch)

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Work started] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on HIVE-18386 started by Jesus Camacho Rodriguez.
--
> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18386.patch
>
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-18386:
---
Attachment: HIVE-18386.patch

> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18386.patch
>
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14615) Temp table leaves behind insert command

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314350#comment-16314350
 ] 

Hive QA commented on HIVE-14615:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904833/HIVE-14615.4.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 11550 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[basicstat_partval] 
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=77)
org.apache.hadoop.hive.cli.TestMiniDruidCliDriver.testCliDriver[druidmini_mv] 
(batchId=245)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8473/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8473/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8473/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904833 - PreCommit-HIVE-Build

> Temp table leaves behind insert command
> ---
>
> Key: HIVE-14615
> URL: https://issues.apache.org/jira/browse/HIVE-14615
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Chaoyu Tang
>Assignee: Andrew Sherman
> Attachments: HIVE-14615.1.patch, HIVE-14615.2.patch, 
> HIVE-14615.3.patch, HIVE-14615.4.patch
>
>
> {code}
> create table test (key int, value string);
> insert into test values (1, 'val1');
> show tables;
> test
> values__tmp__table__1
> {code}
> the temp table values__tmp__table__1 was resulted from insert into ...values
> and exists until logout the session.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-14615) Temp table leaves behind insert command

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-14615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314346#comment-16314346
 ] 

Hive QA commented on HIVE-14615:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a6b88d9 |
| Default Java | 1.8.0_111 |
| modules | C: ql accumulo-handler hbase-handler . itests/hive-blobstore U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8473/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Temp table leaves behind insert command
> ---
>
> Key: HIVE-14615
> URL: https://issues.apache.org/jira/browse/HIVE-14615
> Project: Hive
>  Issue Type: Bug
>  Components: Query Processor
>Reporter: Chaoyu Tang
>Assignee: Andrew Sherman
> Attachments: HIVE-14615.1.patch, HIVE-14615.2.patch, 
> HIVE-14615.3.patch, HIVE-14615.4.patch
>
>
> {code}
> create table test (key int, value string);
> insert into test values (1, 'val1');
> show tables;
> test
> values__tmp__table__1
> {code}
> the temp table values__tmp__table__1 was resulted from insert into ...values
> and exists until logout the session.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18229) add the unmanaged mapping command

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18229:

Status: Patch Available  (was: Open)

> add the unmanaged mapping command
> -
>
> Key: HIVE-18229
> URL: https://issues.apache.org/jira/browse/HIVE-18229
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18229.patch
>
>
> This is to add an option to enable WM but map queries past the default pool 
> and explicitly into unmanaged sessions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18229) add the unmanaged mapping command

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18229:

Attachment: HIVE-18229.patch

> add the unmanaged mapping command
> -
>
> Key: HIVE-18229
> URL: https://issues.apache.org/jira/browse/HIVE-18229
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18229.patch
>
>
> This is to add an option to enable WM but map queries past the default pool 
> and explicitly into unmanaged sessions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-14498) Freshness period for query rewriting using materialized views

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-14498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-14498:
---
Attachment: HIVE-14498.05.patch

> Freshness period for query rewriting using materialized views
> -
>
> Key: HIVE-14498
> URL: https://issues.apache.org/jira/browse/HIVE-14498
> Project: Hive
>  Issue Type: Sub-task
>  Components: Materialized views
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-14498.01.patch, HIVE-14498.02.patch, 
> HIVE-14498.03.patch, HIVE-14498.04.patch, HIVE-14498.05.patch, 
> HIVE-14498.patch
>
>
> Once we have query rewriting in place (HIVE-14496), one of the main issues is 
> data freshness in the materialized views.
> Since we will not support view maintenance at first, we could include a 
> HiveConf property to configure a max freshness period (_n timeunits_). If a 
> query comes, and the materialized view has been populated (by create, 
> refresh, etc.) for a longer period than _n_, then we should not use it for 
> rewriting the query.
> Optionally, we could print a warning for the user indicating that the 
> materialized was not used because it was not fresh.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18061) q.outs: be more selective with masking hdfs paths

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314319#comment-16314319
 ] 

Hive QA commented on HIVE-18061:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904798/HIVE-18061.03.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8472/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8472/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8472/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-06 02:39:02.846
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8472/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-06 02:39:02.850
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at a6b88d9 HIVE-18096 : add a user-friendly show plan command 
(Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at a6b88d9 HIVE-18096 : add a user-friendly show plan command 
(Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-06 02:39:06.364
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8472/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: patch failed: 
ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out:266
Falling back to three-way merge...
Applied patch to 
'ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out' with conflicts.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:625: trailing whitespace.
hdfs://### HDFS PATH ### 
/data/hiveptest/working/scratch/build.patch:650: trailing whitespace.
hdfs://### HDFS PATH ### 
/data/hiveptest/working/scratch/build.patch:726: trailing whitespace.
hdfs://### HDFS PATH ### 
/data/hiveptest/working/scratch/build.patch:751: trailing whitespace.
hdfs://### HDFS PATH ### 
/data/hiveptest/working/scratch/build.patch:3173: trailing whitespace.
  hdfs://### HDFS PATH ### 
error: patch failed: 
ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out:266
Falling back to three-way merge...
Applied patch to 
'ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out' with conflicts.
U ql/src/test/results/clientpositive/spark/bucketmapjoin7.q.out
warning: squelched 17 whitespace errors
warning: 22 lines add whitespace errors.
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904798 - PreCommit-HIVE-Build

> q.outs: be more selective with masking hdfs paths
> -
>
> Key: HIVE-18061
> URL: https://issues.apache.org/jira/browse/HIVE-18061
> Project: Hive
>  Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Laszlo Bodor
> Attachments: HIVE-18061.01.patch, HIVE-18061.02.patch, 
> HIVE-18061.03.patch
>
>
> currently any line which contains a path which looks like an hdfs location is 
> replaced with a "masked pattern was here"...
> it might be releavant to record these messages; since even an exception 
> message might contain an hdfs location
> noticed in
> HIVE-18012



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18161) Remove hive.stats.atomic

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314317#comment-16314317
 ] 

Hive QA commented on HIVE-18161:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904791/HIVE-18161.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11549 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
 (batchId=177)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8471/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8471/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8471/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904791 - PreCommit-HIVE-Build

> Remove hive.stats.atomic
> 
>
> Key: HIVE-18161
> URL: https://issues.apache.org/jira/browse/HIVE-18161
> Project: Hive
>  Issue Type: Sub-task
>  Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Bertalan Kondrat
> Attachments: HIVE-18161.2.patch, HIVE-18161.patch
>
>
> It might have its purpose back then... when it was introduced; but currently 
> enabling this property may at best prevent the stats collector from working 
> properly :)
> And moreover: {{hive.stats.reliable}} is a very similar prop; and it would be 
> better to use only one property which enables correctness checks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[Vihang Karajgaonkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314314#comment-16314314
 ] 

Vihang Karajgaonkar commented on HIVE-17580:


[~alangates] [~owen.omalley] and [~sershe] Can you please take a look? I can 
create multiple sub-tasks to break this request into smaller bits :) Thanks a 
lot!

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>  Labels: pull-request-available
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated HIVE-17580:
--
Labels: pull-request-available  (was: )

> Remove dependency of get_fields_with_environment_context API to serde
> -
>
> Key: HIVE-17580
> URL: https://issues.apache.org/jira/browse/HIVE-17580
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>  Labels: pull-request-available
>
> {{get_fields_with_environment_context}} metastore API uses {{Deserializer}} 
> class to access the fields metadata for the cases where it is stored along 
> with the data files (avro tables). The problem is Deserializer classes is 
> defined in hive-serde module and in order to make metastore independent of 
> Hive we will have to remove this dependency (atleast we should change it to 
> runtime dependency instead of compile time).
> The other option is investigate if we can use SearchArgument to provide this 
> functionality.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18229) add the unmanaged mapping command

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin reassigned HIVE-18229:
---

Assignee: Sergey Shelukhin

> add the unmanaged mapping command
> -
>
> Key: HIVE-18229
> URL: https://issues.apache.org/jira/browse/HIVE-18229
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> This is to add an option to enable WM but map queries past the default pool 
> and explicitly into unmanaged sessions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17580) Remove dependency of get_fields_with_environment_context API to serde

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314309#comment-16314309
 ] 

ASF GitHub Bot commented on HIVE-17580:
---

GitHub user vihangk1 opened a pull request:

https://github.com/apache/hive/pull/287

HIVE-17580 : Remove standalone-metastore's dependency with serdes

Removing the dependency on serdes for the metastore requires a series of 
changes. I have created multiple commits which hopefully would be easier to 
review. Each major commit has a descriptive commit message to give a high level 
idea of what the change is doing. There are still some bits which need to be 
completed but it would be good to a review.

Overview of all the changes done:
1. Creates a new module called serde-api under storage-api like discussed. 
Although I think we can keep it separate as well.
2. Moved List, Map, Struct, Constant, Primitive, Union ObjectInspectors to 
serde-api
3. Moved PrimitiveTypeInfo, PrimitiveTypeEntry and TypeInfo to serde-api.
4. Moved TypeInfoParser, TypeInfoFactory to serde-api
5. Added a new class which reading avro storage schema by copying the code 
from AvroSerde and AvroSerdeUtils. The parsing is done such that String value 
is first converted into TypeInfos and then into FieldSchemas bypassing the need 
for ObjectInspectors. In theory we could get rid of TypeInfos as well but that 
path was getting too difficult with lot of duplicate code between Hive and 
metastore.
6. Introduces a default storage schema reader. I noticed that most of the 
serdes use the same logic to parse the metadata information. This code should 
be refactored to a common place instead of having many copies (one in 
standalone hms and another set in multiple serdes)
7. Moved HiveChar, HiveVarchar, HiveCharWritable, HiveVarcharWritable to 
storage-api. I noticed that HiveDecimal is already in storage-api. It probably 
makes sense to move the other primitive types (timestamp, interval etc)to 
storage-api as well but it requires storage-api to be upgraded to Java 8.
8. Adds a basic test for the schema reader. I plan to add more tests as 
this code is reviewed.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vihangk1/hive vihangk1_HIVE-17580

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/287.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #287


commit bbfb7dc44904db74a840167c02b07f50a6010b69
Author: Vihang Karajgaonkar 
Date:   2017-11-09T00:52:39Z

HIVE-17580 : Remove dependency of get_fields_with_environment_context API 
to serde

commit d54879845eff10c19bc17bda9e09dda16f6fa295
Author: Vihang Karajgaonkar 
Date:   2017-11-29T00:54:23Z

Moved List, Map, Struct OI to storage-api

commit a12d6c7ba3de598c6b6f75da1bd4efcac43036b1
Author: Vihang Karajgaonkar 
Date:   2017-11-29T04:19:39Z

Moved ConstantObjectInspector PrimitiveObjectInspector and 
UnionObjectInspector

commit 13fb832fc2d51958e75d5e609f6781f87449aed8
Author: Vihang Karajgaonkar 
Date:   2017-12-28T01:25:59Z

Moved PrimitiveTypeInfo to serde-api

In order to move PrimitiveTypeInfo we need to move the PrimitiveTypeEntry 
as well.
PrimitiveTypeEntry depends on PrimitiveObjectInspectorUtils which cannot be 
pulled
into serde-api. Hence the static final maps are moved to PrimitiveEntry and 
we provide
static access methods to these maps along with the register method to add 
the key
value pairs in the maps

commit 9cbc789fd3f4ced7ce66a7313c451b75a154976f
Author: Vihang Karajgaonkar 
Date:   2017-12-28T20:51:39Z

Moved the other TypeInfos to serde-api

In order to move the other TypeInfo classes to serde-api we need to move 
the serdeConstants.java
as well. This is a thrift generated class. This commit copies the 
serde.thrift instead of moving.
The only reason I did not move it is in case of backwards compatibility 
reasons (in case someone is using the thrift file location to do something).
If it is okay to move serde.thrift from serde module to serde-api we can 
delete it from serde module in a separate
change.

The other concern is there are some TypeInfo classes which do some 
validation like VarCharTypeInfo, DecimalTypeInfo.
The validating methods use the actual type implementation like HiveChar, 
HiveDecimal etc to ensure that the params
are under the correct limits. This creates a problem since we cannot bring 
in the type implementations as well to
serde-api. Currently, I have marked these as TODO and commented them out.

commit 23aa899a90d17648e560d39602a3bd29bf53661e
Author: Vihang Karajgaonkar 
Date:   

[jira] [Updated] (HIVE-18389) Print out Spark Web UI URL to the console log

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18389:

Status: Patch Available  (was: Open)

> Print out Spark Web UI URL to the console log
> -
>
> Key: HIVE-18389
> URL: https://issues.apache.org/jira/browse/HIVE-18389
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-18389.1.patch
>
>
> Should be accessible via {{SparkContext#uiWebUrl}}. It just needs to be sent 
> from the {{RemoteDriver}} to HS2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18389) Print out Spark Web UI URL to the console log

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18389:

Attachment: HIVE-18389.1.patch

> Print out Spark Web UI URL to the console log
> -
>
> Key: HIVE-18389
> URL: https://issues.apache.org/jira/browse/HIVE-18389
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-18389.1.patch
>
>
> Should be accessible via {{SparkContext#uiWebUrl}}. It just needs to be sent 
> from the {{RemoteDriver}} to HS2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18161) Remove hive.stats.atomic

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314280#comment-16314280
 ] 

Hive QA commented on HIVE-18161:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
32s{color} | {color:red} ql: The patch generated 1 new + 72 unchanged - 1 fixed 
= 73 total (was 73) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 15s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a6b88d9 |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8471/yetus/diff-checkstyle-ql.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8471/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Remove hive.stats.atomic
> 
>
> Key: HIVE-18161
> URL: https://issues.apache.org/jira/browse/HIVE-18161
> Project: Hive
>  Issue Type: Sub-task
>  Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Bertalan Kondrat
> Attachments: HIVE-18161.2.patch, HIVE-18161.patch
>
>
> It might have its purpose back then... when it was introduced; but currently 
> enabling this property may at best prevent the stats collector from working 
> properly :)
> And moreover: {{hive.stats.reliable}} is a very similar prop; and it would be 
> better to use only one property which enables correctness checks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18372) Create testing infra to test different HMS instances

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314265#comment-16314265
 ] 

Hive QA commented on HIVE-18372:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904771/HIVE-18372.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8470/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8470/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8470/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-06 01:23:16.875
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8470/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-06 01:23:16.878
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
   a6435cc..a6b88d9  master -> origin/master
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
+ git reset --hard HEAD
HEAD is now at a6435cc HIVE-18275 : add HS2-level WM metrics (Sergey Shelukhin, 
reviewed by Thejas M Nair)
+ git clean -f -d
Removing ${project.basedir}/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
  (use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at a6b88d9 HIVE-18096 : add a user-friendly show plan command 
(Harish Jaiprakash, reviewed by Sergey Shelukhin)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-06 01:23:21.190
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
cp: cannot stat ?./.git/gc.pid?: No such file or directory
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904771 - PreCommit-HIVE-Build

> Create testing infra to test different HMS instances
> 
>
> Key: HIVE-18372
> URL: https://issues.apache.org/jira/browse/HIVE-18372
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Peter Vary
>Assignee: Peter Vary
> Attachments: HIVE-18372.patch
>
>
> Since there will be multiple tests, it would be good to have a good 
> infrastructure to help creating those faster, easier.
> This patch will also include the test cases for the Database related methods 
> to showcase the infra



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18352) introduce a METADATAONLY option while doing REPL DUMP to allow integrations of other tools

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314263#comment-16314263
 ] 

Hive QA commented on HIVE-18352:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904760/HIVE-18352.1.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11549 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_12_nonnative_export]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hadoop.hive.ql.parse.TestReplicationSemanticAnalyzer.testReplDumpParse
 (batchId=270)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8469/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8469/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8469/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904760 - PreCommit-HIVE-Build

> introduce a METADATAONLY option while doing REPL DUMP to allow integrations 
> of other tools 
> ---
>
> Key: HIVE-18352
> URL: https://issues.apache.org/jira/browse/HIVE-18352
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18352.0.patch, HIVE-18352.1.patch
>
>
> * Introduce a METADATAONLY option as part of the REPL DUMP command which will 
> only try and dump out events for DDL changes, this will be faster as we wont 
> need  scan of files on HDFS for DML changes. 
> * Additionally since we are only going to dump metadata operations, it might 
> be useful to include acid tables as well via an option as well. This option 
> can be removed when ACID support is complete via HIVE-18320
> it will be good to support the "WITH" clause as part of REPL DUMP command as 
> well (repl dump already supports it viaHIVE-17757) to achieve the above as 
> that will prevent less changes to the syntax of the statement and provide 
> more flexibility in future to include additional options as well. 
> {code}
> REPL DUMP [db_name] {FROM [event_id]} {TO [event_id]} {WITH 
> (['key'='value'],.)}
> {code}
> This will enable other tools like security / schema registry /  metadata 
> discovery to use replication related subsystem for their needs as well. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18352) introduce a METADATAONLY option while doing REPL DUMP to allow integrations of other tools

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314219#comment-16314219
 ] 

Hive QA commented on HIVE-18352:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
34s{color} | {color:red} ql: The patch generated 15 new + 134 unchanged - 29 
fixed = 149 total (was 163) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
11s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a6435cc |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8469/yetus/diff-checkstyle-ql.txt
 |
| asflicense | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8469/yetus/patch-asflicense-problems.txt
 |
| modules | C: common ql itests/hive-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8469/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> introduce a METADATAONLY option while doing REPL DUMP to allow integrations 
> of other tools 
> ---
>
> Key: HIVE-18352
> URL: https://issues.apache.org/jira/browse/HIVE-18352
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>  Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18352.0.patch, HIVE-18352.1.patch
>
>
> * Introduce a METADATAONLY option as part of the REPL DUMP command which will 
> only try and dump out events for DDL changes, this will be faster as we wont 
> need  scan of files on HDFS for DML changes. 
> * Additionally since we are only going to dump metadata operations, it might 
> be useful to include acid tables as well via an option as well. This option 
> can be removed when ACID support is complete via HIVE-18320
> it will be good to support the "WITH" clause as part of REPL DUMP command as 
> well (repl dump already supports it viaHIVE-17757) to achieve the above as 
> that will prevent less changes to the syntax of the statement and provide 
> more flexibility in future to include additional options as well. 
> {code}
> REPL DUMP [db_name] 

[jira] [Commented] (HIVE-17896) TopNKey: Create a standalone vectorizable TopNKey operator

[Gopal V (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314210#comment-16314210
 ] 

Gopal V commented on HIVE-17896:


Testing and review ongoing - currently this patch breaks DPP optimizer for some 
reason.

This is TPC-DS Query27 on 1Tb partitioned data, failing explains.

{code}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc.(ExprNodeColumnDesc.java:84)
at 
org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc.(ExprNodeColumnDesc.java:80)
at 
org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.generateSemiJoinOperatorPlan(DynamicPartitionPruningOptimization.java:528)
at 
org.apache.hadoop.hive.ql.optimizer.DynamicPartitionPruningOptimization.process(DynamicPartitionPruningOptimization.java:226)
at 
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at 
org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:74)
at 
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at 
org.apache.hadoop.hive.ql.parse.TezCompiler.runDynamicPartitionPruning(TezCompiler.java:503)
at 
org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:159)
at 
org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:145)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11726)
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:299)
at 
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:268)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:639)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1504)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1632)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1395)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1382)
{code}

> TopNKey: Create a standalone vectorizable TopNKey operator
> --
>
> Key: HIVE-17896
> URL: https://issues.apache.org/jira/browse/HIVE-17896
> Project: Hive
>  Issue Type: New Feature
>  Components: Operators
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Teddy Choi
> Attachments: HIVE-17896.1.patch, HIVE-17896.3.patch, 
> HIVE-17896.4.patch, HIVE-17896.5.patch
>
>
> For TPC-DS Query27, the TopN operation is delayed by the group-by - the 
> group-by operator buffers up all the rows before discarding the 99% of the 
> rows in the TopN Hash within the ReduceSink Operator.
> The RS TopN operator is very restrictive as it only supports doing the 
> filtering on the shuffle keys, but it is better to do this before breaking 
> the vectors into rows and losing the isRepeating properties.
> Adding a TopN Key operator in the physical operator tree allows the following 
> to happen.
> GBY->RS(Top=1)
> can become 
> TNK(1)->GBY->RS(Top=1)
> So that, the TopNKey can remove rows before they are buffered into the GBY 
> and consume memory.
> Here's the equivalent implementation in Presto
> https://github.com/prestodb/presto/blob/master/presto-main/src/main/java/com/facebook/presto/operator/TopNOperator.java#L35
> Adding this as a sub-feature of GroupBy prevents further optimizations if the 
> GBY is on keys "a,b,c" and the TopNKey is on just "a".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18096) add a user-friendly show plan command

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18096:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master, thanks for the patch!

> add a user-friendly show plan command
> -
>
> Key: HIVE-18096
> URL: https://issues.apache.org/jira/browse/HIVE-18096
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Fix For: 3.0.0
>
> Attachments: HIVE-18096.01.patch, HIVE-18096.02.patch, 
> HIVE-18096.03.patch, HIVE-18096.04.patch
>
>
> For admin to be able to get an overview of a resource plan.
> We need to try to do this using sysdb. 
> If that is not possible to do in a nice way, we'd do a text-based one like 
> query explain, or desc extended table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18172) make WM query triggers work with fetch task

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18172:

Summary: make WM query triggers work with fetch task  (was: Make query 
triggers work with fetch task)

> make WM query triggers work with fetch task
> ---
>
> Key: HIVE-18172
> URL: https://issues.apache.org/jira/browse/HIVE-18172
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> Currently triggers work only with counters from cluster tasks. This is to 
> support some cases where triggers can be applied for fetch task as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18172) Make query triggers work with fetch task

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18172:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> Make query triggers work with fetch task
> 
>
> Key: HIVE-18172
> URL: https://issues.apache.org/jira/browse/HIVE-18172
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> Currently triggers work only with counters from cluster tasks. This is to 
> support some cases where triggers can be applied for fetch task as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18096) add a user-friendly show plan command

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314172#comment-16314172
 ] 

Hive QA commented on HIVE-18096:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904744/HIVE-18096.04.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 16 failed/errored test(s), 11549 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] 
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8468/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8468/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8468/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 16 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904744 - PreCommit-HIVE-Build

> add a user-friendly show plan command
> -
>
> Key: HIVE-18096
> URL: https://issues.apache.org/jira/browse/HIVE-18096
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18096.01.patch, HIVE-18096.02.patch, 
> HIVE-18096.03.patch, HIVE-18096.04.patch
>
>
> For admin to be able to get an overview of a resource plan.
> We need to try to do this using sysdb. 
> If that is not possible to do in a nice way, we'd do a text-based one like 
> query explain, or desc extended table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18004) investigate deriving app name from JDBC connection for pool mapping

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18004:

Status: Patch Available  (was: Open)

> investigate deriving app name from JDBC connection for pool mapping
> ---
>
> Key: HIVE-18004
> URL: https://issues.apache.org/jira/browse/HIVE-18004
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18004.patch
>
>
> There are some client info fields that popular apps (Tableau, etc) might 
> populate; this might allow us to map queries to pools based on an application 
> used. Need to take a look (see the doc for an example API we might look into)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18367) Describe Extended output is truncated on a table with an explicit row format containing tabs or newlines.

[Andrew Sherman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Sherman updated HIVE-18367:
--
Attachment: HIVE-18367.2.patch

HIVE-18367.2.patch adds a BeeLine test and uses HiveStringUtils.escapeJava().
I confirmed that CliDriver output is the same by hand - the output is masked 
from a .q file test.

> Describe Extended output is truncated on a table with an explicit row format 
> containing tabs or newlines.
> -
>
> Key: HIVE-18367
> URL: https://issues.apache.org/jira/browse/HIVE-18367
> Project: Hive
>  Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18367.1.patch, HIVE-18367.2.patch
>
>
> 'Describe Extended' dumps information about a table. The protocol for sending 
> this data relies on tabs and newlines to separate pieces of data. If a table 
> has 'FIELDS terminated by XXX' or 'LINES terminated by XXX' where XXX is a 
> tab or newline then the output seen by the user is prematurely truncated. Fix 
> this by replacing tabs and newlines in the table description with “\n” and 
> “\t”.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18004) investigate deriving app name from JDBC connection for pool mapping

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18004:

Attachment: HIVE-18004.patch

> investigate deriving app name from JDBC connection for pool mapping
> ---
>
> Key: HIVE-18004
> URL: https://issues.apache.org/jira/browse/HIVE-18004
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18004.patch
>
>
> There are some client info fields that popular apps (Tableau, etc) might 
> populate; this might allow us to map queries to pools based on an application 
> used. Need to take a look (see the doc for an example API we might look into)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18312) Create or Replace a View fails with "Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions"

[Janaki Lahorani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314137#comment-16314137
 ] 

Janaki Lahorani commented on HIVE-18312:


[~haozhu]Please check if this is a duplicate of HIVE-17764.

> Create or Replace a View fails with "Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions"
> --
>
> Key: HIVE-18312
> URL: https://issues.apache.org/jira/browse/HIVE-18312
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 2.1.0
> Environment: Hive 2.1
>Reporter: Hao Zhu
>
> Create or Replace a view with different column types may fail with error :
> {code}
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> i
> {code}
> Here is minimum reproduce:
> {code}
> CREATE TABLE testtype(i int, s1 string);
> DDL1:
> create or replace view v1 (s1) 
> as Select s1 from testtype;
> DDL2:
> create or replace view v1 (i,s1) 
> as Select i, s1 from testtype ;
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> i
> {code}
> Only Workaround is:
> To drop the view and Create view again.
> We should allow "Create or Replace View" to work even if the column types are 
> different.
> It should work like "Drop View" and then "Create a View".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HIVE-18312) Create or Replace a View fails with "Unable to alter table. The following columns have types incompatible with the existing columns in their respective positions"

[Janaki Lahorani (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314137#comment-16314137
 ] 

Janaki Lahorani edited comment on HIVE-18312 at 1/5/18 11:34 PM:
-

[~haozhu] Please check if this is a duplicate of HIVE-17764.


was (Author: janulatha):
[~haozhu]Please check if this is a duplicate of HIVE-17764.

> Create or Replace a View fails with "Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions"
> --
>
> Key: HIVE-18312
> URL: https://issues.apache.org/jira/browse/HIVE-18312
> Project: Hive
>  Issue Type: Bug
>  Components: Database/Schema
>Affects Versions: 2.1.0
> Environment: Hive 2.1
>Reporter: Hao Zhu
>
> Create or Replace a view with different column types may fail with error :
> {code}
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> i
> {code}
> Here is minimum reproduce:
> {code}
> CREATE TABLE testtype(i int, s1 string);
> DDL1:
> create or replace view v1 (s1) 
> as Select s1 from testtype;
> DDL2:
> create or replace view v1 (i,s1) 
> as Select i, s1 from testtype ;
> FAILED: Execution Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.DDLTask. Unable to alter table. The following 
> columns have types incompatible with the existing columns in their respective 
> positions :
> i
> {code}
> Only Workaround is:
> To drop the view and Create view again.
> We should allow "Create or Replace View" to work even if the column types are 
> different.
> It should work like "Drop View" and then "Create a View".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-17239) HoS doesn't trigger mapjoins against subquery with union all

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani reassigned HIVE-17239:
--

Assignee: Janaki Lahorani

> HoS doesn't trigger mapjoins against subquery with union all
> 
>
> Key: HIVE-17239
> URL: https://issues.apache.org/jira/browse/HIVE-17239
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Janaki Lahorani
>
> HoS doesn't trigger map-joins for the following query:
> {code}
> EXPLAIN SELECT * FROM (SELECT part_col FROM partitioned_table1 UNION ALL 
> SELECT
> part_col FROM partitioned_table2) q1 JOIN regular_table1 JOIN regular_table2
> WHERE q1.part_col = regular_table1.col1 AND q1.part_col = regular_table2.col1;
> {code}
> Hive-on-Tez does.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18389) Print out Spark Web UI URL to the console log

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar reassigned HIVE-18389:
---


> Print out Spark Web UI URL to the console log
> -
>
> Key: HIVE-18389
> URL: https://issues.apache.org/jira/browse/HIVE-18389
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>
> Should be accessible via {{SparkContext#uiWebUrl}}. It just needs to be sent 
> from the {{RemoteDriver}} to HS2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18096) add a user-friendly show plan command

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314103#comment-16314103
 ] 

Hive QA commented on HIVE-18096:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
38s{color} | {color:green} standalone-metastore: The patch generated 0 new + 
2378 unchanged - 1 fixed = 2378 total (was 2379) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
41s{color} | {color:red} ql: The patch generated 3 new + 971 unchanged - 8 
fixed = 974 total (was 979) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 8s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m  1s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a6435cc |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8468/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8468/yetus/whitespace-eol.txt 
|
| modules | C: standalone-metastore ql itests/hcatalog-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8468/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> add a user-friendly show plan command
> -
>
> Key: HIVE-18096
> URL: https://issues.apache.org/jira/browse/HIVE-18096
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18096.01.patch, HIVE-18096.02.patch, 
> HIVE-18096.03.patch, HIVE-18096.04.patch
>
>
> For admin to be able to get an overview of a resource plan.
> We need to try to do this using sysdb. 
> If that is not possible to do in a nice way, we'd do a text-based one like 
> query explain, or desc extended table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18172) Make query triggers work with fetch task

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314088#comment-16314088
 ] 

Prasanth Jayachandran commented on HIVE-18172:
--

No. Not working on this. 

> Make query triggers work with fetch task
> 
>
> Key: HIVE-18172
> URL: https://issues.apache.org/jira/browse/HIVE-18172
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> Currently triggers work only with counters from cluster tasks. This is to 
> support some cases where triggers can be applied for fetch task as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18019) add metrics for workload management and expose via jmx endpoint

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18019:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> add metrics for workload management and expose via jmx endpoint
> ---
>
> Key: HIVE-18019
> URL: https://issues.apache.org/jira/browse/HIVE-18019
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> For debugging and testing purposes, some information from WM should be 
> exposed via JMX.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18172) Make query triggers work with fetch task

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314085#comment-16314085
 ] 

Sergey Shelukhin commented on HIVE-18172:
-

[~prasanth_j] do you have some update here?

> Make query triggers work with fetch task
> 
>
> Key: HIVE-18172
> URL: https://issues.apache.org/jira/browse/HIVE-18172
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> Currently triggers work only with counters from cluster tasks. This is to 
> support some cases where triggers can be applied for fetch task as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HIVE-18019) add metrics for workload management and expose via jmx endpoint

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin resolved HIVE-18019.
-
Resolution: Duplicate

Separate jiras have since been created for specific metrics. A more specific 
issue should be filed if needed in addition to those

> add metrics for workload management and expose via jmx endpoint
> ---
>
> Key: HIVE-18019
> URL: https://issues.apache.org/jira/browse/HIVE-18019
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> For debugging and testing purposes, some information from WM should be 
> exposed via JMX.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18276) add WM trace and some counters to "download data" button in Tez UI

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18276:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> add WM trace and some counters to "download data" button in Tez UI
> --
>
> Key: HIVE-18276
> URL: https://issues.apache.org/jira/browse/HIVE-18276
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18027) Explore the option of using AM's application id or attempt id to detect AM recovery

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18027:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> Explore the option of using AM's application id or attempt id to detect AM 
> recovery
> ---
>
> Key: HIVE-18027
> URL: https://issues.apache.org/jira/browse/HIVE-18027
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> HIVE-17904 uses sequence numbers generated by ZK to identify new AMs. Instead 
> use application id or attempt id or some ID that AM generated during its 
> recovery. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18204) Expose all WmContext json via jmx endpoint for completed queries

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18204:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> Expose all WmContext json via jmx endpoint for completed queries
> 
>
> Key: HIVE-18204
> URL: https://issues.apache.org/jira/browse/HIVE-18204
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> Follow up for HIVE-18088 to expose WmContext json via JMX endpoint for all or 
> recently completed queries.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18027) Explore the option of using AM's application id or attempt id to detect AM recovery

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18027:

Summary: Explore the option of using AM's application id or attempt id to 
detect AM recovery  (was: Explore the option of using AM's application id or 
attempt id for AM recovery)

> Explore the option of using AM's application id or attempt id to detect AM 
> recovery
> ---
>
> Key: HIVE-18027
> URL: https://issues.apache.org/jira/browse/HIVE-18027
> Project: Hive
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> HIVE-17904 uses sequence numbers generated by ZK to identify new AMs. Instead 
> use application id or attempt id or some ID that AM generated during its 
> recovery. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18097) WM query move phase 2 - handling the destination pool being full

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18097:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> WM query move phase 2 - handling the destination pool being full
> 
>
> Key: HIVE-18097
> URL: https://issues.apache.org/jira/browse/HIVE-18097
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> Final design TBD.
> We may add an option to move triggers: "on full kill/ignore/delay kill".
> The former will kill the query if the dest is full. 
> The ignore one would not perform the move... The thinking is that admin would 
> add another trigger to kill queries after some time (e.g. - move after 10s, 
> kill after 30s), and thus give slow/large queries a grace period in case dest 
> is full so they cannot gracefully release the capacity.
> It's possible to add a third option - "delay kill", where we would ignore the 
> move, but kill the query if someone requests capacity and the pool is full.
> Note that none of these require oversubscription; oversubscription is a pain 
> to track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17919) WM phase 2: don't kill all queries when a pool is resized down

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17919:

Summary: WM phase 2: don't kill all queries when a pool is resized down  
(was: phase 2: don't kill all queries when a pool is resized down)

> WM phase 2: don't kill all queries when a pool is resized down
> --
>
> Key: HIVE-17919
> URL: https://issues.apache.org/jira/browse/HIVE-17919
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> It's actually easy to do but during some design discussions was deemed to be 
> more confusing. Came up during code review.
> When a pool is resized from e.g. 5 to 3 queries, and 4 are running, we only 
> need to kill one query, not all 4.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18097) WM query move phase 2 - handling the destination pool being full

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18097:

Summary: WM query move phase 2 - handling the destination pool being full  
(was: query move phase 2 - handling the destination pool being full)

> WM query move phase 2 - handling the destination pool being full
> 
>
> Key: HIVE-18097
> URL: https://issues.apache.org/jira/browse/HIVE-18097
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>
> Final design TBD.
> We may add an option to move triggers: "on full kill/ignore/delay kill".
> The former will kill the query if the dest is full. 
> The ignore one would not perform the move... The thinking is that admin would 
> add another trigger to kill queries after some time (e.g. - move after 10s, 
> kill after 30s), and thus give slow/large queries a grace period in case dest 
> is full so they cannot gracefully release the capacity.
> It's possible to add a third option - "delay kill", where we would ignore the 
> move, but kill the query if someone requests capacity and the pool is full.
> Note that none of these require oversubscription; oversubscription is a pain 
> to track.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17946) WM: provide option to bound task parallelism via triggers

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17946:

Summary: WM: provide option to bound task parallelism via triggers  (was: 
Provide option to bound task parallelism via triggers)

> WM: provide option to bound task parallelism via triggers
> -
>
> Key: HIVE-17946
> URL: https://issues.apache.org/jira/browse/HIVE-17946
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> With TEZ-2914, it is possible to create triggers specifically the one 
> TOTAL_TASKS > some_limit, to enforce max bounds instead of killing the query.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17946) Provide option to bound task parallelism via triggers

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17946:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> Provide option to bound task parallelism via triggers
> -
>
> Key: HIVE-17946
> URL: https://issues.apache.org/jira/browse/HIVE-17946
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>
> With TEZ-2914, it is possible to create triggers specifically the one 
> TOTAL_TASKS > some_limit, to enforce max bounds instead of killing the query.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17919) phase 2: don't kill all queries when a pool is resized down

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17919:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> phase 2: don't kill all queries when a pool is resized down
> ---
>
> Key: HIVE-17919
> URL: https://issues.apache.org/jira/browse/HIVE-17919
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> It's actually easy to do but during some design discussions was deemed to be 
> more confusing. Came up during code review.
> When a pool is resized from e.g. 5 to 3 queries, and 4 are running, we only 
> need to kill one query, not all 4.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17622) Implement ANTLR based expressions for rule triggers

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-17622:

Issue Type: Bug  (was: Sub-task)
Parent: (was: HIVE-17481)

> Implement ANTLR based expressions for rule triggers
> ---
>
> Key: HIVE-17622
> URL: https://issues.apache.org/jira/browse/HIVE-17622
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>
> For more expressiveness of rule trigger expressions move manual expression 
> parsing in HIVE-17508 to ANTLR based grammar.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17396) Support DPP with map joins where the source and target belong in the same stage

[Janaki Lahorani (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Janaki Lahorani updated HIVE-17396:
---
Attachment: HIVE-17396.4.patch

Reattaching patch to trigger pre-commit tests.

> Support DPP with map joins where the source and target belong in the same 
> stage
> ---
>
> Key: HIVE-17396
> URL: https://issues.apache.org/jira/browse/HIVE-17396
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Attachments: HIVE-17396.1.patch, HIVE-17396.1.patch, 
> HIVE-17396.1.patch, HIVE-17396.2.patch, HIVE-17396.3.patch, 
> HIVE-17396.4.patch, HIVE-17396.4.patch
>
>
> When the target of a partition pruning sink operator is in not the same as 
> the target of hash table sink operator, both source and target gets scheduled 
> within the same spark job, and that can result in File Not Found Exception.  
> HIVE-17225 has a fix to disable DPP in that scenario.  This JIRA is to 
> support DPP for such cases.
> Test Case:
> SET hive.spark.dynamic.partition.pruning=true;
> SET hive.auto.convert.join=true;
> SET hive.strict.checks.cartesian.product=false;
> CREATE TABLE part_table1 (col int) PARTITIONED BY (part1_col int);
> CREATE TABLE part_table2 (col int) PARTITIONED BY (part2_col int);
> CREATE TABLE reg_table (col int);
> ALTER TABLE part_table1 ADD PARTITION (part1_col = 1);
> ALTER TABLE part_table2 ADD PARTITION (part2_col = 1);
> ALTER TABLE part_table2 ADD PARTITION (part2_col = 2);
> INSERT INTO TABLE part_table1 PARTITION (part1_col = 1) VALUES (1);
> INSERT INTO TABLE part_table2 PARTITION (part2_col = 1) VALUES (1);
> INSERT INTO TABLE part_table2 PARTITION (part2_col = 2) VALUES (2);
> INSERT INTO table reg_table VALUES (1), (2), (3), (4), (5), (6);
> EXPLAIN SELECT *
> FROM   part_table1 pt1,
>part_table2 pt2,
>reg_table rt
> WHERE  rt.col = pt1.part1_col
> ANDpt2.part2_col = pt1.part1_col;
> Plan:
> STAGE DEPENDENCIES:
>   Stage-2 is a root stage
>   Stage-1 depends on stages: Stage-2
>   Stage-0 depends on stages: Stage-1
> STAGE PLANS:
>   Stage: Stage-2
> Spark
>  A masked pattern was here 
>   Vertices:
> Map 1 
> Map Operator Tree:
> TableScan
>   alias: pt1
>   Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE 
> Column stats: NONE
>   Select Operator
> expressions: col (type: int), part1_col (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
> Spark HashTable Sink Operator
>   keys:
> 0 _col1 (type: int)
> 1 _col1 (type: int)
> 2 _col0 (type: int)
> Select Operator
>   expressions: _col1 (type: int)
>   outputColumnNames: _col0
>   Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
>   Group By Operator
> keys: _col0 (type: int)
> mode: hash
> outputColumnNames: _col0
> Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
> Spark Partition Pruning Sink Operator
>   Target column: part2_col (int)
>   partition key expr: part2_col
>   Statistics: Num rows: 1 Data size: 1 Basic stats: 
> COMPLETE Column stats: NONE
>   target work: Map 2
> Local Work:
>   Map Reduce Local Work
> Map 2 
> Map Operator Tree:
> TableScan
>   alias: pt2
>   Statistics: Num rows: 2 Data size: 2 Basic stats: COMPLETE 
> Column stats: NONE
>   Select Operator
> expressions: col (type: int), part2_col (type: int)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 2 Data size: 2 Basic stats: 
> COMPLETE Column stats: NONE
> Spark HashTable Sink Operator
>   keys:
> 0 _col1 (type: int)
> 1 _col1 (type: int)
> 2 _col0 (type: int)
> Local Work:
>   Map Reduce Local Work
>   Stage: Stage-1
> Spark
>  A masked pattern was here 
>   Vertices:
> Map 3 
> Map Operator Tree:
> TableScan
>   alias: 

[jira] [Commented] (HIVE-18096) add a user-friendly show plan command

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314064#comment-16314064
 ] 

Hive QA commented on HIVE-18096:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904744/HIVE-18096.04.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 11549 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] 
(batchId=48)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] 
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
 (batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementAsync
 (batchId=229)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testExecuteStatementParallel
 (batchId=229)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8467/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8467/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8467/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904744 - PreCommit-HIVE-Build

> add a user-friendly show plan command
> -
>
> Key: HIVE-18096
> URL: https://issues.apache.org/jira/browse/HIVE-18096
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18096.01.patch, HIVE-18096.02.patch, 
> HIVE-18096.03.patch, HIVE-18096.04.patch
>
>
> For admin to be able to get an overview of a resource plan.
> We need to try to do this using sysdb. 
> If that is not possible to do in a nice way, we'd do a text-based one like 
> query explain, or desc extended table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18388) LLAP RecordReader may fail for ACID scan if included cols is null

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18388:
--
Component/s: Transactions

> LLAP RecordReader may fail for ACID scan if included cols is null
> -
>
> Key: HIVE-18388
> URL: https://issues.apache.org/jira/browse/HIVE-18388
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Reporter: Sergey Shelukhin
>
> We haven't ever seen this, but it may theoretically happen. See HIVE-18269.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18275) add HS2-level WM metrics

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18275:

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Committed to master. Thanks for the review.

> add HS2-level WM metrics
> 
>
> Key: HIVE-18275
> URL: https://issues.apache.org/jira/browse/HIVE-18275
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-18275.patch
>
>
> E.g. time spent in pool queue. Some existing UIs use perflogger output, so we 
> should also include that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (HIVE-18377) avoid explicitly setting HIVE_SUPPORT_CONCURRENCY in JUnit tests

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313819#comment-16313819
 ] 

Eugene Koifman edited comment on HIVE-18377 at 1/5/18 10:17 PM:


This won't work because doing this also enabled the default ZK based lock 
manager which wedges a lot of these tests.
Many but not all run with with {{hive.support.concurrency=true}} and 
{{hvie.txn.manager=rg.apache.hadoop.hive.ql.lockmgr.DbTxnManager}}


was (Author: ekoifman):
This won't work because doing this also enabled the default ZK based lock 
manager which wedges a lot of these tests.

> avoid explicitly setting HIVE_SUPPORT_CONCURRENCY in JUnit tests
> 
>
> Key: HIVE-18377
> URL: https://issues.apache.org/jira/browse/HIVE-18377
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18377.02.patch
>
>
> many UTs (e.g. TestHCatMultiOutputFormat, 
> BeelineWithHS2ConnectionFileTestBase, TestOperationLoggingAPIWithMr, 
> HCatBaseTest and many others)
> explicitly set 
> {{hiveConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, "false");}}
> It would be better if they picked up the settings from 
> data/conf/hive-site.xml.
> It adds consistency and makes it possible to run all tests with known config 
> (at least approach this).
> The outline of the process is:
> 1. build copies {{\*-site.xml files from data/conf/\*\*/\*-site.xml}} to 
> target/testconf/
> 2. HiveConf picks up target/testconf/hive-site.xml
> 3. Various forms of *CliDriver may explicitly specify (e.g. 
> MiniLlapLocalCliConfig) which hive-site.xml to use
>  
> The first step is to see how many explicit settings of 
> HIVE_SUPPORT_CONCURRENCY can be removed w/o breaking the tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-17112) Reduce logging in HiveSparkClientFactory and RemoteHiveSparkClient

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-17112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-17112:

Issue Type: Sub-task  (was: Improvement)
Parent: HIVE-17718

> Reduce logging in HiveSparkClientFactory and RemoteHiveSparkClient
> --
>
> Key: HIVE-17112
> URL: https://issues.apache.org/jira/browse/HIVE-17112
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 3.0.0
>
> Attachments: HIVE-17112.1.patch
>
>
> HiveSparkClientFactory has the following line that introduces excess logging:
> {code}
> LOG.info(String.format(
>   "load spark property from %s (%s -> %s).",
>   SPARK_DEFAULT_CONF_FILE, propertyName, 
> LogUtils.maskIfPassword(propertyName,value)));
> {code}
> It basically dumps the entire configuration object to the logs, we can 
> probably change this from INFO to DEBUG.
> Same thing happens in {{RemoteHiveSparkClient#logConfigurations}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18368) Improve Spark Debug RDD Graph

[Sahil Takiar (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313998#comment-16313998
 ] 

Sahil Takiar commented on HIVE-18368:
-

[~chengxiang li] I saw you added the RDD graph in HIVE-10550, [~chinnalalam] I 
saw you added the SparkPlan graph in HIVE-8858. Could you take a look at this 
patch? - RB: https://reviews.apache.org/r/64996/

> Improve Spark Debug RDD Graph
> -
>
> Key: HIVE-18368
> URL: https://issues.apache.org/jira/browse/HIVE-18368
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-18368.1.patch, HIVE-18368.2.patch, Spark UI - Named 
> RDDs.png
>
>
> The {{SparkPlan}} class does some logging to show the mapping between 
> different {{SparkTran}}, what shuffle types are used, and what trans are 
> cached. However, there is room for improvement.
> When debug logging is enabled the RDD graph is logged, but there isn't much 
> information printed about each RDD.
> We should combine both of the graphs and improve them. We could even make the 
> Spark Plan graph part of the {{EXPLAIN EXTENDED}} output.
> Ideally, the final graph shows a clear relationship between Tran objects, 
> RDDs, and BaseWorks. Edge should include information about number of 
> partitions, shuffle types, Spark operations used, etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18269) LLAP: Fast llap io with slow processing pipeline can lead to OOM

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18269:

Attachment: HIVE-18269.02.patch

A stupid bug...

> LLAP: Fast llap io with slow processing pipeline can lead to OOM
> 
>
> Key: HIVE-18269
> URL: https://issues.apache.org/jira/browse/HIVE-18269
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18269.01.patch, HIVE-18269.02.patch, 
> HIVE-18269.1.patch, HIVE-18269.bad.patch, Screen Shot 2017-12-13 at 1.15.16 
> AM.png
>
>
> pendingData linked list in Llap IO elevator (LlapRecordReader.java) may grow 
> indefinitely when Llap IO is faster than processing pipeline. Since we don't 
> have backpressure to slow down the IO, this can lead to indefinite growth of 
> pending data leading to severe GC pressure and eventually lead to OOM.
> This specific instance of LLAP was running on HDFS on top of EBS volume 
> backed by SSD. The query that triggered this is issue was ANALYZE STATISTICS 
> .. FOR COLUMNS which also gather bitvectors. Fast IO and Slow processing case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18368) Improve Spark Debug RDD Graph

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18368:

Attachment: HIVE-18368.2.patch

Fixing checkstyle issue

> Improve Spark Debug RDD Graph
> -
>
> Key: HIVE-18368
> URL: https://issues.apache.org/jira/browse/HIVE-18368
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-18368.1.patch, HIVE-18368.2.patch, Spark UI - Named 
> RDDs.png
>
>
> The {{SparkPlan}} class does some logging to show the mapping between 
> different {{SparkTran}}, what shuffle types are used, and what trans are 
> cached. However, there is room for improvement.
> When debug logging is enabled the RDD graph is logged, but there isn't much 
> information printed about each RDD.
> We should combine both of the graphs and improve them. We could even make the 
> Spark Plan graph part of the {{EXPLAIN EXTENDED}} output.
> Ideally, the final graph shows a clear relationship between Tran objects, 
> RDDs, and BaseWorks. Edge should include information about number of 
> partitions, shuffle types, Spark operations used, etc.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18096) add a user-friendly show plan command

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313975#comment-16313975
 ] 

Hive QA commented on HIVE-18096:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  5m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} standalone-metastore: The patch generated 0 new + 
2378 unchanged - 1 fixed = 2378 total (was 2379) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 3 new + 971 unchanged - 8 
fixed = 974 total (was 979) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
 9s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m  8s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b0e653a |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8467/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8467/yetus/whitespace-eol.txt 
|
| modules | C: standalone-metastore ql itests/hcatalog-unit U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8467/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> add a user-friendly show plan command
> -
>
> Key: HIVE-18096
> URL: https://issues.apache.org/jira/browse/HIVE-18096
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Harish Jaiprakash
> Attachments: HIVE-18096.01.patch, HIVE-18096.02.patch, 
> HIVE-18096.03.patch, HIVE-18096.04.patch
>
>
> For admin to be able to get an overview of a resource plan.
> We need to try to do this using sysdb. 
> If that is not possible to do in a nice way, we'd do a text-based one like 
> query explain, or desc extended table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18353) CompactorMR should call jobclient.close() to trigger cleanup

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313970#comment-16313970
 ] 

Eugene Koifman commented on HIVE-18353:
---

LGTM

> CompactorMR should call jobclient.close() to trigger cleanup
> 
>
> Key: HIVE-18353
> URL: https://issues.apache.org/jira/browse/HIVE-18353
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HIVE-18353.1.patch, HIVE-18353.2.patch, HIVE-18353.patch
>
>
> HiveMetastore process is leaking TrustStore reloader threads when running 
> compaction as JobClient close is not called from CompactorMR - MAPREDUCE-6618 
> and MAPREDUCE-6621 
> {code}
> "Truststore reloader thread" #2814 daemon prio=1 os_prio=0 
> tid=0x00cdc800 nid=0x2f05a waiting on condition [0x7fdaef403000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at 
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:194)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-16484:

Attachment: HIVE-16484.10.patch

> Investigate SparkLauncher for HoS as alternative to bin/spark-submit
> 
>
> Key: HIVE-16484
> URL: https://issues.apache.org/jira/browse/HIVE-16484
> Project: Hive
>  Issue Type: Bug
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-16484.1.patch, HIVE-16484.10.patch, 
> HIVE-16484.2.patch, HIVE-16484.3.patch, HIVE-16484.4.patch, 
> HIVE-16484.5.patch, HIVE-16484.6.patch, HIVE-16484.7.patch, 
> HIVE-16484.8.patch, HIVE-16484.9.patch
>
>
> The {{SparkClientImpl#startDriver}} currently looks for the {{SPARK_HOME}} 
> directory and invokes the {{bin/spark-submit}} script, which spawns a 
> separate process to run the Spark application.
> {{SparkLauncher}} was added in SPARK-4924 and is a programatic way to launch 
> Spark applications.
> I see a few advantages:
> * No need to spawn a separate process to launch a HoS --> lower startup time
> * Simplifies the code in {{SparkClientImpl}} --> easier to debug
> * {{SparkLauncher#startApplication}} returns a {{SparkAppHandle}} which 
> contains some useful utilities for querying the state of the Spark job
> ** It also allows the launcher to specify a list of job listeners



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18326) LLAP Tez scheduler - only preempt tasks if there's a dependency between them

[Jason Dere (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313957#comment-16313957
 ] 

Jason Dere commented on HIVE-18326:
---

+1

> LLAP Tez scheduler - only preempt tasks if there's a dependency between them
> 
>
> Key: HIVE-18326
> URL: https://issues.apache.org/jira/browse/HIVE-18326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-18326.01.patch, HIVE-18326.02.patch, 
> HIVE-18326.02.patch, HIVE-18326.patch
>
>
> It is currently possible for e.g. two sides of a union (or a join for that 
> matter) to have slightly different priorities. We don't want to preempt 
> running tasks on one side in favor of the other side in such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18349) Misc metastore changes for debuggability, error on commit txn failures

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-18349:
-
Attachment: HIVE-18349.6.patch

stats related metastore events did not pass the success state correctly that 
was causing these test failures. 

> Misc metastore changes for debuggability, error on commit txn failures
> --
>
> Key: HIVE-18349
> URL: https://issues.apache.org/jira/browse/HIVE-18349
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18349.1.patch, HIVE-18349.2.patch, 
> HIVE-18349.3.patch, HIVE-18349.4.patch, HIVE-18349.5.patch, HIVE-18349.6.patch
>
>
> 1) Hive metastore audit event log/metastore log does not log the final status 
> (success or failed) of the event. Some operations like for example, 
> drop_table returns a boolean success flag but it never gets logged anywhere. 
> However the same is sent to end event listeners or other metastore event 
> listeners. It will be good to log the final status of the events. 
> 2) Make connection timeout when using connection pool configurable. Currently 
> its hard coded to 30 seconds.
> 3) Provide a config to enable connection leak detection for HikariCP or 
> enable when debug logging is enabled.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-17966) org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveArrayInspector - Review

[BELUGA BEHR (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-17966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313945#comment-16313945
 ] 

BELUGA BEHR commented on HIVE-17966:


[~aihuaxu] [~pvary] Hey team :) Any update on this?

> org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveArrayInspector - Review
> -
>
> Key: HIVE-17966
> URL: https://issues.apache.org/jira/browse/HIVE-17966
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2, Serializers/Deserializers
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HIVE-17966.1.patch, HIVE-17966.2.patch
>
>
> * Simplify
> * Make faster - perform bulk operations instead of iterating
> * Remove compilation warnings



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18326) LLAP Tez scheduler - only preempt tasks if there's a dependency between them

[Sergey Shelukhin (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313942#comment-16313942
 ] 

Sergey Shelukhin commented on HIVE-18326:
-

[~jdere] can you look at the update? small addendum to fix the issue 
https://reviews.apache.org/r/64787/diff/2-3/

> LLAP Tez scheduler - only preempt tasks if there's a dependency between them
> 
>
> Key: HIVE-18326
> URL: https://issues.apache.org/jira/browse/HIVE-18326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-18326.01.patch, HIVE-18326.02.patch, 
> HIVE-18326.02.patch, HIVE-18326.patch
>
>
> It is currently possible for e.g. two sides of a union (or a join for that 
> matter) to have slightly different priorities. We don't want to preempt 
> running tasks on one side in favor of the other side in such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18353) CompactorMR should call jobclient.close() to trigger cleanup

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313939#comment-16313939
 ] 

Thejas M Nair commented on HIVE-18353:
--

+1 

cc [~ekoifman]

> CompactorMR should call jobclient.close() to trigger cleanup
> 
>
> Key: HIVE-18353
> URL: https://issues.apache.org/jira/browse/HIVE-18353
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, Transactions
>Affects Versions: 1.2.1
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
> Attachments: HIVE-18353.1.patch, HIVE-18353.2.patch, HIVE-18353.patch
>
>
> HiveMetastore process is leaking TrustStore reloader threads when running 
> compaction as JobClient close is not called from CompactorMR - MAPREDUCE-6618 
> and MAPREDUCE-6621 
> {code}
> "Truststore reloader thread" #2814 daemon prio=1 os_prio=0 
> tid=0x00cdc800 nid=0x2f05a waiting on condition [0x7fdaef403000]
>java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at 
> org.apache.hadoop.security.ssl.ReloadingX509TrustManager.run(ReloadingX509TrustManager.java:194)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18326) LLAP Tez scheduler - only preempt tasks if there's a dependency between them

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-18326:

Attachment: HIVE-18326.02.patch

Updated; didn't realize the plugin was reused.

> LLAP Tez scheduler - only preempt tasks if there's a dependency between them
> 
>
> Key: HIVE-18326
> URL: https://issues.apache.org/jira/browse/HIVE-18326
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Fix For: 3.0.0
>
> Attachments: HIVE-18326.01.patch, HIVE-18326.02.patch, 
> HIVE-18326.02.patch, HIVE-18326.patch
>
>
> It is currently possible for e.g. two sides of a union (or a join for that 
> matter) to have slightly different priorities. We don't want to preempt 
> running tasks on one side in favor of the other side in such cases.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18359) Extend grouping set limits from int to long

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313927#comment-16313927
 ] 

Hive QA commented on HIVE-18359:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904732/HIVE-18359.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 11548 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_window] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_1] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_grouping_sets]
 (batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query27] 
(batchId=247)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query36] 
(batchId=247)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query70] 
(batchId=247)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query86] 
(batchId=247)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8466/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8466/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8466/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904732 - PreCommit-HIVE-Build

> Extend grouping set limits from int to long
> ---
>
> Key: HIVE-18359
> URL: https://issues.apache.org/jira/browse/HIVE-18359
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18359.1.patch, HIVE-18359.2.patch, 
> HIVE-18359.3.patch
>
>
> Grouping sets is broken for >32 columns because of usage of Int for bitmap 
> (also GROUPING__ID virtual column). This assumption breaks grouping 
> sets/rollups/cube when number of participating aggregation columns is >32. 
> The easier fix would be extend it to Long for now. The correct fix would be 
> to use BitSets everywhere but that would require GROUPING__ID column type to 
> binary which will make predicates on GROUPING__ID difficult to deal with. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18275) add HS2-level WM metrics

[Thejas M Nair (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313913#comment-16313913
 ] 

Thejas M Nair commented on HIVE-18275:
--

+1

> add HS2-level WM metrics
> 
>
> Key: HIVE-18275
> URL: https://issues.apache.org/jira/browse/HIVE-18275
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18275.patch
>
>
> E.g. time spent in pool queue. Some existing UIs use perflogger output, so we 
> should also include that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18221) test acid default

[Eugene Koifman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-18221:
--
Attachment: HIVE-18221.24.patch

> test acid default
> -
>
> Key: HIVE-18221
> URL: https://issues.apache.org/jira/browse/HIVE-18221
> Project: Hive
>  Issue Type: Test
>  Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18221.01.patch, HIVE-18221.02.patch, 
> HIVE-18221.03.patch, HIVE-18221.04.patch, HIVE-18221.07.patch, 
> HIVE-18221.08.patch, HIVE-18221.09.patch, HIVE-18221.10.patch, 
> HIVE-18221.11.patch, HIVE-18221.12.patch, HIVE-18221.13.patch, 
> HIVE-18221.14.patch, HIVE-18221.16.patch, HIVE-18221.18.patch, 
> HIVE-18221.19.patch, HIVE-18221.20.patch, HIVE-18221.21.patch, 
> HIVE-18221.22.patch, HIVE-18221.23.patch, HIVE-18221.24.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18202) Automatically migrate hbase.table.name to hbase.mapreduce.hfileoutputformat.table.name for hbase-based table

[Naveen Gangam (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313881#comment-16313881
 ] 

Naveen Gangam commented on HIVE-18202:
--

Thanks for clarification. The change looks good to me. +1 for me.

> Automatically migrate hbase.table.name to 
> hbase.mapreduce.hfileoutputformat.table.name for hbase-based table
> 
>
> Key: HIVE-18202
> URL: https://issues.apache.org/jira/browse/HIVE-18202
> Project: Hive
>  Issue Type: Sub-task
>  Components: HBase Handler
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-18202.1.patch
>
>
> The property name for Hbase table mapping is changed from hbase.table.name to 
> hbase.mapreduce.hfileoutputformat.table.name in HBase 2.
> We can include such upgrade for existing hbase-based tables in DB upgrade 
> script to automatically change such values.
> For the new tables, the query will be like:
> create table hbase_table(key int, val string) stored by 
> 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties 
> ('hbase.columns.mapping' = ':key,cf:val') tblproperties 
> ('hbase.mapreduce.hfileoutputformat.table.name' = 
> 'positive_hbase_handler_bulk')



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HIVE-18388) LLAP RecordReader may fail for ACID scan if included cols is null

[Sergey Shelukhin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin resolved HIVE-18388.
-
Resolution: Invalid

Will actually fix as part of the other one

> LLAP RecordReader may fail for ACID scan if included cols is null
> -
>
> Key: HIVE-18388
> URL: https://issues.apache.org/jira/browse/HIVE-18388
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey Shelukhin
>
> We haven't ever seen this, but it may theoretically happen. See HIVE-18269.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18359) Extend grouping set limits from int to long

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313830#comment-16313830
 ] 

Hive QA commented on HIVE-18359:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
55s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
45s{color} | {color:red} ql: The patch generated 7 new + 1446 unchanged - 3 
fixed = 1453 total (was 1449) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 96 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 34s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b0e653a |
| Default Java | 1.8.0_111 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8466/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8466/yetus/whitespace-eol.txt 
|
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-8466/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Extend grouping set limits from int to long
> ---
>
> Key: HIVE-18359
> URL: https://issues.apache.org/jira/browse/HIVE-18359
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18359.1.patch, HIVE-18359.2.patch, 
> HIVE-18359.3.patch
>
>
> Grouping sets is broken for >32 columns because of usage of Int for bitmap 
> (also GROUPING__ID virtual column). This assumption breaks grouping 
> sets/rollups/cube when number of participating aggregation columns is >32. 
> The easier fix would be extend it to Long for now. The correct fix would be 
> to use BitSets everywhere but that would require GROUPING__ID column type to 
> binary which will make predicates on GROUPING__ID difficult to deal with. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18377) avoid explicitly setting HIVE_SUPPORT_CONCURRENCY in JUnit tests

[Eugene Koifman (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313819#comment-16313819
 ] 

Eugene Koifman commented on HIVE-18377:
---

This won't work because doing this also enabled the default ZK based lock 
manager which wedges a lot of these tests.

> avoid explicitly setting HIVE_SUPPORT_CONCURRENCY in JUnit tests
> 
>
> Key: HIVE-18377
> URL: https://issues.apache.org/jira/browse/HIVE-18377
> Project: Hive
>  Issue Type: Sub-task
>  Components: Test, Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-18377.02.patch
>
>
> many UTs (e.g. TestHCatMultiOutputFormat, 
> BeelineWithHS2ConnectionFileTestBase, TestOperationLoggingAPIWithMr, 
> HCatBaseTest and many others)
> explicitly set 
> {{hiveConf.set(HiveConf.ConfVars.HIVE_SUPPORT_CONCURRENCY.varname, "false");}}
> It would be better if they picked up the settings from 
> data/conf/hive-site.xml.
> It adds consistency and makes it possible to run all tests with known config 
> (at least approach this).
> The outline of the process is:
> 1. build copies {{\*-site.xml files from data/conf/\*\*/\*-site.xml}} to 
> target/testconf/
> 2. HiveConf picks up target/testconf/hive-site.xml
> 3. Various forms of *CliDriver may explicitly specify (e.g. 
> MiniLlapLocalCliConfig) which hive-site.xml to use
>  
> The first step is to see how many explicit settings of 
> HIVE_SUPPORT_CONCURRENCY can be removed w/o breaking the tests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18350) load data should rename files consistent with insert statements

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313805#comment-16313805
 ] 

Hive QA commented on HIVE-18350:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904723/HIVE-18350.1.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8465/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8465/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8465/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-01-05 20:12:59.353
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8465/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-01-05 20:12:59.357
+ cd apache-github-source-source
+ git fetch origin
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
+ git reset --hard HEAD
HEAD is now at b0e653a HIVE-18354: Fix test TestAcidOnTez  (Zoltan Haindrich, 
reviewed by Eugene Koifman)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b0e653a HIVE-18354: Fix test TestAcidOnTez  (Zoltan Haindrich, 
reviewed by Eugene Koifman)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-01-05 20:13:03.882
+ rm -rf ../yetus
+ mkdir ../yetus
+ cp -R . ../yetus
cp: cannot stat ?./.git/objects/78/c315fa77c09546ca46eebd7721092b0a7868d2?: No 
such file or directory
cp: cannot stat ?./.git/gc.pid?: No such file or directory
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904723 - PreCommit-HIVE-Build

> load data should rename files consistent with insert statements
> ---
>
> Key: HIVE-18350
> URL: https://issues.apache.org/jira/browse/HIVE-18350
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: HIVE-18350.1.patch
>
>
> Insert statements create files of format ending with _0, 0001_0 etc. 
> However, the load data uses the input file name. That results in inconsistent 
> naming convention which makes SMB joins difficult in some scenarios and may 
> cause trouble for other types of queries in future.
> We need consistent naming convention.
> For non-bucketed table, hive renames all the files regardless of how they 
> were named by the user.
> For bucketed table, hive relies on user to name the files matching the bucket 
> in non-strict mode. Hive assumes that the data belongs to same bucket in a 
> file. In strict mode, loading bucketed table is disabled.
> This will likely affect most of the tests which load data which is pretty 
> significant.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18359) Extend grouping set limits from int to long

[Hive QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313802#comment-16313802
 ] 

Hive QA commented on HIVE-18359:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12904732/HIVE-18359.3.patch

{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 11548 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_grouping_window] 
(batchId=32)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
 (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cte_1] 
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby_rollup_empty]
 (batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
 (batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] 
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_groupby_grouping_sets_grouping]
 (batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_grouping_sets]
 (batchId=169)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
 (batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
 (batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] 
(batchId=120)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query27] 
(batchId=247)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query36] 
(batchId=247)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query70] 
(batchId=247)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query86] 
(batchId=247)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
 (batchId=213)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints 
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8464/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8464/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8464/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12904732 - PreCommit-HIVE-Build

> Extend grouping set limits from int to long
> ---
>
> Key: HIVE-18359
> URL: https://issues.apache.org/jira/browse/HIVE-18359
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18359.1.patch, HIVE-18359.2.patch, 
> HIVE-18359.3.patch
>
>
> Grouping sets is broken for >32 columns because of usage of Int for bitmap 
> (also GROUPING__ID virtual column). This assumption breaks grouping 
> sets/rollups/cube when number of participating aggregation columns is >32. 
> The easier fix would be extend it to Long for now. The correct fix would be 
> to use BitSets everywhere but that would require GROUPING__ID column type to 
> binary which will make predicates on GROUPING__ID difficult to deal with. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18387) Minimize time that REBUILD locks the materialized view

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-18387:
--


> Minimize time that REBUILD locks the materialized view
> --
>
> Key: HIVE-18387
> URL: https://issues.apache.org/jira/browse/HIVE-18387
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> Currently, REBUILD will block the materialized view while the final move task 
> is being executed. The idea for this improvement is to create the new 
> materialization in a new folder (new version) and then just flip the pointer 
> to the folder in the MV definition in the metastore. REBUILD operations for a 
> given MV should get an exclusive lock though, i.e., they cannot be executed 
> concurrently.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-16489) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

[Misha Dmitriev (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313767#comment-16313767
 ] 

Misha Dmitriev commented on HIVE-16489:
---

Hi [~szita] - apparently yes, just closed it.

> HMS wastes 26.4% of memory due to dup strings in 
> metastore.api.Partition.parameters
> ---
>
> Key: HIVE-16489
> URL: https://issues.apache.org/jira/browse/HIVE-16489
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>
> I've just analyzed an HMS heap dump. It turns out that it contains a lot of 
> duplicate strings, that waste 26.4% of the heap. Most of them come from 
> HashMaps referenced by 
> org.apache.hadoop.hive.metastore.api.Partition.parameters. Below is the 
> relevant section of the jxray (www.jxray.com) report. Looking at 
> Partition.java, I see that in the past somebody has already added code to 
> intern keys and values in the parameters table when it's first set up. 
> However, looks like when more key-value pairs are added, they are not 
> interned, and that probably explains the reason for all these duplicate 
> strings.
> {code}
> 6. DUPLICATE STRINGS
> Total strings: 3,273,557  Unique strings: 460,390  Duplicate values: 110,232  
> Overhead: 3,220,458K (26.4%)
> Top duplicate strings:
> Ovhd Num char[]s   Num objs   Value
>  46,088K (0.4%) 58715871  
> "HBa4rRAAGx2MEmludGVyZXN0cmF0ZXNwcmVhZBgM/wD/AP8AXqEAERYBFQAXIEAWuK0QAA1s
>  ...[length 4000]"
>  46,088K (0.4%) 58715871  
> "BQcHBQUGBQgGBQcHCAUGCAkECQcFBQwGBgoJBQYHBQUFBQYKBQgIBgUJEgYFDAYJBgcGBAcLBQYGCAgGCQYG
>  ...[length 4000]"
> ...
> ===
> 7. REFERENCE CHAINS FOR DUPLICATE STRINGS
>   2,326,150K (19.1%), 597058 dup strings (36386 unique), 597058 dup backing 
> arrays:
> 39949 of "-1", 39088 of "true", 28959 of "8", 20987 of "1", 18437 of "10", 
> 9583 of "9", 5908 of "269664", 5691 of "174528", 4598 of "133980", 4598 of 
> "BgUGBQgFCAYFCgYIBgUEBgQHBgUGCwYGBwYHBgkKBwYGBggIBwUHBgYGCgUJCQUG ...[length 
> 3560]"
> ... and 419200 more strings, of which 36376 are unique
> Also contains one-char strings: 217 of "6", 147 of "7", 91 of "4", 28 of "5", 
> 28 of "2", 21 of "0"
>  <--  {j.u.HashMap}.values <-- 
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--  
> {j.u.ArrayList} <-- 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success
>  <-- Java Local 
> (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result)
>  [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots
>   463,060K (3.8%), 119644 dup strings (34075 unique), 119644 dup backing 
> arrays:
> 7914 of "true", 7912 of "-1", 6578 of "8", 5606 of "1", 2302 of "10", 1626 of 
> "174528", 1223 of "9", 970 of "171680", 837 of "269664", 657 of "133980"
> ... and 84009 more strings, of which 34065 are unique
> Also contains one-char strings: 42 of "7", 31 of "6", 20 of "4", 8 of "5", 5 
> of "2", 3 of "0"
>  <--  {j.u.HashMap}.values <-- 
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--  
> {j.u.TreeMap}.values <-- Java Local (j.u.TreeMap) [@6f084afa0,@73aac9e68]
>   233,384K (1.9%), 64601 dup strings (27295 unique), 64601 dup backing arrays:
> 4472 of "true", 4173 of "-1", 3798 of "1", 3591 of "8", 813 of "174528", 684 
> of "10", 623 of 
> "CQUJBQcFCAcGBwUFCgUIDAgEBwgFBQcHBwgGBwYEBQoLCggFCAYHBgcIBwkIDgcG ...[length 
> 4000]", 623 of 
> "BQcHBQUGBQgGBQcHCAUGCAkECQcFBQwGBgoJBQYHBQUFBQYKBQgIBgUJEgYFDAYJ ...[length 
> 4000]", 623 of 
> "BgUGBQgFCAYFCgYIBgUEBgQHBgUGCwYGBwYHBgkKBwYGBggIBwUHBgYGCgUJCQUG ...[length 
> 3560]", 623 of 
> "AAMAAAEAAAEAAQABAAEHAwAKAgAEAwAAAgAEAAMD ...[length 
> 4000]"
> ... and 44568 more strings, of which 27285 are unique
> Also contains one-char strings: 305 of "7", 301 of "0", 277 of "4", 146 of 
> "6", 29 of "2", 23 of "5", 19 of "9", 2 of "3"
>  <--  {j.u.HashMap}.values <-- 
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--  
> {j.u.ArrayList} <-- Java Local (j.u.ArrayList) 
> [@4f4cfbd10,@536122408,@726616778]
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (HIVE-16489) HMS wastes 26.4% of memory due to dup strings in metastore.api.Partition.parameters

[Misha Dmitriev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-16489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misha Dmitriev resolved HIVE-16489.
---
Resolution: Duplicate

> HMS wastes 26.4% of memory due to dup strings in 
> metastore.api.Partition.parameters
> ---
>
> Key: HIVE-16489
> URL: https://issues.apache.org/jira/browse/HIVE-16489
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Misha Dmitriev
>Assignee: Misha Dmitriev
>
> I've just analyzed an HMS heap dump. It turns out that it contains a lot of 
> duplicate strings, that waste 26.4% of the heap. Most of them come from 
> HashMaps referenced by 
> org.apache.hadoop.hive.metastore.api.Partition.parameters. Below is the 
> relevant section of the jxray (www.jxray.com) report. Looking at 
> Partition.java, I see that in the past somebody has already added code to 
> intern keys and values in the parameters table when it's first set up. 
> However, looks like when more key-value pairs are added, they are not 
> interned, and that probably explains the reason for all these duplicate 
> strings.
> {code}
> 6. DUPLICATE STRINGS
> Total strings: 3,273,557  Unique strings: 460,390  Duplicate values: 110,232  
> Overhead: 3,220,458K (26.4%)
> Top duplicate strings:
> Ovhd Num char[]s   Num objs   Value
>  46,088K (0.4%) 58715871  
> "HBa4rRAAGx2MEmludGVyZXN0cmF0ZXNwcmVhZBgM/wD/AP8AXqEAERYBFQAXIEAWuK0QAA1s
>  ...[length 4000]"
>  46,088K (0.4%) 58715871  
> "BQcHBQUGBQgGBQcHCAUGCAkECQcFBQwGBgoJBQYHBQUFBQYKBQgIBgUJEgYFDAYJBgcGBAcLBQYGCAgGCQYG
>  ...[length 4000]"
> ...
> ===
> 7. REFERENCE CHAINS FOR DUPLICATE STRINGS
>   2,326,150K (19.1%), 597058 dup strings (36386 unique), 597058 dup backing 
> arrays:
> 39949 of "-1", 39088 of "true", 28959 of "8", 20987 of "1", 18437 of "10", 
> 9583 of "9", 5908 of "269664", 5691 of "174528", 4598 of "133980", 4598 of 
> "BgUGBQgFCAYFCgYIBgUEBgQHBgUGCwYGBwYHBgkKBwYGBggIBwUHBgYGCgUJCQUG ...[length 
> 3560]"
> ... and 419200 more strings, of which 36376 are unique
> Also contains one-char strings: 217 of "6", 147 of "7", 91 of "4", 28 of "5", 
> 28 of "2", 21 of "0"
>  <--  {j.u.HashMap}.values <-- 
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--  
> {j.u.ArrayList} <-- 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result.success
>  <-- Java Local 
> (org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_result)
>  [@6e33618d8,@6eedb9a80,@6eedbad68,@6eedbc788] ... and 3 more GC roots
>   463,060K (3.8%), 119644 dup strings (34075 unique), 119644 dup backing 
> arrays:
> 7914 of "true", 7912 of "-1", 6578 of "8", 5606 of "1", 2302 of "10", 1626 of 
> "174528", 1223 of "9", 970 of "171680", 837 of "269664", 657 of "133980"
> ... and 84009 more strings, of which 34065 are unique
> Also contains one-char strings: 42 of "7", 31 of "6", 20 of "4", 8 of "5", 5 
> of "2", 3 of "0"
>  <--  {j.u.HashMap}.values <-- 
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--  
> {j.u.TreeMap}.values <-- Java Local (j.u.TreeMap) [@6f084afa0,@73aac9e68]
>   233,384K (1.9%), 64601 dup strings (27295 unique), 64601 dup backing arrays:
> 4472 of "true", 4173 of "-1", 3798 of "1", 3591 of "8", 813 of "174528", 684 
> of "10", 623 of 
> "CQUJBQcFCAcGBwUFCgUIDAgEBwgFBQcHBwgGBwYEBQoLCggFCAYHBgcIBwkIDgcG ...[length 
> 4000]", 623 of 
> "BQcHBQUGBQgGBQcHCAUGCAkECQcFBQwGBgoJBQYHBQUFBQYKBQgIBgUJEgYFDAYJ ...[length 
> 4000]", 623 of 
> "BgUGBQgFCAYFCgYIBgUEBgQHBgUGCwYGBwYHBgkKBwYGBggIBwUHBgYGCgUJCQUG ...[length 
> 3560]", 623 of 
> "AAMAAAEAAAEAAQABAAEHAwAKAgAEAwAAAgAEAAMD ...[length 
> 4000]"
> ... and 44568 more strings, of which 27285 are unique
> Also contains one-char strings: 305 of "7", 301 of "0", 277 of "4", 146 of 
> "6", 29 of "2", 23 of "5", 19 of "9", 2 of "3"
>  <--  {j.u.HashMap}.values <-- 
> org.apache.hadoop.hive.metastore.api.Partition.parameters <--  
> {j.u.ArrayList} <-- Java Local (j.u.ArrayList) 
> [@4f4cfbd10,@536122408,@726616778]
> ...
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-15209) Set hive.strict.checks.cartesian.product to false by default

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez updated HIVE-15209:
---
Resolution: Won't Fix
Status: Resolved  (was: Patch Available)

> Set hive.strict.checks.cartesian.product to false by default
> 
>
> Key: HIVE-15209
> URL: https://issues.apache.org/jira/browse/HIVE-15209
> Project: Hive
>  Issue Type: Bug
>  Components: Parser
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-15209.patch
>
>
> If we aim to make Hive compliant with SQL, w should disable this property by 
> default, as expressing a cartesian product, though inefficient, is perfectly 
> valid in SQL.
> Further, if we express complex predicates in the ON clause of a SQL query, we 
> might not be able to push these predicates to the join operator; however, we 
> should still be able to execute the query.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (HIVE-18384) ConcurrentModificationException in log4j2.x library

[Prasanth Jayachandran (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-18384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313762#comment-16313762
 ] 

Prasanth Jayachandran commented on HIVE-18384:
--

Encountered MSHADE-242 issue when upgrading the latest log4j2 version and when 
using -Dpackaging.minimizeJar=false in build.
Latest stable shade plugin seems to fix the issue.


> ConcurrentModificationException in log4j2.x library
> ---
>
> Key: HIVE-18384
> URL: https://issues.apache.org/jira/browse/HIVE-18384
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18384.1.patch, HIVE-18384.2.patch
>
>
> In one of the internal testing, observed the following exception
> {code}
> java.util.ConcurrentModificationException
>   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) 
> ~[?:1.8.0_152]
>   at java.util.ArrayList$Itr.next(ArrayList.java:859) ~[?:1.8.0_152]
>   at 
> java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042) 
> ~[?:1.8.0_152]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.appendCollection(ParameterFormatter.java:596)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.appendPotentiallyRecursiveValue(ParameterFormatter.java:504)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.recursiveDeepToString(ParameterFormatter.java:429)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.formatMessage2(ParameterFormatter.java:189)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterizedMessage.formatTo(ParameterizedMessage.java:224)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterizedMessage.getFormattedMessage(ParameterizedMessage.java:200)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.RingBufferLogEvent.setMessage(RingBufferLogEvent.java:126)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.RingBufferLogEvent.setValues(RingBufferLogEvent.java:104)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.RingBufferLogEventTranslator.translateTo(RingBufferLogEventTranslator.java:56)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.RingBufferLogEventTranslator.translateTo(RingBufferLogEventTranslator.java:34)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> com.lmax.disruptor.RingBuffer.translateAndPublish(RingBuffer.java:930) 
> ~[disruptor-3.3.0.jar:?]
>   at com.lmax.disruptor.RingBuffer.tryPublishEvent(RingBuffer.java:456) 
> ~[disruptor-3.3.0.jar:?]
>   at 
> org.apache.logging.log4j.core.async.AsyncLoggerDisruptor.tryPublish(AsyncLoggerDisruptor.java:190)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.AsyncLogger.publish(AsyncLogger.java:160) 
> ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.AsyncLogger.logWithThreadLocalTranslator(AsyncLogger.java:156)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.AsyncLogger.logMessage(AsyncLogger.java:126)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2011)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1884)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:189) 
> ~[log4j-slf4j-impl-2.6.2.jar:2.6.2]
>   at 
> org.apache.hadoop.hive.druid.security.KerberosHttpClient.inner_go(KerberosHttpClient.java:96)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> org.apache.hadoop.hive.druid.security.KerberosHttpClient.access$100(KerberosHttpClient.java:50)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> org.apache.hadoop.hive.druid.security.KerberosHttpClient$2.onSuccess(KerberosHttpClient.java:144)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> org.apache.hadoop.hive.druid.security.KerberosHttpClient$2.onSuccess(KerberosHttpClient.java:134)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> org.apache.hive.druid.com.google.common.util.concurrent.Futures$4.run(Futures.java:1181)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  ~[?:1.8.0_152]
>   at 
> 

[jira] [Updated] (HIVE-18384) ConcurrentModificationException in log4j2.x library

[Prasanth Jayachandran (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prasanth Jayachandran updated HIVE-18384:
-
Attachment: HIVE-18384.2.patch

> ConcurrentModificationException in log4j2.x library
> ---
>
> Key: HIVE-18384
> URL: https://issues.apache.org/jira/browse/HIVE-18384
> Project: Hive
>  Issue Type: Bug
>  Components: Logging
>Affects Versions: 3.0.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-18384.1.patch, HIVE-18384.2.patch
>
>
> In one of the internal testing, observed the following exception
> {code}
> java.util.ConcurrentModificationException
>   at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:909) 
> ~[?:1.8.0_152]
>   at java.util.ArrayList$Itr.next(ArrayList.java:859) ~[?:1.8.0_152]
>   at 
> java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042) 
> ~[?:1.8.0_152]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.appendCollection(ParameterFormatter.java:596)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.appendPotentiallyRecursiveValue(ParameterFormatter.java:504)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.recursiveDeepToString(ParameterFormatter.java:429)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterFormatter.formatMessage2(ParameterFormatter.java:189)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterizedMessage.formatTo(ParameterizedMessage.java:224)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.message.ParameterizedMessage.getFormattedMessage(ParameterizedMessage.java:200)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.RingBufferLogEvent.setMessage(RingBufferLogEvent.java:126)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.RingBufferLogEvent.setValues(RingBufferLogEvent.java:104)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.RingBufferLogEventTranslator.translateTo(RingBufferLogEventTranslator.java:56)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.RingBufferLogEventTranslator.translateTo(RingBufferLogEventTranslator.java:34)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> com.lmax.disruptor.RingBuffer.translateAndPublish(RingBuffer.java:930) 
> ~[disruptor-3.3.0.jar:?]
>   at com.lmax.disruptor.RingBuffer.tryPublishEvent(RingBuffer.java:456) 
> ~[disruptor-3.3.0.jar:?]
>   at 
> org.apache.logging.log4j.core.async.AsyncLoggerDisruptor.tryPublish(AsyncLoggerDisruptor.java:190)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.AsyncLogger.publish(AsyncLogger.java:160) 
> ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.AsyncLogger.logWithThreadLocalTranslator(AsyncLogger.java:156)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.core.async.AsyncLogger.logMessage(AsyncLogger.java:126)
>  ~[log4j-core-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2011)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at 
> org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1884)
>  ~[log4j-api-2.6.2.jar:2.6.2]
>   at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:189) 
> ~[log4j-slf4j-impl-2.6.2.jar:2.6.2]
>   at 
> org.apache.hadoop.hive.druid.security.KerberosHttpClient.inner_go(KerberosHttpClient.java:96)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> org.apache.hadoop.hive.druid.security.KerberosHttpClient.access$100(KerberosHttpClient.java:50)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> org.apache.hadoop.hive.druid.security.KerberosHttpClient$2.onSuccess(KerberosHttpClient.java:144)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> org.apache.hadoop.hive.druid.security.KerberosHttpClient$2.onSuccess(KerberosHttpClient.java:134)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> org.apache.hive.druid.com.google.common.util.concurrent.Futures$4.run(Futures.java:1181)
>  ~[hive-druid-handler-2.1.0.2.6.4.0-91.jar:2.1.0.2.6.4.0-91]
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  ~[?:1.8.0_152]
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  ~[?:1.8.0_152]
>   at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
> {code}
> The fix for this went into 2.9.1 LOG4J2-1988 onwards. Updating log4j to 
> latest 

[jira] [Commented] (HIVE-15640) Hive/Druid integration: null handling for metrics

[Jesus Camacho Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/HIVE-15640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313748#comment-16313748
 ] 

Jesus Camacho Rodriguez commented on HIVE-15640:


[~nishantbangarwa], I think this is fixed now? Can we close pointing to issue 
that fixed the issue? Thanks

> Hive/Druid integration: null handling for metrics
> -
>
> Key: HIVE-15640
> URL: https://issues.apache.org/jira/browse/HIVE-15640
> Project: Hive
>  Issue Type: Bug
>  Components: Druid integration
>Affects Versions: 2.2.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Critical
>
> Null values for metrics in Druid and Hive are not handled the same way (_0.0_ 
> vs _NULL_).
> In Druid:
> {code:sql}
> SELECT i_brand_id, floor_day(`__time`), max(ss_quantity), 
> sum(ss_wholesale_cost) as s
> FROM store_sales_sold_time_subset
> WHERE floor_day(`__time`) BETWEEN '1999-11-01 00:00:00' AND '1999-11-10 
> 00:00:00'
> GROUP BY i_brand_id, floor_day(`__time`)
> ORDER BY s
> LIMIT 10;
> OK
> 6015006   1999-11-03 00:00:00 0.0 0.0
> 9011009   1999-11-05 00:00:00 0.0 0.0
> 8003009   1999-11-03 00:00:00 11.01.029713897705
> 10005014  1999-11-05 00:00:00 86.01.10023841858
> 6008007   1999-11-09 00:00:00 81.01.370047683716
> 6003003   1999-11-08 00:00:00 45.01.60023841858
> 8008009   1999-11-08 00:00:00 98.01.710381469727
> 8015003   1999-11-02 00:00:00 10.01.740095367432
> 8004008   1999-11-10 00:00:00 45.01.759904632568
> 8009009   1999-11-07 00:00:00 81.01.769809265137
> {code}
> In Hive:
> {code:sql}
> SELECT i_brand_id, floor_day(`__time`), max(ss_quantity), 
> sum(ss_wholesale_cost) as s
> FROM store_sales_sold_time_subset_hive
> WHERE floor_day(`__time`) BETWEEN '1999-11-01 00:00:00' AND '1999-11-10 
> 00:00:00'
> GROUP BY i_brand_id, floor_day(`__time`)
> ORDER BY s
> LIMIT 10;
> OK
> 6015006   1999-11-03 00:00:00 NULLNULL
> 9011009   1999-11-05 00:00:00 NULLNULL
> 8003009   1999-11-03 00:00:00 11  1.03
> 10005014  1999-11-05 00:00:00 86  1.1
> 6008007   1999-11-09 00:00:00 81  1.37
> 6003003   1999-11-08 00:00:00 45  1.6
> 8008009   1999-11-08 00:00:00 98  1.71
> 8015003   1999-11-02 00:00:00 10  1.74
> 8004008   1999-11-10 00:00:00 45  1.76
> 8009009   1999-11-07 00:00:00 81  1.77
> {code}
> However, for Druid dimensions, NULL values seem to be handled properly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Assigned] (HIVE-18386) Create dummy materialized views registry and make it configurable

[Jesus Camacho Rodriguez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jesus Camacho Rodriguez reassigned HIVE-18386:
--


> Create dummy materialized views registry and make it configurable
> -
>
> Key: HIVE-18386
> URL: https://issues.apache.org/jira/browse/HIVE-18386
> Project: Hive
>  Issue Type: Improvement
>  Components: Materialized views
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>
> HiveMaterializedViewsRegistry keeps the materialized views plans in memory to 
> have quick access when queries are planned. For debugging purposes, we will 
> create a dummy materialized views registry that forwards all calls to 
> metastore and make the choice configurable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal updated HIVE-18385:
--
Attachment: hive.log

> mergejoin fails with java.lang.IllegalStateException
> 
>
> Key: HIVE-18385
> URL: https://issues.apache.org/jira/browse/HIVE-18385
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
> Attachments: hive.log
>
>
> mergejoin test fails with java.lang.IllegalStateException when run in 
> MiniLlapLocal.
> This is the query for which it fails,
> [ERROR]   TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed 
> with error code = 2 running "
> select count(*) from tab a join tab_part b on a.key = b.key join src1 c on 
> a.value = c.value" fname=mergejoin.q 
> This is the stack trace,
> failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, 
> vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: 
> ORC split generation failed with exception: java.lang.IllegalStateException: 
> Failed to retrieve dynamic value for RS_12_a_key_min
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at 
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at 
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at 
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: 
> java.lang.IllegalStateException: Failed to retrieve dynamic value for 
> RS_12_a_key_min
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778)
> ... 17 more
> Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value 
> for RS_12_a_key_min
> at 
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142)
> at 
> org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97)
> at 
> org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93)
> at 
> org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120)
> at 
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553)
> at 
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463)
> at 
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1476)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1261)
> 

[jira] [Assigned] (HIVE-18385) mergejoin fails with java.lang.IllegalStateException

[Deepak Jaiswal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Deepak Jaiswal reassigned HIVE-18385:
-


> mergejoin fails with java.lang.IllegalStateException
> 
>
> Key: HIVE-18385
> URL: https://issues.apache.org/jira/browse/HIVE-18385
> Project: Hive
>  Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>
> mergejoin test fails with java.lang.IllegalStateException when run in 
> MiniLlapLocal.
> This is the query for which it fails,
> [ERROR]   TestMiniLlapLocalCliDriver.testCliDriver:59 Client execution failed 
> with error code = 2 running "
> select count(*) from tab a join tab_part b on a.key = b.key join src1 c on 
> a.value = c.value" fname=mergejoin.q 
> This is the stack trace,
> failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: b initializer failed, 
> vertex=vertex_1515180518813_0001_42_05 [Map 8], java.lang.RuntimeException: 
> ORC split generation failed with exception: java.lang.IllegalStateException: 
> Failed to retrieve dynamic value for RS_12_a_key_min
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1784)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1872)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.addSplitsForGroup(HiveInputFormat.java:499)
> at 
> org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:684)
> at 
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
> at 
> org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
> at 
> com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
> at 
> com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
> at 
> com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: 
> java.lang.IllegalStateException: Failed to retrieve dynamic value for 
> RS_12_a_key_min
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1778)
> ... 17 more
> Caused by: java.lang.IllegalStateException: Failed to retrieve dynamic value 
> for RS_12_a_key_min
> at 
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:142)
> at 
> org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97)
> at 
> org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93)
> at 
> org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120)
> at 
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553)
> at 
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463)
> at 
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:440)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.isStripeSatisfyPredicate(OrcInputFormat.java:2163)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripesInternal(OrcInputFormat.java:2140)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.pickStripes(OrcInputFormat.java:2131)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.access$3000(OrcInputFormat.java:157)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.callInternal(OrcInputFormat.java:1476)
> at 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$SplitGenerator.access$2700(OrcInputFormat.java:1261)
> at 
> 

[jira] [Updated] (HIVE-18214) Flaky test: TestSparkClient

[Sahil Takiar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-18214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sahil Takiar updated HIVE-18214:

Attachment: HIVE-18214.2.patch

> Flaky test: TestSparkClient
> ---
>
> Key: HIVE-18214
> URL: https://issues.apache.org/jira/browse/HIVE-18214
> Project: Hive
>  Issue Type: Sub-task
>  Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-18214.1.patch, HIVE-18214.2.patch
>
>
> Looks like there is a race condition in {{TestSparkClient#runTest}}. The test 
> creates a {{RemoteDriver}} in memory, which creates a {{JavaSparkContext}}. A 
> new {{JavaSparkContext}} is created for each test that is run. There is a 
> race condition where the {{RemoteDriver}} isn't given enough time to 
> shutdown, so when the next test starts running it creates another 
> {{JavaSparkContext}} which causes an exception like 
> {{org.apache.spark.SparkException: Only one SparkContext may be running in 
> this JVM (see SPARK-2243)}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)