date:20181027

[jira] [Commented] (HIVE-20803) Hive external table can't read S3 file containing timestamp partition

2018-10-27 Thread t oo (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665999#comment-16665999
 ] 

t oo commented on HIVE-20803:
-

workaround is to write a different path to s3 that url encodes the : colon 
character. The space character can stay

> Hive external table can't read S3 file containing timestamp partition
> -
>
> Key: HIVE-20803
> URL: https://issues.apache.org/jira/browse/HIVE-20803
> Project: Hive
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.2
>Reporter: t oo
>Priority: Major
>
> SparkThriftServer can select * from the table fine and get data. But 
> HiveServer2 throws below error on select *:
>  
> hive.msck.path.validation = ignore in hive-site.xml
> then ran msck repair my_sch.h_l
> aws s3 ls s3://priv1/priv2/H_L/ --recursive
> 2018-10-18 03:00:56 2474983 
> priv1/priv2/H_L/part_dt=20180309/part_src=xyz/part_src_file=MY_LOC/part_ldts=2018-10-18
>  02:59:46/part-0-2536ca01-243c-4220-8e55-6869a045fba2.snappy.parquet
> show create table my_sch.h_l;
> ++
> | createtab_stmt |
> ++
> | CREATE EXTERNAL TABLE `my_sch.h_l`( |
> | `xy_hkey_h_l` binary, |
> | `xy_rtts` timestamp, |
> | `xy_rsrc` string, |
> | `xy_bkcc` string, |
> | `xy_mltid` string, |
> | `location_id` bigint) |
> | PARTITIONED BY ( |
> | `part_dt` string, |
> | `part_src` string, |
> | `part_src_file` string, |
> | `part_ldts` timestamp) |
> | ROW FORMAT SERDE |
> | 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' |
> | STORED AS INPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' |
> | OUTPUTFORMAT |
> | 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' |
> | LOCATION |
> | 's3a://priv1/priv2/H_L' |
> | TBLPROPERTIES ( |
> | 'spark.sql.partitionProvider'='catalog', |
> | 'spark.sql.sources.schema.numPartCols'='4', |
> | 'spark.sql.sources.schema.numParts'='1', |
> | 
> 'spark.sql.sources.schema.part.0'='\{"type":"struct","fields":[{"name":"xy_hkey_h_l","type":"binary","nullable":true,"metadata":{}},\{"name":"xy_rtts","type":"timestamp","nullable":true,"metadata":{}},\{"name":"xy_rsrc","type":"string","nullable":true,"metadata":{}},\{"name":"xy_bkcc","type":"string","nullable":true,"metadata":{}},\{"name":"xy_mltid","type":"string","nullable":true,"metadata":{}},\{"name":"location_id","type":"long","nullable":true,"metadata":{}},\{"name":"part_dt","type":"string","nullable":true,"metadata":{}},\{"name":"part_src","type":"string","nullable":true,"metadata":{}},\{"name":"part_src_file","type":"string","nullable":true,"metadata":{}},\{"name":"part_ldts","type":"timestamp","nullable":true,"metadata":{}}]}',
>  |
> | 'spark.sql.sources.schema.partCol.0'='part_dt', |
> | 'spark.sql.sources.schema.partCol.1'='part_src', |
> | 'spark.sql.sources.schema.partCol.2'='part_src_file', |
> | 'spark.sql.sources.schema.partCol.3'='part_ldts', |
> | 'transient_lastDdlTime'='1540421484') |
> ++
>  select * from my_sch.h_l limit 5;
> Error: java.io.IOException: java.lang.IllegalArgumentException: 
> java.net.URISyntaxException: Relative path in absolute URI: 
> part_ldts=2018-10-18 02:59:46 (state=,code=0)
> org.apache.hive.service.cli.HiveSQLException: java.io.IOException: 
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative 
> path in absolute URI: part_ldts=2018-10-18 02:59:46
>  at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267)
>  at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253)
>  at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:374)
>  at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:53)
>  at 
> org.apache.hive.beeline.IncrementalRowsWithNormalization.(IncrementalRowsWithNormalization.java:50)
>  at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2192)
>  at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1009)
>  at org.apache.hive.beeline.Commands.execute(Commands.java:1205)
>  at org.apache.hive.beeline.Commands.sql(Commands.java:1134)
>  at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314)
>  at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178)
>  at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033)
>  at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519)
>  at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Meth

[jira] [Commented] (HIVE-20259) Cleanup of results cache directory

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1004#comment-1004
 ] 

Hive QA commented on HIVE-20259:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
29s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
38s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
14s{color} | {color:red} common: The patch generated 3 new + 426 unchanged - 0 
fixed = 429 total (was 426) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14649/dev-support/hive-personality.sh
 |
| git revision | master / 1002e89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14649/yetus/diff-checkstyle-common.txt
 |
| modules | C: common ql U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14649/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Cleanup of results cache directory
> --
>
> Key: HIVE-20259
> URL: https://issues.apache.org/jira/browse/HIVE-20259
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20259.1.patch, HIVE-20259.2.patch
>
>
> The query results cache directory is currently deleted at process exit. This 
> does not work in the case of a kill -9 or a sudden process exit of Hive. 
> There should be some cleanup mechanism in place to take care of any old cache 
> directories that were not deleted at process exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20259) Cleanup of results cache directory

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1015#comment-1015
 ] 

Hive QA commented on HIVE-20259:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945635/HIVE-20259.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15507 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning 
(batchId=331)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14649/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14649/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14649/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12945635 - PreCommit-HIVE-Build

> Cleanup of results cache directory
> --
>
> Key: HIVE-20259
> URL: https://issues.apache.org/jira/browse/HIVE-20259
> Project: Hive
>  Issue Type: Sub-task
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-20259.1.patch, HIVE-20259.2.patch
>
>
> The query results cache directory is currently deleted at process exit. This 
> does not work in the case of a kill -9 or a sudden process exit of Hive. 
> There should be some cleanup mechanism in place to take care of any old cache 
> directories that were not deleted at process exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1139#comment-1139
 ] 

Hive QA commented on HIVE-20617:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
48s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
28s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
40s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
36s{color} | {color:red} ql: The patch generated 2 new + 238 unchanged - 12 
fixed = 240 total (was 250) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
51s{color} | {color:red} ql generated 4 new + 2313 unchanged - 4 fixed = 2317 
total (was 2317) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
13s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 40s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
 Object, TypeInfo) invokes inefficient new Byte(String) constructor; use 
Byte.valueOf(String) instead  At TypeCheckProcFactory.java:new Byte(String) 
constructor; use Byte.valueOf(String) instead  At 
TypeCheckProcFactory.java:[line 1448] |
|  |  
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
 Object, TypeInfo) invokes inefficient new Integer(String) constructor; use 
Integer.valueOf(String) instead  At TypeCheckProcFactory.java:new 
Integer(String) constructor; use Integer.valueOf(String) instead  At 
TypeCheckProcFactory.java:[line 1440] |
|  |  
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
 Object, TypeInfo) invokes inefficient new Long(String) constructor; use 
Long.valueOf(String) instead  At TypeCheckProcFactory.java:new Long(String) 
constructor; use Long.valueOf(String) instead  At 
TypeCheckProcFactory.java:[line 1442] |
|  |  
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo,
 Object, TypeInfo) invokes inefficient new Short(String) constructor; use 
Short.valueOf(String) instead  At TypeCheckProcFactory.java:new Short(String) 
constructor; use Short.valueOf(String) instead  At 
TypeCheckProcFactory.java:[line 1450] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |

[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1158#comment-1158
 ] 

Hive QA commented on HIVE-20617:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945683/HIVE-20617.12.patch

{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 15512 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_multi_single_reducer3]
 (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_non_constant_in_expr]
 (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_struct_in] 
(batchId=49)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1]
 (batchId=182)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
 (batchId=170)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[is_distinct_from]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_column_in]
 (batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_column_in_single]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_struct_in]
 (batchId=170)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_in_2] 
(batchId=98)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_simple_select] 
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_multi_single_reducer3]
 (batchId=132)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar] 
(batchId=128)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14650/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14650/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14650/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12945683 - PreCommit-HIVE-Build

> Fix type of constants in IN expressions to have correct type
> 
>
> Key: HIVE-20617
> URL: https://issues.apache.org/jira/browse/HIVE-20617
> Project: Hive
>  Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch, 
> HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch, 
> HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, 
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, 
> HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, 
> HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch, 
> HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch
>
>
> In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the 
> comparision in UDFIn may fail because if a or b is of char/varchar type the 
> constants will retain string type - especially after PointlookupOptimizer 
> compaction.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1209#comment-1209
 ] 

Hive QA commented on HIVE-20807:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945662/HIVE-20807.02.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14651/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14651/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14651/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-10-27 19:41:20.336
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14651/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-10-27 19:41:20.339
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 
9.3.25.v20180904 (Laszlo Bodor  via Thejas Nair)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 
9.3.25.v20180904 (Laszlo Bodor  via Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-10-27 19:41:21.152
+ rm -rf ../yetus_PreCommit-HIVE-Build-14651
+ mkdir ../yetus_PreCommit-HIVE-Build-14651
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14651
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14651/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: a/bin/ext/llapstatus.sh: does not exist in index
error: 
a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapSliderUtils.java: 
does not exist in index
error: 
a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusOptionsProcessor.java:
 does not exist in index
error: 
a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java:
 does not exist in index
error: 
a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusHelpers.java:
 does not exist in index
error: 
a/llap-server/src/test/org/apache/hadoop/hive/llap/cli/TestLlapStatusServiceDriver.java:
 does not exist in index
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:3395: new blank line at EOF.
+
/data/hiveptest/working/scratch/build.patch:3625: new blank line at EOF.
+
warning: 2 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q 
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc3519613450051859971.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc3519613450051859971.exe, 
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
 
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator  Version 3.5.2
protoc-jar: executing: [/tmp/protoc4407913131038094891.exe, --version]
libprotoc 2.5.0
ANTLR Parser Generator  Version 3.5.2
Output file 
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/m

[jira] [Assigned] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

2018-10-27 Thread Eugene Koifman (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman reassigned HIVE-20825:
-

Assignee: Eugene Koifman

> Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in 
> length) causing the "Not a valid ORC file" error
> --
>
> Key: HIVE-20825
> URL: https://issues.apache.org/jira/browse/HIVE-20825
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, ORC, Transactions
>Affects Versions: 2.2.0, 2.3.1, 2.3.2
> Environment: Hive 2.3.x on Amazon EMR 5.8.0 to 5.18.0
>Reporter: Tom Zeng
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: hive-merge-invalid-orc-repro.hql, 
> hive-merge-invalid-orc-repro.log
>
>
> When using Hive ACID Merge (supported with the ORC format) to update/insert 
> data, bucket files with 0 byte or 3 bytes (file content is three character: 
> ORC) are generated during MERGE INTO operations which finish with no errors. 
> Subsequent queries on the base table will get "Not a valid ORC file" error.
>  
> The following script can be used to reproduce the issue:
> set hive.auto.convert.join=false;
> set hive.enforce.bucketing=true;
> set hive.exec.dynamic.partition.mode = nonstrict;
> set hive.support.concurrency=true;
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> drop table if exists mergedelta_txt_1;
> drop table if exists mergedelta_txt_2;
> CREATE TABLE mergedelta_txt_1 (
> id_str varchar(12), time_key int, value bigint)
> PARTITIONED BY (date_key int)
> ROW FORMAT DELIMITED
> STORED AS TEXTFILE;
> CREATE TABLE mergedelta_txt_2 (
> id_str varchar(12), time_key int, value bigint)
> PARTITIONED BY (date_key int)
> ROW FORMAT DELIMITED
> STORED AS TEXTFILE;
> INSERT INTO TABLE mergedelta_txt_1
> partition(date_key=20170103)
> VALUES
>  ("AB94LIENR0",46700,12345676836978),
>  ("AB94LIENR1",46825,12345676836978),
>  ("AB94LIENS0",46709,12345676836978),
>  ("AB94LIENS1",46834,12345676836978),
>  ("AB94LIENT0",46709,12345676836978),
>  ("AB94LIENT1",46834,12345676836978),
>  ("AB94LIENU0",46718,12345676836978),
>  ("AB94LIENU1",46844,12345676836978),
>  ("AB94LIENV0",46719,12345676836978),
>  ("AB94LIENV1",46844,12345676836978),
>  ("AB94LIENW0",46728,12345676836978),
>  ("AB94LIENW1",46854,12345676836978),
>  ("AB94LIENX0",46728,12345676836978),
>  ("AB94LIENX1",46854,12345676836978),
>  ("AB94LIENY0",46737,12345676836978),
>  ("AB94LIENY1",46863,12345676836978),
>  ("AB94LIENZ0",46738,12345676836978),
>  ("AB94LIENZ1",46863,12345676836978),
>  ("AB94LIERA0",47176,12345676836982),
>  ("AB94LIERA1",47302,12345676836982);
> INSERT INTO TABLE mergedelta_txt_2
> partition(date_key=20170103)
> VALUES 
>  ("AB94LIENT1",46834,12345676836978),
>  ("AB94LIENU0",46718,12345676836978),
>  ("AB94LIENU1",46844,12345676836978),
>  ("AB94LIENV0",46719,12345676836978),
>  ("AB94LIENV1",46844,12345676836978),
>  ("AB94LIENW0",46728,12345676836978),
>  ("AB94LIENW1",46854,12345676836978),
>  ("AB94LIENX0",46728,12345676836978),
>  ("AB94LIENX1",46854,12345676836978),
>  ("AB94LIENY0",46737,12345676836978),
>  ("AB94LIENY1",46863,12345676836978),
>  ("AB94LIENZ0",46738,12345676836978),
>  ("AB94LIENZ1",46863,12345676836978),
>  ("AB94LIERA0",47176,12345676836982),
>  ("AB94LIERA1",47302,12345676836982),
>  ("AB94LIERA2",47418,12345676836982),
>  ("AB94LIERB0",47176,12345676836982),
>  ("AB94LIERB1",47302,12345676836982),
>  ("AB94LIERB2",47418,12345676836982),
>  ("AB94LIERC0",47185,12345676836982);
> DROP TABLE IF EXISTS mergebase_1;
> CREATE TABLE mergebase_1 (
> id_str varchar(12) , time_key int , value bigint)
> PARTITIONED BY (date_key int)
> CLUSTERED BY (id_str,time_key) INTO 32 BUCKETS
> STORED AS ORC
> TBLPROPERTIES (
>  'orc.compress'='SNAPPY',
>  'pk_columns'='id_str,date_key,time_key',
>  'NO_AUTO_COMPACTION'='true',
>  'transactional'='true');
> MERGE INTO mergebase_1 AS base
> USING (SELECT * 
>  FROM (
>  SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
> id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
>  FROM mergedelta_txt_1
>  DISTRIBUTE BY date_key
>  ) rankedtbl 
>  WHERE rankedtbl.rk=1
> ) AS delta
> ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
> delta.time_key=base.time_key
> WHEN MATCHED THEN UPDATE SET value=delta.value
> WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
> delta.value, delta.date_key);
> MERGE INTO mergebase_1 AS base
> USING (SELECT * 
>  FROM (
>  SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
> id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
>  FROM mergedelta_txt_2
>  DISTRIBUTE BY date_key
>  ) rankedtbl 
>  WHERE rankedtbl.rk=1
> ) AS d

[jira] [Updated] (HIVE-20486) Kafka: Use Row SerDe + vectorization

2018-10-27 Thread slim bouguerra (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

slim bouguerra updated HIVE-20486:
--
Attachment: HIVE-20486.4.patch

> Kafka: Use Row SerDe + vectorization
> 
>
> Key: HIVE-20486
> URL: https://issues.apache.org/jira/browse/HIVE-20486
> Project: Hive
>  Issue Type: Improvement
>  Components: kafka integration
>Reporter: Gopal V
>Assignee: slim bouguerra
>Priority: Major
>  Labels: kafka, vectorization
> Fix For: 4.0.0
>
> Attachments: HIVE-20486.3.patch, HIVE-20486.3.patch, 
> HIVE-20486.4.patch, HIVE-20486.patch
>
>
> KafkaHandler returns unvectorized rows which causes the operators downstream 
> to be slower and sub-optimal.
> Hive has a vectorization shim which allows Kafka streams without complex 
> projections to be wrapped into a vectorized reader via 
> {{hive.vectorized.use.row.serde.deserialize}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic

2018-10-27 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20778:
---
Status: Open  (was: Patch Available)

> Join reordering may not be triggered if all joins in plan are created by 
> decorrelation logic
> 
>
> Key: HIVE-20778
> URL: https://issues.apache.org/jira/browse/HIVE-20778
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, 
> HIVE-20778.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic

2018-10-27 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20778:
---
Attachment: HIVE-20778.3.patch

> Join reordering may not be triggered if all joins in plan are created by 
> decorrelation logic
> 
>
> Key: HIVE-20778
> URL: https://issues.apache.org/jira/browse/HIVE-20778
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, 
> HIVE-20778.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join

2018-10-27 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg reassigned HIVE-20826:
--


> Enhance HiveSemiJoin rule to convert join + group by on left side to Left 
> Semi Join
> ---
>
> Key: HIVE-20826
> URL: https://issues.apache.org/jira/browse/HIVE-20826
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
>
> Currently HiveSemiJoin rule looks for pattern where group by is on right side.
> We can convert joins which have group by on left side (assuming group by keys 
> are same as join keys and none of the columns are being projected from left 
> side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as:
> {code:sql}
> explain select pp.p_partkey from (select distinct p_name from part) p join 
> part pp on pp.p_name = p.p_name;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic

2018-10-27 Thread Vineet Garg (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vineet Garg updated HIVE-20778:
---
Status: Patch Available  (was: Open)

Updated the golden files. Subquery tests have join reordering and as a result 
HiveSemiJoin rule, to covert to Left Semi Join, is not kicking in from some 
tests. There are other tests which now have Semi Join.

I have opened HIVE-20826 to improve HiveSemiJoin rule.

> Join reordering may not be triggered if all joins in plan are created by 
> decorrelation logic
> 
>
> Key: HIVE-20778
> URL: https://issues.apache.org/jira/browse/HIVE-20778
> Project: Hive
>  Issue Type: Bug
>  Components: CBO
>Reporter: Jesus Camacho Rodriguez
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, 
> HIVE-20778.3.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method

2018-10-27 Thread Vihang Karajgaonkar (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vihang Karajgaonkar updated HIVE-20740:
---
Attachment: HIVE-20740.05.patch

> Remove global lock in ObjectStore.setConf method
> 
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, 
> HIVE-20740.04.patch, HIVE-20740.05.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other 
> clients in concurrent workloads.
> {code}
> @Override
>   @SuppressWarnings("nls")
>   public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there 
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
>   isInitialized = false;
>   this.conf = conf;
>   this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, 
> ConfVars.HIVE_TXN_STATS_ENABLED);
>   configureSSL(conf);
>   Properties propsFromConf = getDataSourceProps(conf);
>   boolean propsChanged = !propsFromConf.equals(prop);
>   if (propsChanged) {
> if (pmf != null){
>   clearOutPmfClassLoaderCache(pmf);
>   if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
>   }
> }
> pmf = null;
> prop = null;
>   }
>   assert(!isActiveTransaction());
>   shutdown();
>   // Always want to re-create pm as we don't know if it were created by 
> the
>   // most recent instance of the pmf
>   pm = null;
>   directSql = null;
>   expressionProxy = null;
>   openTrasactionCalls = 0;
>   currentTransaction = null;
>   transactionStatus = TXN_STATUS.NO_STATE;
>   initialize(propsFromConf);
>   String partitionValidationRegex =
>   MetastoreConf.getVar(this.conf, 
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
>   if (partitionValidationRegex != null && 
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern = 
> Pattern.compile(partitionValidationRegex);
>   } else {
> partitionValidationPattern = null;
>   }
>   // Note, if metrics have not been initialized this will return null, 
> which means we aren't
>   // using metrics.  Thus we should always check whether this is non-null 
> before using.
>   MetricRegistry registry = Metrics.getRegistry();
>   if (registry != null) {
> directSqlErrors = 
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
>   }
>   this.batchSize = MetastoreConf.getIntVar(conf, 
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
>   if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
>   } else {
> LOG.debug("Initialized ObjectStore");
>   }
> } finally {
>   pmfPropLock.unlock();
> }
>   }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new 
> connection to HMS which is trying to instantiate ObjectStore. We should 
> either remove the lock or reduce the scope of the lock so that it is held for 
> a very small amount of time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method

2018-10-27 Thread Vihang Karajgaonkar (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1237#comment-1237
 ] 

Vihang Karajgaonkar commented on HIVE-20740:


Spent some more time on this. Based on what I understand, there is a race in 
the existing code which could lead to a NPE. Eg.

Thread A calls setConf method with a new datasource properties. Thread A 
acquires {{proplock}} and sets {{pmf = null}}

Meanwhile Thread B is in {{getPMF}} method which even though is static 
synchronized doesn't know if pmf is being updated by some other thread in 
{{setConf}} object. This could lead to a NPE on Thread B. Or some error on 
Thread A when it tries to get {{PersistenceManager}} from unitialized pmf.

The good news this is very unlikely to happen. When datasource properties are 
changed, it needs a restart of HMS service which means there are no other 
threads requesting {{pmf}} until HMS up and running.

But still having {{pmf}} and {{props}} objects as private members which need to 
be synchronized when updating can be handled more cleaning if we move them to a 
separate wrapper class which deals with these details. Latest patch refactors 
some of this code and hopefully makes is cleaner than the current 
implementation.

> Remove global lock in ObjectStore.setConf method
> 
>
> Key: HIVE-20740
> URL: https://issues.apache.org/jira/browse/HIVE-20740
> Project: Hive
>  Issue Type: Improvement
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>Priority: Major
> Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, 
> HIVE-20740.04.patch, HIVE-20740.05.patch
>
>
> The ObjectStore#setConf method has a global lock which can block other 
> clients in concurrent workloads.
> {code}
> @Override
>   @SuppressWarnings("nls")
>   public void setConf(Configuration conf) {
> // Although an instance of ObjectStore is accessed by one thread, there 
> may
> // be many threads with ObjectStore instances. So the static variables
> // pmf and prop need to be protected with locks.
> pmfPropLock.lock();
> try {
>   isInitialized = false;
>   this.conf = conf;
>   this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, 
> ConfVars.HIVE_TXN_STATS_ENABLED);
>   configureSSL(conf);
>   Properties propsFromConf = getDataSourceProps(conf);
>   boolean propsChanged = !propsFromConf.equals(prop);
>   if (propsChanged) {
> if (pmf != null){
>   clearOutPmfClassLoaderCache(pmf);
>   if (!forTwoMetastoreTesting) {
> // close the underlying connection pool to avoid leaks
> pmf.close();
>   }
> }
> pmf = null;
> prop = null;
>   }
>   assert(!isActiveTransaction());
>   shutdown();
>   // Always want to re-create pm as we don't know if it were created by 
> the
>   // most recent instance of the pmf
>   pm = null;
>   directSql = null;
>   expressionProxy = null;
>   openTrasactionCalls = 0;
>   currentTransaction = null;
>   transactionStatus = TXN_STATUS.NO_STATE;
>   initialize(propsFromConf);
>   String partitionValidationRegex =
>   MetastoreConf.getVar(this.conf, 
> ConfVars.PARTITION_NAME_WHITELIST_PATTERN);
>   if (partitionValidationRegex != null && 
> !partitionValidationRegex.isEmpty()) {
> partitionValidationPattern = 
> Pattern.compile(partitionValidationRegex);
>   } else {
> partitionValidationPattern = null;
>   }
>   // Note, if metrics have not been initialized this will return null, 
> which means we aren't
>   // using metrics.  Thus we should always check whether this is non-null 
> before using.
>   MetricRegistry registry = Metrics.getRegistry();
>   if (registry != null) {
> directSqlErrors = 
> Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS);
>   }
>   this.batchSize = MetastoreConf.getIntVar(conf, 
> ConfVars.RAWSTORE_PARTITION_BATCH_SIZE);
>   if (!isInitialized) {
> throw new RuntimeException(
> "Unable to create persistence manager. Check dss.log for details");
>   } else {
> LOG.debug("Initialized ObjectStore");
>   }
> } finally {
>   pmfPropLock.unlock();
> }
>   }
> {code}
> The {{pmfPropLock}} is a static object and it disallows any other new 
> connection to HMS which is trying to instantiate ObjectStore. We should 
> either remove the lock or reduce the scope of the lock so that it is held for 
> a very small amount of time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20804) Further improvements to group by optimization with constraints

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1246#comment-1246
 ] 

Hive QA commented on HIVE-20804:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
36s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
43s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
34s{color} | {color:red} ql: The patch generated 7 new + 6 unchanged - 0 fixed 
= 13 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  3m 
49s{color} | {color:red} ql generated 1 new + 2317 unchanged - 0 fixed = 2318 
total (was 2317) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 25s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
|  |  Dead store to mapInToOutPos in 
org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelOptUtil$CardinalityChange.isCardinalitySameAsSource(HiveProject,
 ImmutableBitSet)  At 
HiveRelOptUtil.java:org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelOptUtil$CardinalityChange.isCardinalitySameAsSource(HiveProject,
 ImmutableBitSet)  At HiveRelOptUtil.java:[line 808] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14652/dev-support/hive-personality.sh
 |
| git revision | master / 1002e89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14652/yetus/diff-checkstyle-ql.txt
 |
| findbugs | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14652/yetus/new-findbugs-ql.html
 |
| modules | C: ql U: ql |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14652/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Further improvements to group by optimization with constraints
> --
>
> Key: HIVE-20804
> URL: https://issues.apache.org/jira/browse/HIVE-20804
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20804.1.patch, HIVE-20804.2.patch
>
>
> Continuation of HIVE-17043



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-27 Thread Rajesh Balamohan (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1248#comment-1248
 ] 

Rajesh Balamohan commented on HIVE-20816:
-

Closing this for now. I will reopen it when I generate smaller dataset for this.

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

2018-10-27 Thread Rajesh Balamohan (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Balamohan resolved HIVE-20816.
-
Resolution: Duplicate

> FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
> --
>
> Key: HIVE-20816
> URL: https://issues.apache.org/jira/browse/HIVE-20816
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Affects Versions: 2.3.2
>Reporter: Rajesh Balamohan
>Priority: Minor
>  Labels: fastdecimal
> Attachments: HIVE-20816.1.patch
>
>
> {noformat}
> with t1 as (
> ...
> ...
> )
> select id, max(abs(c1))) from t1 group by id;
> {noformat}
> throws the following exception
> {noformat}
> g.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
>  at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
>  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711)
> 
> ...
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: 
> Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123)
> ... 18 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: 
> java.lang.RuntimeException: Unexpected #3
> at 
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047)
> at 
> org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067)
> ... 19 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20804) Further improvements to group by optimization with constraints

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1249#comment-1249
 ] 

Hive QA commented on HIVE-20804:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945651/HIVE-20804.2.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 15507 tests passed

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14652/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14652/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14652/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12945651 - PreCommit-HIVE-Build

> Further improvements to group by optimization with constraints
> --
>
> Key: HIVE-20804
> URL: https://issues.apache.org/jira/browse/HIVE-20804
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20804.1.patch, HIVE-20804.2.patch
>
>
> Continuation of HIVE-17043



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1259#comment-1259
 ] 

Hive QA commented on HIVE-14557:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12934433/HIVE-14557.2.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14653/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14653/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14653/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-10-28 02:15:41.523
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ 
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-14653/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-10-28 02:15:41.526
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 
9.3.25.v20180904 (Laszlo Bodor  via Thejas Nair)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 
9.3.25.v20180904 (Laszlo Bodor  via Thejas Nair)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-10-28 02:15:42.166
+ rm -rf ../yetus_PreCommit-HIVE-Build-14653
+ mkdir ../yetus_PreCommit-HIVE-Build-14653
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-14653
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14653/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh 
/data/hiveptest/working/scratch/build.patch
error: 
a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java:
 does not exist in index
error: 
a/ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out: does 
not exist in index
error: patch failed: 
ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out:29
Falling back to three-way merge...
Applied patch to 
'ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out' with 
conflicts.
Going to apply patch with: git apply -p1
/data/hiveptest/working/scratch/build.patch:92: trailing whitespace.
$hdt$_0:$hdt$_1:src2 
/data/hiveptest/working/scratch/build.patch:97: trailing whitespace.
$hdt$_0:$hdt$_1:src2 
/data/hiveptest/working/scratch/build.patch:177: trailing whitespace.
$INTNAME1 
/data/hiveptest/working/scratch/build.patch:182: trailing whitespace.
$INTNAME1 
error: patch failed: 
ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out:29
Falling back to three-way merge...
Applied patch to 
'ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out' with 
conflicts.
U ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out
warning: 4 lines add whitespace errors.
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-14653
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12934433 - PreCommit-HIVE-Build

> Nullpointer When both SkewJoin  and Mapjoin Enabled
> ---
>
> Key: HIVE-14557
> URL: https://issues.apache.org/jira/browse/HIVE-14557
> Project: Hive
>  Issue Type: Bug
>  Components: Physical Optimizer
>Affects Versions: 1.1.0, 2.1.0
>Reporter: Nemon Lou
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-14557.2.patch, HIVE-14557.patch
>
>
> The following sql fail

[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1274#comment-1274
 ] 

Hive QA commented on HIVE-20807:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945662/HIVE-20807.02.patch

{color:red}ERROR:{color} -1 due to build exiting with an error

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14654/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14654/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14654/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL 
https://issues.apache.org/jira/secure/attachment/12945662/HIVE-20807.02.patch 
was found in seen patch url's cache and a test was probably run already on it. 
Aborting...
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12945662 - PreCommit-HIVE-Build

> Refactor LlapStatusServiceDriver
> 
>
> Key: HIVE-20807
> URL: https://issues.apache.org/jira/browse/HIVE-20807
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Affects Versions: 4.0.0
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20807.01.patch, HIVE-20807.02.patch
>
>
> LlapStatusServiceDriver is the class used to determine if LLAP has started. 
> The following problems should be solved by refactoring:
> 1. The main class is more than 800 lines long,should be cut into multiple 
> smaller classes.
> 2. The current design makes it extremely hard to write unit tests.
> 3. There are some overcomplicated, over-engineered parts of the code.
> 4. Most of the code is under org.apache.hadoop.hive.llap.cli, but some parts 
> are under org.apache.hadoop.hive.llap.cli.status. The whole program could be 
> moved to the latter.
> 5. LlapStatusHelpers serves as a class for holding classes, which doesn't 
> make much sense.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

2018-10-27 Thread Tom Zeng (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Zeng updated HIVE-20825:

Description: 
When using Hive ACID Merge (supported with the ORC format) to update/insert 
data, bucket files with 0 byte or 3 bytes (file content is three character: 
ORC) are generated during MERGE INTO operations which finish with no errors. 
Subsequent queries on the base table will get "Not a valid ORC file" error.

 

The following script can be used to reproduce the issue(note that with small 
amount of data like this increase the number of bucket could result in query to 
work, but with large data set it will fail no matter what bucket size):

set hive.auto.convert.join=false;
 set hive.enforce.bucketing=true;
 set hive.exec.dynamic.partition.mode = nonstrict;
 set hive.support.concurrency=true;
 set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

drop table if exists mergedelta_txt_1;
 drop table if exists mergedelta_txt_2;

CREATE TABLE mergedelta_txt_1 (
 id_str varchar(12), time_key int, value bigint)
 PARTITIONED BY (date_key int)
 ROW FORMAT DELIMITED
 STORED AS TEXTFILE;

CREATE TABLE mergedelta_txt_2 (
 id_str varchar(12), time_key int, value bigint)
 PARTITIONED BY (date_key int)
 ROW FORMAT DELIMITED
 STORED AS TEXTFILE;

INSERT INTO TABLE mergedelta_txt_1
 partition(date_key=20170103)
 VALUES
 ("AB94LIENR0",46700,12345676836978),
 ("AB94LIENR1",46825,12345676836978),
 ("AB94LIENS0",46709,12345676836978),
 ("AB94LIENS1",46834,12345676836978),
 ("AB94LIENT0",46709,12345676836978),
 ("AB94LIENT1",46834,12345676836978),
 ("AB94LIENU0",46718,12345676836978),
 ("AB94LIENU1",46844,12345676836978),
 ("AB94LIENV0",46719,12345676836978),
 ("AB94LIENV1",46844,12345676836978),
 ("AB94LIENW0",46728,12345676836978),
 ("AB94LIENW1",46854,12345676836978),
 ("AB94LIENX0",46728,12345676836978),
 ("AB94LIENX1",46854,12345676836978),
 ("AB94LIENY0",46737,12345676836978),
 ("AB94LIENY1",46863,12345676836978),
 ("AB94LIENZ0",46738,12345676836978),
 ("AB94LIENZ1",46863,12345676836978),
 ("AB94LIERA0",47176,12345676836982),
 ("AB94LIERA1",47302,12345676836982);

INSERT INTO TABLE mergedelta_txt_2
 partition(date_key=20170103)
 VALUES 
 ("AB94LIENT1",46834,12345676836978),
 ("AB94LIENU0",46718,12345676836978),
 ("AB94LIENU1",46844,12345676836978),
 ("AB94LIENV0",46719,12345676836978),
 ("AB94LIENV1",46844,12345676836978),
 ("AB94LIENW0",46728,12345676836978),
 ("AB94LIENW1",46854,12345676836978),
 ("AB94LIENX0",46728,12345676836978),
 ("AB94LIENX1",46854,12345676836978),
 ("AB94LIENY0",46737,12345676836978),
 ("AB94LIENY1",46863,12345676836978),
 ("AB94LIENZ0",46738,12345676836978),
 ("AB94LIENZ1",46863,12345676836978),
 ("AB94LIERA0",47176,12345676836982),
 ("AB94LIERA1",47302,12345676836982),
 ("AB94LIERA2",47418,12345676836982),
 ("AB94LIERB0",47176,12345676836982),
 ("AB94LIERB1",47302,12345676836982),
 ("AB94LIERB2",47418,12345676836982),
 ("AB94LIERC0",47185,12345676836982);

DROP TABLE IF EXISTS mergebase_1;
 CREATE TABLE mergebase_1 (
 id_str varchar(12) , time_key int , value bigint)
 PARTITIONED BY (date_key int)
 CLUSTERED BY (id_str,time_key) INTO 4 BUCKETS
 STORED AS ORC
 TBLPROPERTIES (
 'orc.compress'='SNAPPY',
 'pk_columns'='id_str,date_key,time_key',
 'NO_AUTO_COMPACTION'='true',
 'transactional'='true');

MERGE INTO mergebase_1 AS base
 USING (SELECT * 
 FROM (
 SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
 FROM mergedelta_txt_1
 DISTRIBUTE BY date_key
 ) rankedtbl 
 WHERE rankedtbl.rk=1
 ) AS delta
 ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
delta.time_key=base.time_key
 WHEN MATCHED THEN UPDATE SET value=delta.value
 WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
delta.value, delta.date_key);

MERGE INTO mergebase_1 AS base
 USING (SELECT * 
 FROM (
 SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
 FROM mergedelta_txt_2
 DISTRIBUTE BY date_key
 ) rankedtbl 
 WHERE rankedtbl.rk=1
 ) AS delta
 ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
delta.time_key=base.time_key
 WHEN MATCHED THEN UPDATE SET value=delta.value
 WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
delta.value, delta.date_key);

select count(*) from mergebase_1;

  was:
When using Hive ACID Merge (supported with the ORC format) to update/insert 
data, bucket files with 0 byte or 3 bytes (file content is three character: 
ORC) are generated during MERGE INTO operations which finish with no errors. 
Subsequent queries on the base table will get "Not a valid ORC file" error.

 

The following script can be used to reproduce the issue:

set hive.auto.convert.join=false;
set hive.enforce.bucketing=true;
set hive.exec.dynamic.partition.mode = nonstrict;
set hive.sup

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

2018-10-27 Thread Tom Zeng (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Zeng updated HIVE-20825:

Attachment: (was: hive-merge-invalid-orc-repro.hql)

> Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in 
> length) causing the "Not a valid ORC file" error
> --
>
> Key: HIVE-20825
> URL: https://issues.apache.org/jira/browse/HIVE-20825
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, ORC, Transactions
>Affects Versions: 2.2.0, 2.3.1, 2.3.2
> Environment: Hive 2.3.x on Amazon EMR 5.8.0 to 5.18.0
>Reporter: Tom Zeng
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: hive-merge-invalid-orc-repro.log
>
>
> When using Hive ACID Merge (supported with the ORC format) to update/insert 
> data, bucket files with 0 byte or 3 bytes (file content is three character: 
> ORC) are generated during MERGE INTO operations which finish with no errors. 
> Subsequent queries on the base table will get "Not a valid ORC file" error.
>  
> The following script can be used to reproduce the issue(note that with small 
> amount of data like this increase the number of bucket could result in query 
> to work, but with large data set it will fail no matter what bucket size):
> set hive.auto.convert.join=false;
>  set hive.enforce.bucketing=true;
>  set hive.exec.dynamic.partition.mode = nonstrict;
>  set hive.support.concurrency=true;
>  set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> drop table if exists mergedelta_txt_1;
>  drop table if exists mergedelta_txt_2;
> CREATE TABLE mergedelta_txt_1 (
>  id_str varchar(12), time_key int, value bigint)
>  PARTITIONED BY (date_key int)
>  ROW FORMAT DELIMITED
>  STORED AS TEXTFILE;
> CREATE TABLE mergedelta_txt_2 (
>  id_str varchar(12), time_key int, value bigint)
>  PARTITIONED BY (date_key int)
>  ROW FORMAT DELIMITED
>  STORED AS TEXTFILE;
> INSERT INTO TABLE mergedelta_txt_1
>  partition(date_key=20170103)
>  VALUES
>  ("AB94LIENR0",46700,12345676836978),
>  ("AB94LIENR1",46825,12345676836978),
>  ("AB94LIENS0",46709,12345676836978),
>  ("AB94LIENS1",46834,12345676836978),
>  ("AB94LIENT0",46709,12345676836978),
>  ("AB94LIENT1",46834,12345676836978),
>  ("AB94LIENU0",46718,12345676836978),
>  ("AB94LIENU1",46844,12345676836978),
>  ("AB94LIENV0",46719,12345676836978),
>  ("AB94LIENV1",46844,12345676836978),
>  ("AB94LIENW0",46728,12345676836978),
>  ("AB94LIENW1",46854,12345676836978),
>  ("AB94LIENX0",46728,12345676836978),
>  ("AB94LIENX1",46854,12345676836978),
>  ("AB94LIENY0",46737,12345676836978),
>  ("AB94LIENY1",46863,12345676836978),
>  ("AB94LIENZ0",46738,12345676836978),
>  ("AB94LIENZ1",46863,12345676836978),
>  ("AB94LIERA0",47176,12345676836982),
>  ("AB94LIERA1",47302,12345676836982);
> INSERT INTO TABLE mergedelta_txt_2
>  partition(date_key=20170103)
>  VALUES 
>  ("AB94LIENT1",46834,12345676836978),
>  ("AB94LIENU0",46718,12345676836978),
>  ("AB94LIENU1",46844,12345676836978),
>  ("AB94LIENV0",46719,12345676836978),
>  ("AB94LIENV1",46844,12345676836978),
>  ("AB94LIENW0",46728,12345676836978),
>  ("AB94LIENW1",46854,12345676836978),
>  ("AB94LIENX0",46728,12345676836978),
>  ("AB94LIENX1",46854,12345676836978),
>  ("AB94LIENY0",46737,12345676836978),
>  ("AB94LIENY1",46863,12345676836978),
>  ("AB94LIENZ0",46738,12345676836978),
>  ("AB94LIENZ1",46863,12345676836978),
>  ("AB94LIERA0",47176,12345676836982),
>  ("AB94LIERA1",47302,12345676836982),
>  ("AB94LIERA2",47418,12345676836982),
>  ("AB94LIERB0",47176,12345676836982),
>  ("AB94LIERB1",47302,12345676836982),
>  ("AB94LIERB2",47418,12345676836982),
>  ("AB94LIERC0",47185,12345676836982);
> DROP TABLE IF EXISTS mergebase_1;
>  CREATE TABLE mergebase_1 (
>  id_str varchar(12) , time_key int , value bigint)
>  PARTITIONED BY (date_key int)
>  CLUSTERED BY (id_str,time_key) INTO 4 BUCKETS
>  STORED AS ORC
>  TBLPROPERTIES (
>  'orc.compress'='SNAPPY',
>  'pk_columns'='id_str,date_key,time_key',
>  'NO_AUTO_COMPACTION'='true',
>  'transactional'='true');
> MERGE INTO mergebase_1 AS base
>  USING (SELECT * 
>  FROM (
>  SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
> id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
>  FROM mergedelta_txt_1
>  DISTRIBUTE BY date_key
>  ) rankedtbl 
>  WHERE rankedtbl.rk=1
>  ) AS delta
>  ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
> delta.time_key=base.time_key
>  WHEN MATCHED THEN UPDATE SET value=delta.value
>  WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
> delta.value, delta.date_key);
> MERGE INTO mergebase_1 AS base
>  USING (SELECT * 
>  FROM (
>  SELECT id_str ,time_key ,value, date_key, rank() OVER (P

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

2018-10-27 Thread Tom Zeng (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Zeng updated HIVE-20825:

Attachment: hive-merge-invalid-orc-repro.hql

> Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in 
> length) causing the "Not a valid ORC file" error
> --
>
> Key: HIVE-20825
> URL: https://issues.apache.org/jira/browse/HIVE-20825
> Project: Hive
>  Issue Type: Bug
>  Components: Hive, ORC, Transactions
>Affects Versions: 2.2.0, 2.3.1, 2.3.2
> Environment: Hive 2.3.x on Amazon EMR 5.8.0 to 5.18.0
>Reporter: Tom Zeng
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: hive-merge-invalid-orc-repro.hql, 
> hive-merge-invalid-orc-repro.log
>
>
> When using Hive ACID Merge (supported with the ORC format) to update/insert 
> data, bucket files with 0 byte or 3 bytes (file content is three character: 
> ORC) are generated during MERGE INTO operations which finish with no errors. 
> Subsequent queries on the base table will get "Not a valid ORC file" error.
>  
> The following script can be used to reproduce the issue(note that with small 
> amount of data like this increase the number of bucket could result in query 
> to work, but with large data set it will fail no matter what bucket size):
> set hive.auto.convert.join=false;
>  set hive.enforce.bucketing=true;
>  set hive.exec.dynamic.partition.mode = nonstrict;
>  set hive.support.concurrency=true;
>  set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> drop table if exists mergedelta_txt_1;
>  drop table if exists mergedelta_txt_2;
> CREATE TABLE mergedelta_txt_1 (
>  id_str varchar(12), time_key int, value bigint)
>  PARTITIONED BY (date_key int)
>  ROW FORMAT DELIMITED
>  STORED AS TEXTFILE;
> CREATE TABLE mergedelta_txt_2 (
>  id_str varchar(12), time_key int, value bigint)
>  PARTITIONED BY (date_key int)
>  ROW FORMAT DELIMITED
>  STORED AS TEXTFILE;
> INSERT INTO TABLE mergedelta_txt_1
>  partition(date_key=20170103)
>  VALUES
>  ("AB94LIENR0",46700,12345676836978),
>  ("AB94LIENR1",46825,12345676836978),
>  ("AB94LIENS0",46709,12345676836978),
>  ("AB94LIENS1",46834,12345676836978),
>  ("AB94LIENT0",46709,12345676836978),
>  ("AB94LIENT1",46834,12345676836978),
>  ("AB94LIENU0",46718,12345676836978),
>  ("AB94LIENU1",46844,12345676836978),
>  ("AB94LIENV0",46719,12345676836978),
>  ("AB94LIENV1",46844,12345676836978),
>  ("AB94LIENW0",46728,12345676836978),
>  ("AB94LIENW1",46854,12345676836978),
>  ("AB94LIENX0",46728,12345676836978),
>  ("AB94LIENX1",46854,12345676836978),
>  ("AB94LIENY0",46737,12345676836978),
>  ("AB94LIENY1",46863,12345676836978),
>  ("AB94LIENZ0",46738,12345676836978),
>  ("AB94LIENZ1",46863,12345676836978),
>  ("AB94LIERA0",47176,12345676836982),
>  ("AB94LIERA1",47302,12345676836982);
> INSERT INTO TABLE mergedelta_txt_2
>  partition(date_key=20170103)
>  VALUES 
>  ("AB94LIENT1",46834,12345676836978),
>  ("AB94LIENU0",46718,12345676836978),
>  ("AB94LIENU1",46844,12345676836978),
>  ("AB94LIENV0",46719,12345676836978),
>  ("AB94LIENV1",46844,12345676836978),
>  ("AB94LIENW0",46728,12345676836978),
>  ("AB94LIENW1",46854,12345676836978),
>  ("AB94LIENX0",46728,12345676836978),
>  ("AB94LIENX1",46854,12345676836978),
>  ("AB94LIENY0",46737,12345676836978),
>  ("AB94LIENY1",46863,12345676836978),
>  ("AB94LIENZ0",46738,12345676836978),
>  ("AB94LIENZ1",46863,12345676836978),
>  ("AB94LIERA0",47176,12345676836982),
>  ("AB94LIERA1",47302,12345676836982),
>  ("AB94LIERA2",47418,12345676836982),
>  ("AB94LIERB0",47176,12345676836982),
>  ("AB94LIERB1",47302,12345676836982),
>  ("AB94LIERB2",47418,12345676836982),
>  ("AB94LIERC0",47185,12345676836982);
> DROP TABLE IF EXISTS mergebase_1;
>  CREATE TABLE mergebase_1 (
>  id_str varchar(12) , time_key int , value bigint)
>  PARTITIONED BY (date_key int)
>  CLUSTERED BY (id_str,time_key) INTO 4 BUCKETS
>  STORED AS ORC
>  TBLPROPERTIES (
>  'orc.compress'='SNAPPY',
>  'pk_columns'='id_str,date_key,time_key',
>  'NO_AUTO_COMPACTION'='true',
>  'transactional'='true');
> MERGE INTO mergebase_1 AS base
>  USING (SELECT * 
>  FROM (
>  SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
> id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
>  FROM mergedelta_txt_1
>  DISTRIBUTE BY date_key
>  ) rankedtbl 
>  WHERE rankedtbl.rk=1
>  ) AS delta
>  ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
> delta.time_key=base.time_key
>  WHEN MATCHED THEN UPDATE SET value=delta.value
>  WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
> delta.value, delta.date_key);
> MERGE INTO mergebase_1 AS base
>  USING (SELECT * 
>  FROM (
>  SELECT id_str ,time_key ,value

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

2018-10-27 Thread Tom Zeng (JIRA)



 [ 
https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Zeng updated HIVE-20825:

Description: 
When using Hive ACID Merge (supported with the ORC format) to update/insert 
data, bucket files with 0 byte or 3 bytes (file content contains three 
characters: ORC) are generated during MERGE INTO operations which finish with 
no errors. Subsequent queries on the base table will get "Not a valid ORC file" 
error.

 

The following script can be used to reproduce the issue(note that with small 
amount of data like this increasing the number of buckets could result in query 
working, but with large data set it will fail no matter what bucket size):

set hive.auto.convert.join=false;
 set hive.enforce.bucketing=true;
 set hive.exec.dynamic.partition.mode = nonstrict;
 set hive.support.concurrency=true;
 set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

drop table if exists mergedelta_txt_1;
 drop table if exists mergedelta_txt_2;

CREATE TABLE mergedelta_txt_1 (
 id_str varchar(12), time_key int, value bigint)
 PARTITIONED BY (date_key int)
 ROW FORMAT DELIMITED
 STORED AS TEXTFILE;

CREATE TABLE mergedelta_txt_2 (
 id_str varchar(12), time_key int, value bigint)
 PARTITIONED BY (date_key int)
 ROW FORMAT DELIMITED
 STORED AS TEXTFILE;

INSERT INTO TABLE mergedelta_txt_1
 partition(date_key=20170103)
 VALUES
 ("AB94LIENR0",46700,12345676836978),
 ("AB94LIENR1",46825,12345676836978),
 ("AB94LIENS0",46709,12345676836978),
 ("AB94LIENS1",46834,12345676836978),
 ("AB94LIENT0",46709,12345676836978),
 ("AB94LIENT1",46834,12345676836978),
 ("AB94LIENU0",46718,12345676836978),
 ("AB94LIENU1",46844,12345676836978),
 ("AB94LIENV0",46719,12345676836978),
 ("AB94LIENV1",46844,12345676836978),
 ("AB94LIENW0",46728,12345676836978),
 ("AB94LIENW1",46854,12345676836978),
 ("AB94LIENX0",46728,12345676836978),
 ("AB94LIENX1",46854,12345676836978),
 ("AB94LIENY0",46737,12345676836978),
 ("AB94LIENY1",46863,12345676836978),
 ("AB94LIENZ0",46738,12345676836978),
 ("AB94LIENZ1",46863,12345676836978),
 ("AB94LIERA0",47176,12345676836982),
 ("AB94LIERA1",47302,12345676836982);

INSERT INTO TABLE mergedelta_txt_2
 partition(date_key=20170103)
 VALUES 
 ("AB94LIENT1",46834,12345676836978),
 ("AB94LIENU0",46718,12345676836978),
 ("AB94LIENU1",46844,12345676836978),
 ("AB94LIENV0",46719,12345676836978),
 ("AB94LIENV1",46844,12345676836978),
 ("AB94LIENW0",46728,12345676836978),
 ("AB94LIENW1",46854,12345676836978),
 ("AB94LIENX0",46728,12345676836978),
 ("AB94LIENX1",46854,12345676836978),
 ("AB94LIENY0",46737,12345676836978),
 ("AB94LIENY1",46863,12345676836978),
 ("AB94LIENZ0",46738,12345676836978),
 ("AB94LIENZ1",46863,12345676836978),
 ("AB94LIERA0",47176,12345676836982),
 ("AB94LIERA1",47302,12345676836982),
 ("AB94LIERA2",47418,12345676836982),
 ("AB94LIERB0",47176,12345676836982),
 ("AB94LIERB1",47302,12345676836982),
 ("AB94LIERB2",47418,12345676836982),
 ("AB94LIERC0",47185,12345676836982);

DROP TABLE IF EXISTS mergebase_1;
 CREATE TABLE mergebase_1 (
 id_str varchar(12) , time_key int , value bigint)
 PARTITIONED BY (date_key int)
 CLUSTERED BY (id_str,time_key) INTO 4 BUCKETS
 STORED AS ORC
 TBLPROPERTIES (
 'orc.compress'='SNAPPY',
 'pk_columns'='id_str,date_key,time_key',
 'NO_AUTO_COMPACTION'='true',
 'transactional'='true');

MERGE INTO mergebase_1 AS base
 USING (SELECT * 
 FROM (
 SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
 FROM mergedelta_txt_1
 DISTRIBUTE BY date_key
 ) rankedtbl 
 WHERE rankedtbl.rk=1
 ) AS delta
 ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
delta.time_key=base.time_key
 WHEN MATCHED THEN UPDATE SET value=delta.value
 WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
delta.value, delta.date_key);

MERGE INTO mergebase_1 AS base
 USING (SELECT * 
 FROM (
 SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY 
id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk 
 FROM mergedelta_txt_2
 DISTRIBUTE BY date_key
 ) rankedtbl 
 WHERE rankedtbl.rk=1
 ) AS delta
 ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND 
delta.time_key=base.time_key
 WHEN MATCHED THEN UPDATE SET value=delta.value
 WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , 
delta.value, delta.date_key);

select count(*) from mergebase_1;

  was:
When using Hive ACID Merge (supported with the ORC format) to update/insert 
data, bucket files with 0 byte or 3 bytes (file content is three character: 
ORC) are generated during MERGE INTO operations which finish with no errors. 
Subsequent queries on the base table will get "Not a valid ORC file" error.

 

The following script can be used to reproduce the issue(note that with small 
amount of data like this increase the number of bucket could result in query to 
work, but with la

[jira] [Commented] (HIVE-20811) Turn on dynamic partitioned hash join

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1288#comment-1288
 ] 

Hive QA commented on HIVE-20811:


| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
31s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 26s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14655/dev-support/hive-personality.sh
 |
| git revision | master / 1002e89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: common U: common |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14655/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> Turn on dynamic partitioned hash join
> -
>
> Key: HIVE-20811
> URL: https://issues.apache.org/jira/browse/HIVE-20811
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20811.1.patch
>
>
> Currently it is off by default.
> Turning if ON by default will help uncover and fix issues if there are any.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20811) Turn on dynamic partitioned hash join

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1297#comment-1297
 ] 

Hive QA commented on HIVE-20811:




Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12945661/HIVE-20811.1.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15507 tests 
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2]
 (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty]
 (batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_3]
 (batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_hint]
 (batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[murmur_hash_migration]
 (batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash]
 (batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main]
 (batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_full_outer_join]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_leftsemi_mapjoin]
 (batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_llap_text_1]
 (batchId=162)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query6] 
(batchId=271)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query6]
 (batchId=271)
{noformat}

Test results: 
https://builds.apache.org/job/PreCommit-HIVE-Build/14655/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14655/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14655/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12945661 - PreCommit-HIVE-Build

> Turn on dynamic partitioned hash join
> -
>
> Key: HIVE-20811
> URL: https://issues.apache.org/jira/browse/HIVE-20811
> Project: Hive
>  Issue Type: Improvement
>  Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-20811.1.patch
>
>
> Currently it is off by default.
> Turning if ON by default will help uncover and fix issues if there are any.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HIVE-20793) add RP namespacing to workload management

2018-10-27 Thread Hive QA (JIRA)



[ 
https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1301#comment-1301
 ] 

Hive QA commented on HIVE-20793:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
24s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m 
30s{color} | {color:blue} common in master has 65 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  2m 
17s{color} | {color:blue} standalone-metastore/metastore-common in master has 
28 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  3m 
44s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. 
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  1m  
6s{color} | {color:blue} standalone-metastore/metastore-server in master has 
181 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m  
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
20s{color} | {color:red} hcatalog-unit in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red}  0m 
40s{color} | {color:red} ql: The patch generated 8 new + 381 unchanged - 1 
fixed = 389 total (was 382) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 34 line(s) that end in whitespace. Use 
git apply --whitespace=fix <>. Refer 
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  8m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
12s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 27s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests |  asflicense  javac  javadoc  findbugs  checkstyle  compile  |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-14656/dev-support/hive-personality.sh
 |
| git revision | master / 1002e89 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| mvninstall | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14656/yetus/patch-mvninstall-itests_hcatalog-unit.txt
 |
| checkstyle | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14656/yetus/diff-checkstyle-ql.txt
 |
| whitespace | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14656/yetus/whitespace-eol.txt
 |
| modules | C: common standalone-metastore/metastore-common metastore 
itests/hcatalog-unit ql standalone-metastore/metastore-server U: . |
| Console output | 
http://104.198.109.242/logs//PreCommit-HIVE-Build-14656/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |


This message was automatically generated.



> add RP namespacing to workload management
> -
>
> Key: HIVE-20793
> URL: https://issues.apache.org/jira/browse/HIVE-20793
> Project: Hive
>  Issue Type: Bug
>Reporter: Sergey S

[jira] [Commented] (HIVE-20803) Hive external table can't read S3 file containing timestamp partition

[jira] [Commented] (HIVE-20259) Cleanup of results cache directory

[jira] [Commented] (HIVE-20259) Cleanup of results cache directory

[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type

[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type

[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver

[jira] [Assigned] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

[jira] [Updated] (HIVE-20486) Kafka: Use Row SerDe + vectorization

[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic

[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic

[jira] [Assigned] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join

[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic

[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method

[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method

[jira] [Commented] (HIVE-20804) Further improvements to group by optimization with constraints

[jira] [Commented] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

[jira] [Resolved] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)

[jira] [Commented] (HIVE-20804) Further improvements to group by optimization with constraints

[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error

[jira] [Commented] (HIVE-20811) Turn on dynamic partitioned hash join

[jira] [Commented] (HIVE-20811) Turn on dynamic partitioned hash join

[jira] [Commented] (HIVE-20793) add RP namespacing to workload management

27 matches

Site Navigation

Mail list logo

Footer information