[jira] [Commented] (HIVE-20803) Hive external table can't read S3 file containing timestamp partition
[ https://issues.apache.org/jira/browse/HIVE-20803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16665999#comment-16665999 ] t oo commented on HIVE-20803: - workaround is to write a different path to s3 that url encodes the : colon character. The space character can stay > Hive external table can't read S3 file containing timestamp partition > - > > Key: HIVE-20803 > URL: https://issues.apache.org/jira/browse/HIVE-20803 > Project: Hive > Issue Type: Bug > Components: SQL >Affects Versions: 2.3.2 >Reporter: t oo >Priority: Major > > SparkThriftServer can select * from the table fine and get data. But > HiveServer2 throws below error on select *: > > hive.msck.path.validation = ignore in hive-site.xml > then ran msck repair my_sch.h_l > aws s3 ls s3://priv1/priv2/H_L/ --recursive > 2018-10-18 03:00:56 2474983 > priv1/priv2/H_L/part_dt=20180309/part_src=xyz/part_src_file=MY_LOC/part_ldts=2018-10-18 > 02:59:46/part-0-2536ca01-243c-4220-8e55-6869a045fba2.snappy.parquet > show create table my_sch.h_l; > ++ > | createtab_stmt | > ++ > | CREATE EXTERNAL TABLE `my_sch.h_l`( | > | `xy_hkey_h_l` binary, | > | `xy_rtts` timestamp, | > | `xy_rsrc` string, | > | `xy_bkcc` string, | > | `xy_mltid` string, | > | `location_id` bigint) | > | PARTITIONED BY ( | > | `part_dt` string, | > | `part_src` string, | > | `part_src_file` string, | > | `part_ldts` timestamp) | > | ROW FORMAT SERDE | > | 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' | > | STORED AS INPUTFORMAT | > | 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' | > | OUTPUTFORMAT | > | 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' | > | LOCATION | > | 's3a://priv1/priv2/H_L' | > | TBLPROPERTIES ( | > | 'spark.sql.partitionProvider'='catalog', | > | 'spark.sql.sources.schema.numPartCols'='4', | > | 'spark.sql.sources.schema.numParts'='1', | > | > 'spark.sql.sources.schema.part.0'='\{"type":"struct","fields":[{"name":"xy_hkey_h_l","type":"binary","nullable":true,"metadata":{}},\{"name":"xy_rtts","type":"timestamp","nullable":true,"metadata":{}},\{"name":"xy_rsrc","type":"string","nullable":true,"metadata":{}},\{"name":"xy_bkcc","type":"string","nullable":true,"metadata":{}},\{"name":"xy_mltid","type":"string","nullable":true,"metadata":{}},\{"name":"location_id","type":"long","nullable":true,"metadata":{}},\{"name":"part_dt","type":"string","nullable":true,"metadata":{}},\{"name":"part_src","type":"string","nullable":true,"metadata":{}},\{"name":"part_src_file","type":"string","nullable":true,"metadata":{}},\{"name":"part_ldts","type":"timestamp","nullable":true,"metadata":{}}]}', > | > | 'spark.sql.sources.schema.partCol.0'='part_dt', | > | 'spark.sql.sources.schema.partCol.1'='part_src', | > | 'spark.sql.sources.schema.partCol.2'='part_src_file', | > | 'spark.sql.sources.schema.partCol.3'='part_ldts', | > | 'transient_lastDdlTime'='1540421484') | > ++ > select * from my_sch.h_l limit 5; > Error: java.io.IOException: java.lang.IllegalArgumentException: > java.net.URISyntaxException: Relative path in absolute URI: > part_ldts=2018-10-18 02:59:46 (state=,code=0) > org.apache.hive.service.cli.HiveSQLException: java.io.IOException: > java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative > path in absolute URI: part_ldts=2018-10-18 02:59:46 > at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:267) > at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:253) > at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:374) > at org.apache.hive.beeline.BufferedRows.(BufferedRows.java:53) > at > org.apache.hive.beeline.IncrementalRowsWithNormalization.(IncrementalRowsWithNormalization.java:50) > at org.apache.hive.beeline.BeeLine.print(BeeLine.java:2192) > at org.apache.hive.beeline.Commands.executeInternal(Commands.java:1009) > at org.apache.hive.beeline.Commands.execute(Commands.java:1205) > at org.apache.hive.beeline.Commands.sql(Commands.java:1134) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1314) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:1178) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1033) > at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:519) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:501) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Meth
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1004#comment-1004 ] Hive QA commented on HIVE-20259: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 50s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 38s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s{color} | {color:red} common: The patch generated 3 new + 426 unchanged - 0 fixed = 429 total (was 426) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 38s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14649/dev-support/hive-personality.sh | | git revision | master / 1002e89 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14649/yetus/diff-checkstyle-common.txt | | modules | C: common ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14649/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch, HIVE-20259.2.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20259) Cleanup of results cache directory
[ https://issues.apache.org/jira/browse/HIVE-20259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1015#comment-1015 ] Hive QA commented on HIVE-20259: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945635/HIVE-20259.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15507 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning (batchId=331) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14649/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14649/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14649/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945635 - PreCommit-HIVE-Build > Cleanup of results cache directory > -- > > Key: HIVE-20259 > URL: https://issues.apache.org/jira/browse/HIVE-20259 > Project: Hive > Issue Type: Sub-task >Reporter: Jason Dere >Assignee: Jason Dere >Priority: Major > Attachments: HIVE-20259.1.patch, HIVE-20259.2.patch > > > The query results cache directory is currently deleted at process exit. This > does not work in the case of a kill -9 or a sudden process exit of Hive. > There should be some cleanup mechanism in place to take care of any old cache > directories that were not deleted at process exit. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[ https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1139#comment-1139 ] Hive QA commented on HIVE-20617: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 48s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 28s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 40s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s{color} | {color:red} ql: The patch generated 2 new + 238 unchanged - 12 fixed = 240 total (was 250) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 1 line(s) with tabs. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 51s{color} | {color:red} ql generated 4 new + 2313 unchanged - 4 fixed = 2317 total (was 2317) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 25m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo, Object, TypeInfo) invokes inefficient new Byte(String) constructor; use Byte.valueOf(String) instead At TypeCheckProcFactory.java:new Byte(String) constructor; use Byte.valueOf(String) instead At TypeCheckProcFactory.java:[line 1448] | | | org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo, Object, TypeInfo) invokes inefficient new Integer(String) constructor; use Integer.valueOf(String) instead At TypeCheckProcFactory.java:new Integer(String) constructor; use Integer.valueOf(String) instead At TypeCheckProcFactory.java:[line 1440] | | | org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo, Object, TypeInfo) invokes inefficient new Long(String) constructor; use Long.valueOf(String) instead At TypeCheckProcFactory.java:new Long(String) constructor; use Long.valueOf(String) instead At TypeCheckProcFactory.java:[line 1442] | | | org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.interpretConstantAsPrimitive(PrimitiveTypeInfo, Object, TypeInfo) invokes inefficient new Short(String) constructor; use Short.valueOf(String) instead At TypeCheckProcFactory.java:new Short(String) constructor; use Short.valueOf(String) instead At TypeCheckProcFactory.java:[line 1450] | \\ \\ || Subsystem || Report/Notes || | Optional Tests |
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[ https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1158#comment-1158 ] Hive QA commented on HIVE-20617: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945683/HIVE-20617.12.patch {color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 15512 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_multi_single_reducer3] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[pcs] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_non_constant_in_expr] (batchId=81) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_struct_in] (batchId=49) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1] (batchId=182) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[is_distinct_from] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_column_in] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[multi_column_in_single] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_struct_in] (batchId=170) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_in_2] (batchId=98) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_simple_select] (batchId=118) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby_multi_single_reducer3] (batchId=132) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar] (batchId=128) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14650/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14650/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14650/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 15 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945683 - PreCommit-HIVE-Build > Fix type of constants in IN expressions to have correct type > > > Key: HIVE-20617 > URL: https://issues.apache.org/jira/browse/HIVE-20617 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch, > HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch, > HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch, > HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch > > > In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the > comparision in UDFIn may fail because if a or b is of char/varchar type the > constants will retain string type - especially after PointlookupOptimizer > compaction. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1209#comment-1209 ] Hive QA commented on HIVE-20807: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945662/HIVE-20807.02.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14651/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14651/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14651/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-10-27 19:41:20.336 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-14651/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-10-27 19:41:20.339 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 9.3.25.v20180904 (Laszlo Bodor via Thejas Nair) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 9.3.25.v20180904 (Laszlo Bodor via Thejas Nair) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-10-27 19:41:21.152 + rm -rf ../yetus_PreCommit-HIVE-Build-14651 + mkdir ../yetus_PreCommit-HIVE-Build-14651 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-14651 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14651/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/bin/ext/llapstatus.sh: does not exist in index error: a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapSliderUtils.java: does not exist in index error: a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusOptionsProcessor.java: does not exist in index error: a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/LlapStatusServiceDriver.java: does not exist in index error: a/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusHelpers.java: does not exist in index error: a/llap-server/src/test/org/apache/hadoop/hive/llap/cli/TestLlapStatusServiceDriver.java: does not exist in index Going to apply patch with: git apply -p1 /data/hiveptest/working/scratch/build.patch:3395: new blank line at EOF. + /data/hiveptest/working/scratch/build.patch:3625: new blank line at EOF. + warning: 2 lines add whitespace errors. + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc3519613450051859971.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc3519613450051859971.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] ANTLR Parser Generator Version 3.5.2 protoc-jar: executing: [/tmp/protoc4407913131038094891.exe, --version] libprotoc 2.5.0 ANTLR Parser Generator Version 3.5.2 Output file /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/m
[jira] [Assigned] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error
[ https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-20825: - Assignee: Eugene Koifman > Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in > length) causing the "Not a valid ORC file" error > -- > > Key: HIVE-20825 > URL: https://issues.apache.org/jira/browse/HIVE-20825 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 2.2.0, 2.3.1, 2.3.2 > Environment: Hive 2.3.x on Amazon EMR 5.8.0 to 5.18.0 >Reporter: Tom Zeng >Assignee: Eugene Koifman >Priority: Major > Attachments: hive-merge-invalid-orc-repro.hql, > hive-merge-invalid-orc-repro.log > > > When using Hive ACID Merge (supported with the ORC format) to update/insert > data, bucket files with 0 byte or 3 bytes (file content is three character: > ORC) are generated during MERGE INTO operations which finish with no errors. > Subsequent queries on the base table will get "Not a valid ORC file" error. > > The following script can be used to reproduce the issue: > set hive.auto.convert.join=false; > set hive.enforce.bucketing=true; > set hive.exec.dynamic.partition.mode = nonstrict; > set hive.support.concurrency=true; > set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > drop table if exists mergedelta_txt_1; > drop table if exists mergedelta_txt_2; > CREATE TABLE mergedelta_txt_1 ( > id_str varchar(12), time_key int, value bigint) > PARTITIONED BY (date_key int) > ROW FORMAT DELIMITED > STORED AS TEXTFILE; > CREATE TABLE mergedelta_txt_2 ( > id_str varchar(12), time_key int, value bigint) > PARTITIONED BY (date_key int) > ROW FORMAT DELIMITED > STORED AS TEXTFILE; > INSERT INTO TABLE mergedelta_txt_1 > partition(date_key=20170103) > VALUES > ("AB94LIENR0",46700,12345676836978), > ("AB94LIENR1",46825,12345676836978), > ("AB94LIENS0",46709,12345676836978), > ("AB94LIENS1",46834,12345676836978), > ("AB94LIENT0",46709,12345676836978), > ("AB94LIENT1",46834,12345676836978), > ("AB94LIENU0",46718,12345676836978), > ("AB94LIENU1",46844,12345676836978), > ("AB94LIENV0",46719,12345676836978), > ("AB94LIENV1",46844,12345676836978), > ("AB94LIENW0",46728,12345676836978), > ("AB94LIENW1",46854,12345676836978), > ("AB94LIENX0",46728,12345676836978), > ("AB94LIENX1",46854,12345676836978), > ("AB94LIENY0",46737,12345676836978), > ("AB94LIENY1",46863,12345676836978), > ("AB94LIENZ0",46738,12345676836978), > ("AB94LIENZ1",46863,12345676836978), > ("AB94LIERA0",47176,12345676836982), > ("AB94LIERA1",47302,12345676836982); > INSERT INTO TABLE mergedelta_txt_2 > partition(date_key=20170103) > VALUES > ("AB94LIENT1",46834,12345676836978), > ("AB94LIENU0",46718,12345676836978), > ("AB94LIENU1",46844,12345676836978), > ("AB94LIENV0",46719,12345676836978), > ("AB94LIENV1",46844,12345676836978), > ("AB94LIENW0",46728,12345676836978), > ("AB94LIENW1",46854,12345676836978), > ("AB94LIENX0",46728,12345676836978), > ("AB94LIENX1",46854,12345676836978), > ("AB94LIENY0",46737,12345676836978), > ("AB94LIENY1",46863,12345676836978), > ("AB94LIENZ0",46738,12345676836978), > ("AB94LIENZ1",46863,12345676836978), > ("AB94LIERA0",47176,12345676836982), > ("AB94LIERA1",47302,12345676836982), > ("AB94LIERA2",47418,12345676836982), > ("AB94LIERB0",47176,12345676836982), > ("AB94LIERB1",47302,12345676836982), > ("AB94LIERB2",47418,12345676836982), > ("AB94LIERC0",47185,12345676836982); > DROP TABLE IF EXISTS mergebase_1; > CREATE TABLE mergebase_1 ( > id_str varchar(12) , time_key int , value bigint) > PARTITIONED BY (date_key int) > CLUSTERED BY (id_str,time_key) INTO 32 BUCKETS > STORED AS ORC > TBLPROPERTIES ( > 'orc.compress'='SNAPPY', > 'pk_columns'='id_str,date_key,time_key', > 'NO_AUTO_COMPACTION'='true', > 'transactional'='true'); > MERGE INTO mergebase_1 AS base > USING (SELECT * > FROM ( > SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY > id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk > FROM mergedelta_txt_1 > DISTRIBUTE BY date_key > ) rankedtbl > WHERE rankedtbl.rk=1 > ) AS delta > ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND > delta.time_key=base.time_key > WHEN MATCHED THEN UPDATE SET value=delta.value > WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , > delta.value, delta.date_key); > MERGE INTO mergebase_1 AS base > USING (SELECT * > FROM ( > SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY > id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk > FROM mergedelta_txt_2 > DISTRIBUTE BY date_key > ) rankedtbl > WHERE rankedtbl.rk=1 > ) AS d
[jira] [Updated] (HIVE-20486) Kafka: Use Row SerDe + vectorization
[ https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20486: -- Attachment: HIVE-20486.4.patch > Kafka: Use Row SerDe + vectorization > > > Key: HIVE-20486 > URL: https://issues.apache.org/jira/browse/HIVE-20486 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Reporter: Gopal V >Assignee: slim bouguerra >Priority: Major > Labels: kafka, vectorization > Fix For: 4.0.0 > > Attachments: HIVE-20486.3.patch, HIVE-20486.3.patch, > HIVE-20486.4.patch, HIVE-20486.patch > > > KafkaHandler returns unvectorized rows which causes the operators downstream > to be slower and sub-optimal. > Hive has a vectorization shim which allows Kafka streams without complex > projections to be wrapped into a vectorized reader via > {{hive.vectorized.use.row.serde.deserialize}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[ https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20778: --- Status: Open (was: Patch Available) > Join reordering may not be triggered if all joins in plan are created by > decorrelation logic > > > Key: HIVE-20778 > URL: https://issues.apache.org/jira/browse/HIVE-20778 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, > HIVE-20778.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[ https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20778: --- Attachment: HIVE-20778.3.patch > Join reordering may not be triggered if all joins in plan are created by > decorrelation logic > > > Key: HIVE-20778 > URL: https://issues.apache.org/jira/browse/HIVE-20778 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, > HIVE-20778.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20826) Enhance HiveSemiJoin rule to convert join + group by on left side to Left Semi Join
[ https://issues.apache.org/jira/browse/HIVE-20826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-20826: -- > Enhance HiveSemiJoin rule to convert join + group by on left side to Left > Semi Join > --- > > Key: HIVE-20826 > URL: https://issues.apache.org/jira/browse/HIVE-20826 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > > Currently HiveSemiJoin rule looks for pattern where group by is on right side. > We can convert joins which have group by on left side (assuming group by keys > are same as join keys and none of the columns are being projected from left > side) to LEFT SEMI JOIN by swapping the inputs. e.g. queries such as: > {code:sql} > explain select pp.p_partkey from (select distinct p_name from part) p join > part pp on pp.p_name = p.p_name; > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[ https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20778: --- Status: Patch Available (was: Open) Updated the golden files. Subquery tests have join reordering and as a result HiveSemiJoin rule, to covert to Left Semi Join, is not kicking in from some tests. There are other tests which now have Semi Join. I have opened HIVE-20826 to improve HiveSemiJoin rule. > Join reordering may not be triggered if all joins in plan are created by > decorrelation logic > > > Key: HIVE-20778 > URL: https://issues.apache.org/jira/browse/HIVE-20778 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, > HIVE-20778.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-20740: --- Attachment: HIVE-20740.05.patch > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1237#comment-1237 ] Vihang Karajgaonkar commented on HIVE-20740: Spent some more time on this. Based on what I understand, there is a race in the existing code which could lead to a NPE. Eg. Thread A calls setConf method with a new datasource properties. Thread A acquires {{proplock}} and sets {{pmf = null}} Meanwhile Thread B is in {{getPMF}} method which even though is static synchronized doesn't know if pmf is being updated by some other thread in {{setConf}} object. This could lead to a NPE on Thread B. Or some error on Thread A when it tries to get {{PersistenceManager}} from unitialized pmf. The good news this is very unlikely to happen. When datasource properties are changed, it needs a restart of HMS service which means there are no other threads requesting {{pmf}} until HMS up and running. But still having {{pmf}} and {{props}} objects as private members which need to be synchronized when updating can be handled more cleaning if we move them to a separate wrapper class which deals with these details. Latest patch refactors some of this code and hopefully makes is cleaner than the current implementation. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20804) Further improvements to group by optimization with constraints
[ https://issues.apache.org/jira/browse/HIVE-20804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1246#comment-1246 ] Hive QA commented on HIVE-20804: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 5s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 43s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s{color} | {color:red} ql: The patch generated 7 new + 6 unchanged - 0 fixed = 13 total (was 6) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 49s{color} | {color:red} ql generated 1 new + 2317 unchanged - 0 fixed = 2318 total (was 2317) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 21m 25s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Dead store to mapInToOutPos in org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelOptUtil$CardinalityChange.isCardinalitySameAsSource(HiveProject, ImmutableBitSet) At HiveRelOptUtil.java:org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelOptUtil$CardinalityChange.isCardinalitySameAsSource(HiveProject, ImmutableBitSet) At HiveRelOptUtil.java:[line 808] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14652/dev-support/hive-personality.sh | | git revision | master / 1002e89 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14652/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14652/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14652/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Further improvements to group by optimization with constraints > -- > > Key: HIVE-20804 > URL: https://issues.apache.org/jira/browse/HIVE-20804 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20804.1.patch, HIVE-20804.2.patch > > > Continuation of HIVE-17043 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
[ https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1248#comment-1248 ] Rajesh Balamohan commented on HIVE-20816: - Closing this for now. I will reopen it when I generate smaller dataset for this. > FastHiveDecimal throws Exception (RuntimeException: Unexpected #3) > -- > > Key: HIVE-20816 > URL: https://issues.apache.org/jira/browse/HIVE-20816 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.3.2 >Reporter: Rajesh Balamohan >Priority: Minor > Labels: fastdecimal > Attachments: HIVE-20816.1.patch > > > {noformat} > with t1 as ( > ... > ... > ) > select id, max(abs(c1))) from t1 group by id; > {noformat} > throws the following exception > {noformat} > g.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unexpected #3 > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711) > > ... > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unexpected #3 > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084) > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123) > ... 18 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: Unexpected #3 > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897) > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047) > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067) > ... 19 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20816) FastHiveDecimal throws Exception (RuntimeException: Unexpected #3)
[ https://issues.apache.org/jira/browse/HIVE-20816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Balamohan resolved HIVE-20816. - Resolution: Duplicate > FastHiveDecimal throws Exception (RuntimeException: Unexpected #3) > -- > > Key: HIVE-20816 > URL: https://issues.apache.org/jira/browse/HIVE-20816 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 2.3.2 >Reporter: Rajesh Balamohan >Priority: Minor > Labels: fastdecimal > Attachments: HIVE-20816.1.patch > > > {noformat} > with t1 as ( > ... > ... > ) > select id, max(abs(c1))) from t1 group by id; > {noformat} > throws the following exception > {noformat} > g.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unexpected #3 > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1126) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:711) > > ... > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: > Unexpected #3 > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1084) > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1123) > ... 18 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.RuntimeException: Unexpected #3 > at > org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:397) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897) > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1047) > at > org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1067) > ... 19 more > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20804) Further improvements to group by optimization with constraints
[ https://issues.apache.org/jira/browse/HIVE-20804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1249#comment-1249 ] Hive QA commented on HIVE-20804: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945651/HIVE-20804.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15507 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14652/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14652/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14652/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12945651 - PreCommit-HIVE-Build > Further improvements to group by optimization with constraints > -- > > Key: HIVE-20804 > URL: https://issues.apache.org/jira/browse/HIVE-20804 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20804.1.patch, HIVE-20804.2.patch > > > Continuation of HIVE-17043 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1259#comment-1259 ] Hive QA commented on HIVE-14557: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12934433/HIVE-14557.2.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14653/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14653/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14653/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-10-28 02:15:41.523 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-14653/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-10-28 02:15:41.526 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 9.3.25.v20180904 (Laszlo Bodor via Thejas Nair) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 9.3.25.v20180904 (Laszlo Bodor via Thejas Nair) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-10-28 02:15:42.166 + rm -rf ../yetus_PreCommit-HIVE-Build-14653 + mkdir ../yetus_PreCommit-HIVE-Build-14653 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-14653 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14653/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java: does not exist in index error: a/ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out: does not exist in index error: patch failed: ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out:29 Falling back to three-way merge... Applied patch to 'ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out' with conflicts. Going to apply patch with: git apply -p1 /data/hiveptest/working/scratch/build.patch:92: trailing whitespace. $hdt$_0:$hdt$_1:src2 /data/hiveptest/working/scratch/build.patch:97: trailing whitespace. $hdt$_0:$hdt$_1:src2 /data/hiveptest/working/scratch/build.patch:177: trailing whitespace. $INTNAME1 /data/hiveptest/working/scratch/build.patch:182: trailing whitespace. $INTNAME1 error: patch failed: ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out:29 Falling back to three-way merge... Applied patch to 'ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out' with conflicts. U ql/src/test/results/clientpositive/runtime_skewjoin_mapjoin_spark.q.out warning: 4 lines add whitespace errors. + result=1 + '[' 1 -ne 0 ']' + rm -rf yetus_PreCommit-HIVE-Build-14653 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12934433 - PreCommit-HIVE-Build > Nullpointer When both SkewJoin and Mapjoin Enabled > --- > > Key: HIVE-14557 > URL: https://issues.apache.org/jira/browse/HIVE-14557 > Project: Hive > Issue Type: Bug > Components: Physical Optimizer >Affects Versions: 1.1.0, 2.1.0 >Reporter: Nemon Lou >Assignee: Rui Li >Priority: Major > Attachments: HIVE-14557.2.patch, HIVE-14557.patch > > > The following sql fail
[jira] [Commented] (HIVE-20807) Refactor LlapStatusServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-20807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1274#comment-1274 ] Hive QA commented on HIVE-20807: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945662/HIVE-20807.02.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14654/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14654/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14654/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12945662/HIVE-20807.02.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12945662 - PreCommit-HIVE-Build > Refactor LlapStatusServiceDriver > > > Key: HIVE-20807 > URL: https://issues.apache.org/jira/browse/HIVE-20807 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20807.01.patch, HIVE-20807.02.patch > > > LlapStatusServiceDriver is the class used to determine if LLAP has started. > The following problems should be solved by refactoring: > 1. The main class is more than 800 lines long,should be cut into multiple > smaller classes. > 2. The current design makes it extremely hard to write unit tests. > 3. There are some overcomplicated, over-engineered parts of the code. > 4. Most of the code is under org.apache.hadoop.hive.llap.cli, but some parts > are under org.apache.hadoop.hive.llap.cli.status. The whole program could be > moved to the latter. > 5. LlapStatusHelpers serves as a class for holding classes, which doesn't > make much sense. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error
[ https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Zeng updated HIVE-20825: Description: When using Hive ACID Merge (supported with the ORC format) to update/insert data, bucket files with 0 byte or 3 bytes (file content is three character: ORC) are generated during MERGE INTO operations which finish with no errors. Subsequent queries on the base table will get "Not a valid ORC file" error. The following script can be used to reproduce the issue(note that with small amount of data like this increase the number of bucket could result in query to work, but with large data set it will fail no matter what bucket size): set hive.auto.convert.join=false; set hive.enforce.bucketing=true; set hive.exec.dynamic.partition.mode = nonstrict; set hive.support.concurrency=true; set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; drop table if exists mergedelta_txt_1; drop table if exists mergedelta_txt_2; CREATE TABLE mergedelta_txt_1 ( id_str varchar(12), time_key int, value bigint) PARTITIONED BY (date_key int) ROW FORMAT DELIMITED STORED AS TEXTFILE; CREATE TABLE mergedelta_txt_2 ( id_str varchar(12), time_key int, value bigint) PARTITIONED BY (date_key int) ROW FORMAT DELIMITED STORED AS TEXTFILE; INSERT INTO TABLE mergedelta_txt_1 partition(date_key=20170103) VALUES ("AB94LIENR0",46700,12345676836978), ("AB94LIENR1",46825,12345676836978), ("AB94LIENS0",46709,12345676836978), ("AB94LIENS1",46834,12345676836978), ("AB94LIENT0",46709,12345676836978), ("AB94LIENT1",46834,12345676836978), ("AB94LIENU0",46718,12345676836978), ("AB94LIENU1",46844,12345676836978), ("AB94LIENV0",46719,12345676836978), ("AB94LIENV1",46844,12345676836978), ("AB94LIENW0",46728,12345676836978), ("AB94LIENW1",46854,12345676836978), ("AB94LIENX0",46728,12345676836978), ("AB94LIENX1",46854,12345676836978), ("AB94LIENY0",46737,12345676836978), ("AB94LIENY1",46863,12345676836978), ("AB94LIENZ0",46738,12345676836978), ("AB94LIENZ1",46863,12345676836978), ("AB94LIERA0",47176,12345676836982), ("AB94LIERA1",47302,12345676836982); INSERT INTO TABLE mergedelta_txt_2 partition(date_key=20170103) VALUES ("AB94LIENT1",46834,12345676836978), ("AB94LIENU0",46718,12345676836978), ("AB94LIENU1",46844,12345676836978), ("AB94LIENV0",46719,12345676836978), ("AB94LIENV1",46844,12345676836978), ("AB94LIENW0",46728,12345676836978), ("AB94LIENW1",46854,12345676836978), ("AB94LIENX0",46728,12345676836978), ("AB94LIENX1",46854,12345676836978), ("AB94LIENY0",46737,12345676836978), ("AB94LIENY1",46863,12345676836978), ("AB94LIENZ0",46738,12345676836978), ("AB94LIENZ1",46863,12345676836978), ("AB94LIERA0",47176,12345676836982), ("AB94LIERA1",47302,12345676836982), ("AB94LIERA2",47418,12345676836982), ("AB94LIERB0",47176,12345676836982), ("AB94LIERB1",47302,12345676836982), ("AB94LIERB2",47418,12345676836982), ("AB94LIERC0",47185,12345676836982); DROP TABLE IF EXISTS mergebase_1; CREATE TABLE mergebase_1 ( id_str varchar(12) , time_key int , value bigint) PARTITIONED BY (date_key int) CLUSTERED BY (id_str,time_key) INTO 4 BUCKETS STORED AS ORC TBLPROPERTIES ( 'orc.compress'='SNAPPY', 'pk_columns'='id_str,date_key,time_key', 'NO_AUTO_COMPACTION'='true', 'transactional'='true'); MERGE INTO mergebase_1 AS base USING (SELECT * FROM ( SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk FROM mergedelta_txt_1 DISTRIBUTE BY date_key ) rankedtbl WHERE rankedtbl.rk=1 ) AS delta ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND delta.time_key=base.time_key WHEN MATCHED THEN UPDATE SET value=delta.value WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , delta.value, delta.date_key); MERGE INTO mergebase_1 AS base USING (SELECT * FROM ( SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk FROM mergedelta_txt_2 DISTRIBUTE BY date_key ) rankedtbl WHERE rankedtbl.rk=1 ) AS delta ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND delta.time_key=base.time_key WHEN MATCHED THEN UPDATE SET value=delta.value WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , delta.value, delta.date_key); select count(*) from mergebase_1; was: When using Hive ACID Merge (supported with the ORC format) to update/insert data, bucket files with 0 byte or 3 bytes (file content is three character: ORC) are generated during MERGE INTO operations which finish with no errors. Subsequent queries on the base table will get "Not a valid ORC file" error. The following script can be used to reproduce the issue: set hive.auto.convert.join=false; set hive.enforce.bucketing=true; set hive.exec.dynamic.partition.mode = nonstrict; set hive.sup
[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error
[ https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Zeng updated HIVE-20825: Attachment: (was: hive-merge-invalid-orc-repro.hql) > Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in > length) causing the "Not a valid ORC file" error > -- > > Key: HIVE-20825 > URL: https://issues.apache.org/jira/browse/HIVE-20825 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 2.2.0, 2.3.1, 2.3.2 > Environment: Hive 2.3.x on Amazon EMR 5.8.0 to 5.18.0 >Reporter: Tom Zeng >Assignee: Eugene Koifman >Priority: Major > Attachments: hive-merge-invalid-orc-repro.log > > > When using Hive ACID Merge (supported with the ORC format) to update/insert > data, bucket files with 0 byte or 3 bytes (file content is three character: > ORC) are generated during MERGE INTO operations which finish with no errors. > Subsequent queries on the base table will get "Not a valid ORC file" error. > > The following script can be used to reproduce the issue(note that with small > amount of data like this increase the number of bucket could result in query > to work, but with large data set it will fail no matter what bucket size): > set hive.auto.convert.join=false; > set hive.enforce.bucketing=true; > set hive.exec.dynamic.partition.mode = nonstrict; > set hive.support.concurrency=true; > set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > drop table if exists mergedelta_txt_1; > drop table if exists mergedelta_txt_2; > CREATE TABLE mergedelta_txt_1 ( > id_str varchar(12), time_key int, value bigint) > PARTITIONED BY (date_key int) > ROW FORMAT DELIMITED > STORED AS TEXTFILE; > CREATE TABLE mergedelta_txt_2 ( > id_str varchar(12), time_key int, value bigint) > PARTITIONED BY (date_key int) > ROW FORMAT DELIMITED > STORED AS TEXTFILE; > INSERT INTO TABLE mergedelta_txt_1 > partition(date_key=20170103) > VALUES > ("AB94LIENR0",46700,12345676836978), > ("AB94LIENR1",46825,12345676836978), > ("AB94LIENS0",46709,12345676836978), > ("AB94LIENS1",46834,12345676836978), > ("AB94LIENT0",46709,12345676836978), > ("AB94LIENT1",46834,12345676836978), > ("AB94LIENU0",46718,12345676836978), > ("AB94LIENU1",46844,12345676836978), > ("AB94LIENV0",46719,12345676836978), > ("AB94LIENV1",46844,12345676836978), > ("AB94LIENW0",46728,12345676836978), > ("AB94LIENW1",46854,12345676836978), > ("AB94LIENX0",46728,12345676836978), > ("AB94LIENX1",46854,12345676836978), > ("AB94LIENY0",46737,12345676836978), > ("AB94LIENY1",46863,12345676836978), > ("AB94LIENZ0",46738,12345676836978), > ("AB94LIENZ1",46863,12345676836978), > ("AB94LIERA0",47176,12345676836982), > ("AB94LIERA1",47302,12345676836982); > INSERT INTO TABLE mergedelta_txt_2 > partition(date_key=20170103) > VALUES > ("AB94LIENT1",46834,12345676836978), > ("AB94LIENU0",46718,12345676836978), > ("AB94LIENU1",46844,12345676836978), > ("AB94LIENV0",46719,12345676836978), > ("AB94LIENV1",46844,12345676836978), > ("AB94LIENW0",46728,12345676836978), > ("AB94LIENW1",46854,12345676836978), > ("AB94LIENX0",46728,12345676836978), > ("AB94LIENX1",46854,12345676836978), > ("AB94LIENY0",46737,12345676836978), > ("AB94LIENY1",46863,12345676836978), > ("AB94LIENZ0",46738,12345676836978), > ("AB94LIENZ1",46863,12345676836978), > ("AB94LIERA0",47176,12345676836982), > ("AB94LIERA1",47302,12345676836982), > ("AB94LIERA2",47418,12345676836982), > ("AB94LIERB0",47176,12345676836982), > ("AB94LIERB1",47302,12345676836982), > ("AB94LIERB2",47418,12345676836982), > ("AB94LIERC0",47185,12345676836982); > DROP TABLE IF EXISTS mergebase_1; > CREATE TABLE mergebase_1 ( > id_str varchar(12) , time_key int , value bigint) > PARTITIONED BY (date_key int) > CLUSTERED BY (id_str,time_key) INTO 4 BUCKETS > STORED AS ORC > TBLPROPERTIES ( > 'orc.compress'='SNAPPY', > 'pk_columns'='id_str,date_key,time_key', > 'NO_AUTO_COMPACTION'='true', > 'transactional'='true'); > MERGE INTO mergebase_1 AS base > USING (SELECT * > FROM ( > SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY > id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk > FROM mergedelta_txt_1 > DISTRIBUTE BY date_key > ) rankedtbl > WHERE rankedtbl.rk=1 > ) AS delta > ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND > delta.time_key=base.time_key > WHEN MATCHED THEN UPDATE SET value=delta.value > WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , > delta.value, delta.date_key); > MERGE INTO mergebase_1 AS base > USING (SELECT * > FROM ( > SELECT id_str ,time_key ,value, date_key, rank() OVER (P
[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error
[ https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Zeng updated HIVE-20825: Attachment: hive-merge-invalid-orc-repro.hql > Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in > length) causing the "Not a valid ORC file" error > -- > > Key: HIVE-20825 > URL: https://issues.apache.org/jira/browse/HIVE-20825 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 2.2.0, 2.3.1, 2.3.2 > Environment: Hive 2.3.x on Amazon EMR 5.8.0 to 5.18.0 >Reporter: Tom Zeng >Assignee: Eugene Koifman >Priority: Major > Attachments: hive-merge-invalid-orc-repro.hql, > hive-merge-invalid-orc-repro.log > > > When using Hive ACID Merge (supported with the ORC format) to update/insert > data, bucket files with 0 byte or 3 bytes (file content is three character: > ORC) are generated during MERGE INTO operations which finish with no errors. > Subsequent queries on the base table will get "Not a valid ORC file" error. > > The following script can be used to reproduce the issue(note that with small > amount of data like this increase the number of bucket could result in query > to work, but with large data set it will fail no matter what bucket size): > set hive.auto.convert.join=false; > set hive.enforce.bucketing=true; > set hive.exec.dynamic.partition.mode = nonstrict; > set hive.support.concurrency=true; > set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > drop table if exists mergedelta_txt_1; > drop table if exists mergedelta_txt_2; > CREATE TABLE mergedelta_txt_1 ( > id_str varchar(12), time_key int, value bigint) > PARTITIONED BY (date_key int) > ROW FORMAT DELIMITED > STORED AS TEXTFILE; > CREATE TABLE mergedelta_txt_2 ( > id_str varchar(12), time_key int, value bigint) > PARTITIONED BY (date_key int) > ROW FORMAT DELIMITED > STORED AS TEXTFILE; > INSERT INTO TABLE mergedelta_txt_1 > partition(date_key=20170103) > VALUES > ("AB94LIENR0",46700,12345676836978), > ("AB94LIENR1",46825,12345676836978), > ("AB94LIENS0",46709,12345676836978), > ("AB94LIENS1",46834,12345676836978), > ("AB94LIENT0",46709,12345676836978), > ("AB94LIENT1",46834,12345676836978), > ("AB94LIENU0",46718,12345676836978), > ("AB94LIENU1",46844,12345676836978), > ("AB94LIENV0",46719,12345676836978), > ("AB94LIENV1",46844,12345676836978), > ("AB94LIENW0",46728,12345676836978), > ("AB94LIENW1",46854,12345676836978), > ("AB94LIENX0",46728,12345676836978), > ("AB94LIENX1",46854,12345676836978), > ("AB94LIENY0",46737,12345676836978), > ("AB94LIENY1",46863,12345676836978), > ("AB94LIENZ0",46738,12345676836978), > ("AB94LIENZ1",46863,12345676836978), > ("AB94LIERA0",47176,12345676836982), > ("AB94LIERA1",47302,12345676836982); > INSERT INTO TABLE mergedelta_txt_2 > partition(date_key=20170103) > VALUES > ("AB94LIENT1",46834,12345676836978), > ("AB94LIENU0",46718,12345676836978), > ("AB94LIENU1",46844,12345676836978), > ("AB94LIENV0",46719,12345676836978), > ("AB94LIENV1",46844,12345676836978), > ("AB94LIENW0",46728,12345676836978), > ("AB94LIENW1",46854,12345676836978), > ("AB94LIENX0",46728,12345676836978), > ("AB94LIENX1",46854,12345676836978), > ("AB94LIENY0",46737,12345676836978), > ("AB94LIENY1",46863,12345676836978), > ("AB94LIENZ0",46738,12345676836978), > ("AB94LIENZ1",46863,12345676836978), > ("AB94LIERA0",47176,12345676836982), > ("AB94LIERA1",47302,12345676836982), > ("AB94LIERA2",47418,12345676836982), > ("AB94LIERB0",47176,12345676836982), > ("AB94LIERB1",47302,12345676836982), > ("AB94LIERB2",47418,12345676836982), > ("AB94LIERC0",47185,12345676836982); > DROP TABLE IF EXISTS mergebase_1; > CREATE TABLE mergebase_1 ( > id_str varchar(12) , time_key int , value bigint) > PARTITIONED BY (date_key int) > CLUSTERED BY (id_str,time_key) INTO 4 BUCKETS > STORED AS ORC > TBLPROPERTIES ( > 'orc.compress'='SNAPPY', > 'pk_columns'='id_str,date_key,time_key', > 'NO_AUTO_COMPACTION'='true', > 'transactional'='true'); > MERGE INTO mergebase_1 AS base > USING (SELECT * > FROM ( > SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY > id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk > FROM mergedelta_txt_1 > DISTRIBUTE BY date_key > ) rankedtbl > WHERE rankedtbl.rk=1 > ) AS delta > ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND > delta.time_key=base.time_key > WHEN MATCHED THEN UPDATE SET value=delta.value > WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , > delta.value, delta.date_key); > MERGE INTO mergebase_1 AS base > USING (SELECT * > FROM ( > SELECT id_str ,time_key ,value
[jira] [Updated] (HIVE-20825) Hive ACID Merge generates invalid ORC files (bucket files 0 or 3 bytes in length) causing the "Not a valid ORC file" error
[ https://issues.apache.org/jira/browse/HIVE-20825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Zeng updated HIVE-20825: Description: When using Hive ACID Merge (supported with the ORC format) to update/insert data, bucket files with 0 byte or 3 bytes (file content contains three characters: ORC) are generated during MERGE INTO operations which finish with no errors. Subsequent queries on the base table will get "Not a valid ORC file" error. The following script can be used to reproduce the issue(note that with small amount of data like this increasing the number of buckets could result in query working, but with large data set it will fail no matter what bucket size): set hive.auto.convert.join=false; set hive.enforce.bucketing=true; set hive.exec.dynamic.partition.mode = nonstrict; set hive.support.concurrency=true; set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; drop table if exists mergedelta_txt_1; drop table if exists mergedelta_txt_2; CREATE TABLE mergedelta_txt_1 ( id_str varchar(12), time_key int, value bigint) PARTITIONED BY (date_key int) ROW FORMAT DELIMITED STORED AS TEXTFILE; CREATE TABLE mergedelta_txt_2 ( id_str varchar(12), time_key int, value bigint) PARTITIONED BY (date_key int) ROW FORMAT DELIMITED STORED AS TEXTFILE; INSERT INTO TABLE mergedelta_txt_1 partition(date_key=20170103) VALUES ("AB94LIENR0",46700,12345676836978), ("AB94LIENR1",46825,12345676836978), ("AB94LIENS0",46709,12345676836978), ("AB94LIENS1",46834,12345676836978), ("AB94LIENT0",46709,12345676836978), ("AB94LIENT1",46834,12345676836978), ("AB94LIENU0",46718,12345676836978), ("AB94LIENU1",46844,12345676836978), ("AB94LIENV0",46719,12345676836978), ("AB94LIENV1",46844,12345676836978), ("AB94LIENW0",46728,12345676836978), ("AB94LIENW1",46854,12345676836978), ("AB94LIENX0",46728,12345676836978), ("AB94LIENX1",46854,12345676836978), ("AB94LIENY0",46737,12345676836978), ("AB94LIENY1",46863,12345676836978), ("AB94LIENZ0",46738,12345676836978), ("AB94LIENZ1",46863,12345676836978), ("AB94LIERA0",47176,12345676836982), ("AB94LIERA1",47302,12345676836982); INSERT INTO TABLE mergedelta_txt_2 partition(date_key=20170103) VALUES ("AB94LIENT1",46834,12345676836978), ("AB94LIENU0",46718,12345676836978), ("AB94LIENU1",46844,12345676836978), ("AB94LIENV0",46719,12345676836978), ("AB94LIENV1",46844,12345676836978), ("AB94LIENW0",46728,12345676836978), ("AB94LIENW1",46854,12345676836978), ("AB94LIENX0",46728,12345676836978), ("AB94LIENX1",46854,12345676836978), ("AB94LIENY0",46737,12345676836978), ("AB94LIENY1",46863,12345676836978), ("AB94LIENZ0",46738,12345676836978), ("AB94LIENZ1",46863,12345676836978), ("AB94LIERA0",47176,12345676836982), ("AB94LIERA1",47302,12345676836982), ("AB94LIERA2",47418,12345676836982), ("AB94LIERB0",47176,12345676836982), ("AB94LIERB1",47302,12345676836982), ("AB94LIERB2",47418,12345676836982), ("AB94LIERC0",47185,12345676836982); DROP TABLE IF EXISTS mergebase_1; CREATE TABLE mergebase_1 ( id_str varchar(12) , time_key int , value bigint) PARTITIONED BY (date_key int) CLUSTERED BY (id_str,time_key) INTO 4 BUCKETS STORED AS ORC TBLPROPERTIES ( 'orc.compress'='SNAPPY', 'pk_columns'='id_str,date_key,time_key', 'NO_AUTO_COMPACTION'='true', 'transactional'='true'); MERGE INTO mergebase_1 AS base USING (SELECT * FROM ( SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk FROM mergedelta_txt_1 DISTRIBUTE BY date_key ) rankedtbl WHERE rankedtbl.rk=1 ) AS delta ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND delta.time_key=base.time_key WHEN MATCHED THEN UPDATE SET value=delta.value WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , delta.value, delta.date_key); MERGE INTO mergebase_1 AS base USING (SELECT * FROM ( SELECT id_str ,time_key ,value, date_key, rank() OVER (PARTITION BY id_str,date_key,time_key ORDER BY id_str,date_key,time_key) AS rk FROM mergedelta_txt_2 DISTRIBUTE BY date_key ) rankedtbl WHERE rankedtbl.rk=1 ) AS delta ON delta.id_str=base.id_str AND delta.date_key=base.date_key AND delta.time_key=base.time_key WHEN MATCHED THEN UPDATE SET value=delta.value WHEN NOT MATCHED THEN INSERT VALUES ( delta.id_str , delta.time_key , delta.value, delta.date_key); select count(*) from mergebase_1; was: When using Hive ACID Merge (supported with the ORC format) to update/insert data, bucket files with 0 byte or 3 bytes (file content is three character: ORC) are generated during MERGE INTO operations which finish with no errors. Subsequent queries on the base table will get "Not a valid ORC file" error. The following script can be used to reproduce the issue(note that with small amount of data like this increase the number of bucket could result in query to work, but with la
[jira] [Commented] (HIVE-20811) Turn on dynamic partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1288#comment-1288 ] Hive QA commented on HIVE-20811: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 31s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 26s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14655/dev-support/hive-personality.sh | | git revision | master / 1002e89 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: common U: common | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14655/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Turn on dynamic partitioned hash join > - > > Key: HIVE-20811 > URL: https://issues.apache.org/jira/browse/HIVE-20811 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20811.1.patch > > > Currently it is off by default. > Turning if ON by default will help uncover and fix issues if there are any. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20811) Turn on dynamic partitioned hash join
[ https://issues.apache.org/jira/browse/HIVE-20811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1297#comment-1297 ] Hive QA commented on HIVE-20811: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945661/HIVE-20811.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15507 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez2] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez_empty] (batchId=164) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cross_prod_3] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_hint] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[murmur_hash_migration] (batchId=174) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_join_hash] (batchId=172) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_main] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_full_outer_join] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_leftsemi_mapjoin] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_llap_text_1] (batchId=162) org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query6] (batchId=271) org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[query6] (batchId=271) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14655/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14655/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14655/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945661 - PreCommit-HIVE-Build > Turn on dynamic partitioned hash join > - > > Key: HIVE-20811 > URL: https://issues.apache.org/jira/browse/HIVE-20811 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20811.1.patch > > > Currently it is off by default. > Turning if ON by default will help uncover and fix issues if there are any. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1301#comment-1301 ] Hive QA commented on HIVE-20793: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 24s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 30s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 17s{color} | {color:blue} standalone-metastore/metastore-common in master has 28 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 44s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 6s{color} | {color:blue} standalone-metastore/metastore-server in master has 181 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 32s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 20s{color} | {color:red} hcatalog-unit in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 37s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s{color} | {color:red} ql: The patch generated 8 new + 381 unchanged - 1 fixed = 389 total (was 382) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 34 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 42m 27s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14656/dev-support/hive-personality.sh | | git revision | master / 1002e89 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | mvninstall | http://104.198.109.242/logs//PreCommit-HIVE-Build-14656/yetus/patch-mvninstall-itests_hcatalog-unit.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14656/yetus/diff-checkstyle-ql.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-14656/yetus/whitespace-eol.txt | | modules | C: common standalone-metastore/metastore-common metastore itests/hcatalog-unit ql standalone-metastore/metastore-server U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14656/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey S