[jira] [Commented] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1309#comment-1309 ] Hive QA commented on HIVE-20793: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945834/HIVE-20793.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 15509 tests executed *Failed tests:* {noformat} org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testComplexQuery (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testDataTypes (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testEscapedStrings (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testKillQuery (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testLlapInputFormatEndToEnd (batchId=257) org.apache.hive.jdbc.TestJdbcWithMiniLlapArrow.testNonAsciiStrings (batchId=257) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14656/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14656/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14656/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 6 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945834 - PreCommit-HIVE-Build > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, > HIVE-20793.02.nogen.patch, HIVE-20793.02.patch, HIVE-20793.nogen.patch, > HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20813) udf to_epoch_milli need to support timestamp without time zone as well
[ https://issues.apache.org/jira/browse/HIVE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1329#comment-1329 ] Hive QA commented on HIVE-20813: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 7s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 40s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 35s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14658/dev-support/hive-personality.sh | | git revision | master / 1002e89 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14658/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > udf to_epoch_milli need to support timestamp without time zone as well > -- > > Key: HIVE-20813 > URL: https://issues.apache.org/jira/browse/HIVE-20813 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20813.patch > > > Currently the following query will fail with a cast exception (tries to cast > timestamp to timestamp with local timezone). > {code} > select to_epoch_milli(current_timestamp) > {code} > As a simple fix we need to add support for timestamp object inspector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20813) udf to_epoch_milli need to support timestamp without time zone as well
[ https://issues.apache.org/jira/browse/HIVE-20813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1338#comment-1338 ] Hive QA commented on HIVE-20813: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945677/HIVE-20813.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15502 tests executed *Failed tests:* {noformat} TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=195) [druidmini_dynamic_partition.q,druidmini_test_ts.q,druidmini_expressions.q,druidmini_test_alter.q,druidmini_test_insert.q] {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14658/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14658/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14658/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945677 - PreCommit-HIVE-Build > udf to_epoch_milli need to support timestamp without time zone as well > -- > > Key: HIVE-20813 > URL: https://issues.apache.org/jira/browse/HIVE-20813 > Project: Hive > Issue Type: Bug >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-20813.patch > > > Currently the following query will fail with a cast exception (tries to cast > timestamp to timestamp with local timezone). > {code} > select to_epoch_milli(current_timestamp) > {code} > As a simple fix we need to add support for timestamp object inspector. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[ https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1351#comment-1351 ] Hive QA commented on HIVE-20617: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945683/HIVE-20617.12.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14659/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14659/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14659/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12945683/HIVE-20617.12.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12945683 - PreCommit-HIVE-Build > Fix type of constants in IN expressions to have correct type > > > Key: HIVE-20617 > URL: https://issues.apache.org/jira/browse/HIVE-20617 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch, > HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch, > HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch, > HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch > > > In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the > comparision in UDFIn may fail because if a or b is of char/varchar type the > constants will retain string type - especially after PointlookupOptimizer > compaction. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20815) JdbcRecordReader.next shall not eat exception
[ https://issues.apache.org/jira/browse/HIVE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1371#comment-1371 ] Hive QA commented on HIVE-20815: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 9s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 21s{color} | {color:blue} jdbc-handler in master has 12 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 10m 37s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14660/dev-support/hive-personality.sh | | git revision | master / 1002e89 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: jdbc-handler U: jdbc-handler | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14660/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > JdbcRecordReader.next shall not eat exception > - > > Key: HIVE-20815 > URL: https://issues.apache.org/jira/browse/HIVE-20815 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20815.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20815) JdbcRecordReader.next shall not eat exception
[ https://issues.apache.org/jira/browse/HIVE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1393#comment-1393 ] Hive QA commented on HIVE-20815: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945711/HIVE-20815.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15507 tests executed *Failed tests:* {noformat} org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions (batchId=257) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=257) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent (batchId=257) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead (batchId=257) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=257) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime (batchId=257) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime (batchId=257) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14660/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14660/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14660/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945711 - PreCommit-HIVE-Build > JdbcRecordReader.next shall not eat exception > - > > Key: HIVE-20815 > URL: https://issues.apache.org/jira/browse/HIVE-20815 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20815.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19353) Vectorization: ConstantVectorExpression --> RuntimeException: Unexpected column vector type LIST
[ https://issues.apache.org/jira/browse/HIVE-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1400#comment-1400 ] Hive QA commented on HIVE-19353: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12921195/HIVE-19353.02.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14661/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14661/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14661/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-10-28 13:34:07.063 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-14661/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-10-28 13:34:07.082 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 9.3.25.v20180904 (Laszlo Bodor via Thejas Nair) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 1002e89 HIVE-20638 : Upgrade version of Jetty to 9.3.25.v20180904 (Laszlo Bodor via Thejas Nair) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-10-28 13:34:12.223 + rm -rf ../yetus_PreCommit-HIVE-Build-14661 + mkdir ../yetus_PreCommit-HIVE-Build-14661 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-14661 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14661/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: itests/src/test/resources/testconfiguration.properties:747 Falling back to three-way merge... Applied patch to 'itests/src/test/resources/testconfiguration.properties' with conflicts. Going to apply patch with: git apply -p0 /data/hiveptest/working/scratch/build.patch:45: trailing whitespace. c17 timestamp, /data/hiveptest/working/scratch/build.patch:46: trailing whitespace. c18 decimal(16,7), /data/hiveptest/working/scratch/build.patch:47: trailing whitespace. c19 binary, /data/hiveptest/working/scratch/build.patch:75: trailing whitespace. c17 timestamp, /data/hiveptest/working/scratch/build.patch:76: trailing whitespace. c18 decimal(16,7), error: patch failed: itests/src/test/resources/testconfiguration.properties:747 Falling back to three-way merge... Applied patch to 'itests/src/test/resources/testconfiguration.properties' with conflicts. U itests/src/test/resources/testconfiguration.properties warning: squelched 6 whitespace errors warning: 11 lines add whitespace errors. + result=1 + '[' 1 -ne 0 ']' + rm -rf yetus_PreCommit-HIVE-Build-14661 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12921195 - PreCommit-HIVE-Build > Vectorization: ConstantVectorExpression --> RuntimeException: Unexpected > column vector type LIST > - > > Key: HIVE-19353 > URL: https://issues.apache.org/jira/browse/HIVE-19353 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Priority: Critical > Attachments: HIVE-19353.01.patch, HIVE-19353.02.patch > > > Found by enabling vectorization for > org.apache.hive.jdbc.TestJdbcDriver2.testResultSetMetaData > {noformat} > Caused by: java.lang.RuntimeException: Unexpected colum
[jira] [Commented] (HIVE-20824) HiveServer2 is still leaking file handles when using ADD JAR
[ https://issues.apache.org/jira/browse/HIVE-20824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1402#comment-1402 ] hezhang commented on HIVE-20824: [~aihuaxu] hi What do you think of this question? > HiveServer2 is still leaking file handles when using ADD JAR > - > > Key: HIVE-20824 > URL: https://issues.apache.org/jira/browse/HIVE-20824 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: hezhang >Priority: Major > > In Hive version 2.1.1, hiveserver2 is stll leaking jar file handle when using > add jar . When using UDF jar and creating function with this jar, the jar > file will not delete. > {noformat} > Beeline version 2.1.1 by Apache Hive > 0: jdbc:hive2://localhost:7001/> add jar hdfs:///udf/UDF-1.0-SNAPSHOT.jar;No > rows affected (0.45 seconds) > 0: jdbc:hive2://localhost:7001/> create temporary function my_md5 as > 'com.neu.hive.UDF.Md5';No rows affected (0.082 seconds) > 0: jdbc:hive2://localhost:7001/> select my_md5(name) from > test;+---+--+ > | c0 | > +---+--+ > | ad6f6c09bd4accc056568714692ff4cd | > | 836db9ecc83d8397a5d0205eb8344d4c | > | 5a7fcd4f1c785c8ef4931a5a9c698ac0 | > | 0e2bb4d743f2a009d4b84a9338c98f7c | > | 3f6697692f4506cf311c95848f3536d3 | > | 0de5fc94d0ba53fc7a44f0f136e82fbb | > | 04ef3e58a6b569c1a404efb90d50f906 | > | f505e0a965c8e721eef7e111c66e5c29 | > | c053092216d6f623615d5be978b98e67 | > | 8c2c58079bc3d10be0da79c5d9940f31 | > +---+--+ > 10 rows selected (0.12 seconds) > {noformat} > the hiveserver2 PID is 3856,then list the file handle. > {noformat} > [hadoop@10 ~]$ lsof -p 3856 | grep SNAPSHOT.jar > java 3856 hadoop DEL REG 252,16 263184 > /data/emr/hive/tmp/2f3bc7b2-c385-4552-a54e-b575b4338d63_resources/UDF-1.0-SNAPSHOT.jar > java 3856 hadoop DEL REG 252,16 263178 > /data/emr/hive/tmp/986015db-52bf-4f6a-99f8-977de471d528_resources/UDF-4.0-SNAPSHOT.jar > java 3856 hadoop 774r REG 252,16 90885742 263178 > /data/emr/hive/tmp/986015db-52bf-4f6a-99f8-977de471d528_resources/UDF-4.0-SNAPSHOT.jar > (deleted) > java 3856 hadoop 775r REG 252,16 104220075 263184 > /data/emr/hive/tmp/2f3bc7b2-c385-4552-a54e-b575b4338d63_resources/UDF-1.0-SNAPSHOT.jar > (deleted) > {noformat} > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-20824) HiveServer2 is still leaking file handles when using ADD JAR
[ https://issues.apache.org/jira/browse/HIVE-20824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1402#comment-1402 ] hezhang edited comment on HIVE-20824 at 10/28/18 1:47 PM: -- [~aihuaxu] hi What do you think of this question? thanks :D was (Author: heiheizhang): [~aihuaxu] hi What do you think of this question? > HiveServer2 is still leaking file handles when using ADD JAR > - > > Key: HIVE-20824 > URL: https://issues.apache.org/jira/browse/HIVE-20824 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.1 >Reporter: hezhang >Priority: Major > > In Hive version 2.1.1, hiveserver2 is stll leaking jar file handle when using > add jar . When using UDF jar and creating function with this jar, the jar > file will not delete. > {noformat} > Beeline version 2.1.1 by Apache Hive > 0: jdbc:hive2://localhost:7001/> add jar hdfs:///udf/UDF-1.0-SNAPSHOT.jar;No > rows affected (0.45 seconds) > 0: jdbc:hive2://localhost:7001/> create temporary function my_md5 as > 'com.neu.hive.UDF.Md5';No rows affected (0.082 seconds) > 0: jdbc:hive2://localhost:7001/> select my_md5(name) from > test;+---+--+ > | c0 | > +---+--+ > | ad6f6c09bd4accc056568714692ff4cd | > | 836db9ecc83d8397a5d0205eb8344d4c | > | 5a7fcd4f1c785c8ef4931a5a9c698ac0 | > | 0e2bb4d743f2a009d4b84a9338c98f7c | > | 3f6697692f4506cf311c95848f3536d3 | > | 0de5fc94d0ba53fc7a44f0f136e82fbb | > | 04ef3e58a6b569c1a404efb90d50f906 | > | f505e0a965c8e721eef7e111c66e5c29 | > | c053092216d6f623615d5be978b98e67 | > | 8c2c58079bc3d10be0da79c5d9940f31 | > +---+--+ > 10 rows selected (0.12 seconds) > {noformat} > the hiveserver2 PID is 3856,then list the file handle. > {noformat} > [hadoop@10 ~]$ lsof -p 3856 | grep SNAPSHOT.jar > java 3856 hadoop DEL REG 252,16 263184 > /data/emr/hive/tmp/2f3bc7b2-c385-4552-a54e-b575b4338d63_resources/UDF-1.0-SNAPSHOT.jar > java 3856 hadoop DEL REG 252,16 263178 > /data/emr/hive/tmp/986015db-52bf-4f6a-99f8-977de471d528_resources/UDF-4.0-SNAPSHOT.jar > java 3856 hadoop 774r REG 252,16 90885742 263178 > /data/emr/hive/tmp/986015db-52bf-4f6a-99f8-977de471d528_resources/UDF-4.0-SNAPSHOT.jar > (deleted) > java 3856 hadoop 775r REG 252,16 104220075 263184 > /data/emr/hive/tmp/2f3bc7b2-c385-4552-a54e-b575b4338d63_resources/UDF-1.0-SNAPSHOT.jar > (deleted) > {noformat} > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[ https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1413#comment-1413 ] Hive QA commented on HIVE-20682: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 10s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 53s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 36s{color} | {color:blue} service in master has 48 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} The patch hive-unit passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} ql: The patch generated 0 new + 217 unchanged - 2 fixed = 217 total (was 219) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} The patch service passed checkstyle {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 31m 57s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14662/dev-support/hive-personality.sh | | git revision | master / 1002e89 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: itests/hive-unit ql service U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14662/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Async query execution can potentially fail if shared sessionHive is closed by > master thread. > > > Key: HIVE-20682 > URL: https://issues.apache.org/jira/browse/HIVE-20682 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0, 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, > HIVE-20682.03.patch, HIVE-20682.04.pa
[jira] [Updated] (HIVE-20824) HiveServer2 is still leaking file handles when using ADD JAR
[ https://issues.apache.org/jira/browse/HIVE-20824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hezhang updated HIVE-20824: --- Affects Version/s: 2.3.2 > HiveServer2 is still leaking file handles when using ADD JAR > - > > Key: HIVE-20824 > URL: https://issues.apache.org/jira/browse/HIVE-20824 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 2.1.1, 2.3.2 >Reporter: hezhang >Priority: Major > > In Hive version 2.1.1, hiveserver2 is stll leaking jar file handle when using > add jar . When using UDF jar and creating function with this jar, the jar > file will not delete. > {noformat} > Beeline version 2.1.1 by Apache Hive > 0: jdbc:hive2://localhost:7001/> add jar hdfs:///udf/UDF-1.0-SNAPSHOT.jar;No > rows affected (0.45 seconds) > 0: jdbc:hive2://localhost:7001/> create temporary function my_md5 as > 'com.neu.hive.UDF.Md5';No rows affected (0.082 seconds) > 0: jdbc:hive2://localhost:7001/> select my_md5(name) from > test;+---+--+ > | c0 | > +---+--+ > | ad6f6c09bd4accc056568714692ff4cd | > | 836db9ecc83d8397a5d0205eb8344d4c | > | 5a7fcd4f1c785c8ef4931a5a9c698ac0 | > | 0e2bb4d743f2a009d4b84a9338c98f7c | > | 3f6697692f4506cf311c95848f3536d3 | > | 0de5fc94d0ba53fc7a44f0f136e82fbb | > | 04ef3e58a6b569c1a404efb90d50f906 | > | f505e0a965c8e721eef7e111c66e5c29 | > | c053092216d6f623615d5be978b98e67 | > | 8c2c58079bc3d10be0da79c5d9940f31 | > +---+--+ > 10 rows selected (0.12 seconds) > {noformat} > the hiveserver2 PID is 3856,then list the file handle. > {noformat} > [hadoop@10 ~]$ lsof -p 3856 | grep SNAPSHOT.jar > java 3856 hadoop DEL REG 252,16 263184 > /data/emr/hive/tmp/2f3bc7b2-c385-4552-a54e-b575b4338d63_resources/UDF-1.0-SNAPSHOT.jar > java 3856 hadoop DEL REG 252,16 263178 > /data/emr/hive/tmp/986015db-52bf-4f6a-99f8-977de471d528_resources/UDF-4.0-SNAPSHOT.jar > java 3856 hadoop 774r REG 252,16 90885742 263178 > /data/emr/hive/tmp/986015db-52bf-4f6a-99f8-977de471d528_resources/UDF-4.0-SNAPSHOT.jar > (deleted) > java 3856 hadoop 775r REG 252,16 104220075 263184 > /data/emr/hive/tmp/2f3bc7b2-c385-4552-a54e-b575b4338d63_resources/UDF-1.0-SNAPSHOT.jar > (deleted) > {noformat} > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[ https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1430#comment-1430 ] Hive QA commented on HIVE-20682: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945741/HIVE-20682.04.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15507 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14662/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14662/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14662/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12945741 - PreCommit-HIVE-Build > Async query execution can potentially fail if shared sessionHive is closed by > master thread. > > > Key: HIVE-20682 > URL: https://issues.apache.org/jira/browse/HIVE-20682 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0, 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, > HIVE-20682.03.patch, HIVE-20682.04.patch > > > *Problem description:* > The master thread initializes the *sessionHive* object in *HiveSessionImpl* > class when we open a new session for a client connection and by default all > queries from this connection shares the same sessionHive object. > If the master thread executes a *synchronous* query, it closes the > sessionHive object (referred via thread local hiveDb) if > {{Hive.isCompatible}} returns false and sets new Hive object in thread local > HiveDb but doesn't change the sessionHive object in the session. Whereas, > *asynchronous* query execution via async threads never closes the sessionHive > object and it just creates a new one if needed and sets it as their thread > local hiveDb. > So, the problem can happen in the case where an *asynchronous* query is being > executed by async threads refers to sessionHive object and the master thread > receives a *synchronous* query that closes the same sessionHive object. > Also, each query execution overwrites the thread local hiveDb object to > sessionHive object which potentially leaks a metastore connection if the > previous synchronous query execution re-created the Hive object. > *Possible Fix:* > The *sessionHive* object could be shared my multiple threads and so it > shouldn't be allowed to be closed by any query execution threads when they > re-create the Hive object due to changes in Hive configurations. But the Hive > objects created by query execution threads should be closed when the thread > exits. > So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive > object which should be set to *false* for *sessionHive* and would be > forcefully closed when the session is closed or released. > cc [~pvary] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20682) Async query execution can potentially fail if shared sessionHive is closed by master thread.
[ https://issues.apache.org/jira/browse/HIVE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1432#comment-1432 ] Hive QA commented on HIVE-20682: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945741/HIVE-20682.04.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14663/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14663/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14663/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12945741/HIVE-20682.04.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12945741 - PreCommit-HIVE-Build > Async query execution can potentially fail if shared sessionHive is closed by > master thread. > > > Key: HIVE-20682 > URL: https://issues.apache.org/jira/browse/HIVE-20682 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.0, 4.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-20682.01.patch, HIVE-20682.02.patch, > HIVE-20682.03.patch, HIVE-20682.04.patch > > > *Problem description:* > The master thread initializes the *sessionHive* object in *HiveSessionImpl* > class when we open a new session for a client connection and by default all > queries from this connection shares the same sessionHive object. > If the master thread executes a *synchronous* query, it closes the > sessionHive object (referred via thread local hiveDb) if > {{Hive.isCompatible}} returns false and sets new Hive object in thread local > HiveDb but doesn't change the sessionHive object in the session. Whereas, > *asynchronous* query execution via async threads never closes the sessionHive > object and it just creates a new one if needed and sets it as their thread > local hiveDb. > So, the problem can happen in the case where an *asynchronous* query is being > executed by async threads refers to sessionHive object and the master thread > receives a *synchronous* query that closes the same sessionHive object. > Also, each query execution overwrites the thread local hiveDb object to > sessionHive object which potentially leaks a metastore connection if the > previous synchronous query execution re-created the Hive object. > *Possible Fix:* > The *sessionHive* object could be shared my multiple threads and so it > shouldn't be allowed to be closed by any query execution threads when they > re-create the Hive object due to changes in Hive configurations. But the Hive > objects created by query execution threads should be closed when the thread > exits. > So, it is proposed to have an *isAllowClose* flag (default: *true*) in Hive > object which should be set to *false* for *sessionHive* and would be > forcefully closed when the session is closed or released. > cc [~pvary] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20792) Inserting timestamp with zones truncates the data
[ https://issues.apache.org/jira/browse/HIVE-20792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1434#comment-1434 ] Jesus Camacho Rodriguez commented on HIVE-20792: +1 > Inserting timestamp with zones truncates the data > - > > Key: HIVE-20792 > URL: https://issues.apache.org/jira/browse/HIVE-20792 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Attachments: HIVE-20792.1.patch, HIVE-20792.2.patch, > HIVE-20792.3.patch > > > For example with the table: > {code} > CREATE TABLE myTable > ( > a TIMESTAMP > ) > STORED AS ORC > tblproperties("transactional"="true"); > {code} > The following inserts store the wrong data: > {code} > INSERT INTO myTable VALUES("2018-10-19 10:35:00 UTC"); -> 2018-10-19 > 00:00:00.0 > INSERT INTO myTable VALUES("2018-10-19 10:35:00 ZZZ"); -> 2018-10-19 > 00:00:00.0 > {code} > The second one should fail since ZZZ is not a time zone. > Similarly if the column is of type DATE, > {code} > INSERT INTO myTableDate VALUES("2018-10-19 "); -> 2018-10-19 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20792) Inserting timestamp with zones truncates the data
[ https://issues.apache.org/jira/browse/HIVE-20792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-20792: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master, thanks [~jmarhuen]! > Inserting timestamp with zones truncates the data > - > > Key: HIVE-20792 > URL: https://issues.apache.org/jira/browse/HIVE-20792 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 3.1.0 >Reporter: Jaume M >Assignee: Jaume M >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20792.1.patch, HIVE-20792.2.patch, > HIVE-20792.3.patch > > > For example with the table: > {code} > CREATE TABLE myTable > ( > a TIMESTAMP > ) > STORED AS ORC > tblproperties("transactional"="true"); > {code} > The following inserts store the wrong data: > {code} > INSERT INTO myTable VALUES("2018-10-19 10:35:00 UTC"); -> 2018-10-19 > 00:00:00.0 > INSERT INTO myTable VALUES("2018-10-19 10:35:00 ZZZ"); -> 2018-10-19 > 00:00:00.0 > {code} > The second one should fail since ZZZ is not a time zone. > Similarly if the column is of type DATE, > {code} > INSERT INTO myTableDate VALUES("2018-10-19 "); -> 2018-10-19 > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20796) jdbc URL can contain sensitive information that should not be logged
[ https://issues.apache.org/jira/browse/HIVE-20796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1444#comment-1444 ] Hive QA commented on HIVE-20796: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 7s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 59s{color} | {color:blue} standalone-metastore/metastore-server in master has 181 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 32s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14664/dev-support/hive-personality.sh | | git revision | master / 1002e89 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: standalone-metastore/metastore-server U: standalone-metastore/metastore-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14664/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > jdbc URL can contain sensitive information that should not be logged > > > Key: HIVE-20796 > URL: https://issues.apache.org/jira/browse/HIVE-20796 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20796.01.patch, HIVE-20796.02.patch, > HIVE-20796.03.patch, HIVE-20796.04.patch, HIVE-20796.05.patch > > > It is possible to put passwords in the jdbc connection url and some jdbc > drivers will supposedly use that. (derby, mysql). This information is > considered sensitive, and should be masked out, while logging the connection > url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20796) jdbc URL can contain sensitive information that should not be logged
[ https://issues.apache.org/jira/browse/HIVE-20796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1465#comment-1465 ] Hive QA commented on HIVE-20796: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945743/HIVE-20796.05.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15508 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14664/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14664/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14664/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12945743 - PreCommit-HIVE-Build > jdbc URL can contain sensitive information that should not be logged > > > Key: HIVE-20796 > URL: https://issues.apache.org/jira/browse/HIVE-20796 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Laszlo Pinter >Assignee: Laszlo Pinter >Priority: Major > Attachments: HIVE-20796.01.patch, HIVE-20796.02.patch, > HIVE-20796.03.patch, HIVE-20796.04.patch, HIVE-20796.05.patch > > > It is possible to put passwords in the jdbc connection url and some jdbc > drivers will supposedly use that. (derby, mysql). This information is > considered sensitive, and should be masked out, while logging the connection > url. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20782) Cleaning some unused code
[ https://issues.apache.org/jira/browse/HIVE-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1474#comment-1474 ] Hive QA commented on HIVE-20782: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 46s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} ql: The patch generated 0 new + 0 unchanged - 6 fixed = 0 total (was 6) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 55s{color} | {color:green} ql generated 0 new + 2316 unchanged - 1 fixed = 2316 total (was 2317) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 31s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14666/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14666/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Cleaning some unused code > - > > Key: HIVE-20782 > URL: https://issues.apache.org/jira/browse/HIVE-20782 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-20782.2.patch, HIVE-20782.2.patch, > HIVE-20782.3.patch, HIVE-20782.patch > > > Am making my way into the vectorize code and trying understand the APIs. Ran > into this unused one, i guess it is not used anymore. > [~ashutoshc] maybe can explain as you are the main contributor to this file > {code} > a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedSerde.java{code} > ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20617) Fix type of constants in IN expressions to have correct type
[ https://issues.apache.org/jira/browse/HIVE-20617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-20617: -- Assignee: Zoltan Haindrich (was: Jesus Camacho Rodriguez) > Fix type of constants in IN expressions to have correct type > > > Key: HIVE-20617 > URL: https://issues.apache.org/jira/browse/HIVE-20617 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-20617.01.patch, HIVE-20617.02.patch, > HIVE-20617.03.patch, HIVE-20617.05.patch, HIVE-20617.06.patch, > HIVE-20617.07.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.08.patch, HIVE-20617.08.patch, HIVE-20617.08.patch, > HIVE-20617.09.patch, HIVE-20617.10.patch, HIVE-20617.10.patch, > HIVE-20617.11.patch, HIVE-20617.11.patch, HIVE-20617.12.patch > > > In statements like {{struct(a,b) IN (const struct('x','y'), ... )}} the > comparision in UDFIn may fail because if a or b is of char/varchar type the > constants will retain string type - especially after PointlookupOptimizer > compaction. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20782) Cleaning some unused code
[ https://issues.apache.org/jira/browse/HIVE-20782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1499#comment-1499 ] Hive QA commented on HIVE-20782: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945804/HIVE-20782.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15506 tests executed *Failed tests:* {noformat} TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=196) [druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q] {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14666/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14666/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14666/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945804 - PreCommit-HIVE-Build > Cleaning some unused code > - > > Key: HIVE-20782 > URL: https://issues.apache.org/jira/browse/HIVE-20782 > Project: Hive > Issue Type: Improvement >Reporter: slim bouguerra >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-20782.2.patch, HIVE-20782.2.patch, > HIVE-20782.3.patch, HIVE-20782.patch > > > Am making my way into the vectorize code and trying understand the APIs. Ran > into this unused one, i guess it is not used anymore. > [~ashutoshc] maybe can explain as you are the main contributor to this file > {code} > a/ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedSerde.java{code} > ? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20748) Disable materialized view rewriting when plan pattern is not allowed
[ https://issues.apache.org/jira/browse/HIVE-20748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1511#comment-1511 ] Hive QA commented on HIVE-20748: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 39s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 39s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} ql: The patch generated 0 new + 1062 unchanged - 3 fixed = 1062 total (was 1065) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 53s{color} | {color:red} ql generated 1 new + 2316 unchanged - 1 fixed = 2317 total (was 2317) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 33s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Load of known null value in org.apache.hadoop.hive.ql.parse.CalcitePlanner.canCBOHandleAst(ASTNode, QB, CalcitePlanner$PreCboCtx) At CalcitePlanner.java:in org.apache.hadoop.hive.ql.parse.CalcitePlanner.canCBOHandleAst(ASTNode, QB, CalcitePlanner$PreCboCtx) At CalcitePlanner.java:[line 860] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14667/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14667/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14667/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Disable materialized view rewriting when plan pattern is not allowed > > > Key: HIVE-20748 > URL: https://issues.apache.org/jira/browse/HIVE-20748 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20748.01.patch, HIVE-20748.01.patch, > HIVE-20748.02.patch, HIVE-20748.02.patch, HIVE-20748.patch > > > For instance, currently rewriting algorithm does not support some operators. > Or we cannot have non-deterministic function in the MV definition. In those > cases, we should fail either when we try to create the MV with rewriting > enabled, or when when we enable the rewriting for a MV already created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20748) Disable materialized view rewriting when plan pattern is not allowed
[ https://issues.apache.org/jira/browse/HIVE-20748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1522#comment-1522 ] Hive QA commented on HIVE-20748: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945811/HIVE-20748.02.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 15510 tests executed *Failed tests:* {noformat} TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=196) [druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join0] (batchId=63) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parallel_join0] (batchId=80) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[explainuser_2] (batchId=156) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1] (batchId=167) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=190) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1] (batchId=108) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_3] (batchId=108) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=108) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14667/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14667/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14667/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945811 - PreCommit-HIVE-Build > Disable materialized view rewriting when plan pattern is not allowed > > > Key: HIVE-20748 > URL: https://issues.apache.org/jira/browse/HIVE-20748 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20748.01.patch, HIVE-20748.01.patch, > HIVE-20748.02.patch, HIVE-20748.02.patch, HIVE-20748.patch > > > For instance, currently rewriting algorithm does not support some operators. > Or we cannot have non-deterministic function in the MV definition. In those > cases, we should fail either when we try to create the MV with rewriting > enabled, or when when we enable the rewriting for a MV already created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20486) Kafka: Use Row SerDe + vectorization
[ https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1524#comment-1524 ] Hive QA commented on HIVE-20486: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945942/HIVE-20486.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14668/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14668/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14668/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2018-10-28 19:07:24.195 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-14668/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2018-10-28 19:07:24.198 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 54bba9c HIVE-20792: Inserting timestamp with zones truncates the data (Jaume Marhuenda, reviewed by Jesus Camacho Rodriguez) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 54bba9c HIVE-20792: Inserting timestamp with zones truncates the data (Jaume Marhuenda, reviewed by Jesus Camacho Rodriguez) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2018-10-28 19:07:24.813 + rm -rf ../yetus_PreCommit-HIVE-Build-14668 + mkdir ../yetus_PreCommit-HIVE-Build-14668 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-14668 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-14668/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: git apply -p0 /data/hiveptest/working/scratch/build.patch:671: trailing whitespace. Map 1 /data/hiveptest/working/scratch/build.patch:699: trailing whitespace. hdfs://### HDFS PATH ### /data/hiveptest/working/scratch/build.patch:795: trailing whitespace. /data/hiveptest/working/scratch/build.patch:902: trailing whitespace. Reducer 2 /data/hiveptest/working/scratch/build.patch:993: trailing whitespace. Map 1 warning: squelched 3 whitespace errors warning: 8 lines add whitespace errors. + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc1866517667617096157.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc1866517667617096157.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] ANTLR Parser Generator Version 3.5.2 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (process-resource-bundles) on project hive-pre-upgrade: Execution process-resource-bundles of goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed. ConcurrentModificationException -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information a
[jira] [Commented] (HIVE-20820) MV partition on clause position
[ https://issues.apache.org/jira/browse/HIVE-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1527#comment-1527 ] Hive QA commented on HIVE-20820: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 56s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 1m 32s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14669/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14669/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > MV partition on clause position > --- > > Key: HIVE-20820 > URL: https://issues.apache.org/jira/browse/HIVE-20820 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-20820.patch > > > It should obey the following syntax as per > https://cwiki.apache.org/confluence/display/Hive/Materialized+views: > {code} > CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name > [DISABLE REWRITE] > [COMMENT materialized_view_comment] > [PARTITIONED ON (col_name, ...)] > [ > [ROW FORMAT row_format] > [STORED AS file_format] > | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)] > ] > [LOCATION hdfs_path] > [TBLPROPERTIES (property_name=property_value, ...)] > AS > ; > {code} > Currently it is positioned just before TBLPROPERTIES. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20820) MV partition on clause position
[ https://issues.apache.org/jira/browse/HIVE-20820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1541#comment-1541 ] Hive QA commented on HIVE-20820: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945821/HIVE-20820.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15508 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14669/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14669/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14669/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12945821 - PreCommit-HIVE-Build > MV partition on clause position > --- > > Key: HIVE-20820 > URL: https://issues.apache.org/jira/browse/HIVE-20820 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-20820.patch > > > It should obey the following syntax as per > https://cwiki.apache.org/confluence/display/Hive/Materialized+views: > {code} > CREATE MATERIALIZED VIEW [IF NOT EXISTS] [db_name.]materialized_view_name > [DISABLE REWRITE] > [COMMENT materialized_view_comment] > [PARTITIONED ON (col_name, ...)] > [ > [ROW FORMAT row_format] > [STORED AS file_format] > | STORED BY 'storage.handler.class.name' [WITH SERDEPROPERTIES (...)] > ] > [LOCATION hdfs_path] > [TBLPROPERTIES (property_name=property_value, ...)] > AS > ; > {code} > Currently it is positioned just before TBLPROPERTIES. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination
[ https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1560#comment-1560 ] Hive QA commented on HIVE-20821: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 42s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 7s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14670/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14670/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Rewrite SUM0 into SUM + COALESCE combination > > > Key: HIVE-20821 > URL: https://issues.apache.org/jira/browse/HIVE-20821 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20821.patch > > > Since SUM0 is not vectorized, but SUM + COALESCE are. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20821) Rewrite SUM0 into SUM + COALESCE combination
[ https://issues.apache.org/jira/browse/HIVE-20821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1578#comment-1578 ] Hive QA commented on HIVE-20821: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945826/HIVE-20821.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15508 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_10] (batchId=176) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_7] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_rewrite_no_join_opt] (batchId=182) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14670/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14670/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14670/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945826 - PreCommit-HIVE-Build > Rewrite SUM0 into SUM + COALESCE combination > > > Key: HIVE-20821 > URL: https://issues.apache.org/jira/browse/HIVE-20821 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-20821.patch > > > Since SUM0 is not vectorized, but SUM + COALESCE are. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20793) add RP namespacing to workload management
[ https://issues.apache.org/jira/browse/HIVE-20793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1585#comment-1585 ] Hive QA commented on HIVE-20793: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945834/HIVE-20793.02.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14671/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14671/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14671/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12945834/HIVE-20793.02.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12945834 - PreCommit-HIVE-Build > add RP namespacing to workload management > - > > Key: HIVE-20793 > URL: https://issues.apache.org/jira/browse/HIVE-20793 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-20793.01.nogen.patch, HIVE-20793.01.patch, > HIVE-20793.02.nogen.patch, HIVE-20793.02.patch, HIVE-20793.nogen.patch, > HIVE-20793.patch > > > The idea is to be able to use the same warehouse for multiple clusters in the > cloud use cases. This scenario is not currently supported by WM. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[ https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1593#comment-1593 ] Hive QA commented on HIVE-20512: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 42s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 42s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} ql: The patch generated 0 new + 4 unchanged - 5 fixed = 4 total (was 9) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 45s{color} | {color:red} ql generated 1 new + 2317 unchanged - 0 fixed = 2318 total (was 2317) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 11s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Increment of volatile field org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.rowNumber in org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber() At SparkRecordHandler.java:in org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber() At SparkRecordHandler.java:[line 109] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14672/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14672/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14672/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Improve record and memory usage logging in SparkRecordHandler > - > > Key: HIVE-20512 > URL: https://issues.apache.org/jira/browse/HIVE-20512 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch, > HIVE-20512.3.patch, HIVE-20512.4.patch > > > We currently log memory usage and # of records processed in Spark tasks, but > we should improve the methodology for how frequently we log this info. > Currently we use the following code: > {code:java} > private long getNextLogThreshold(long currentThreshold) { > // A very simple counter to keep track of number of rows processed by the > // reducer. It dumps > // every 1 million times, and quickly before that > if (currentThreshold >= 100) { > return currentThreshold + 100; > } > return 10 * currentThreshold; > } >
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[ https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1610#comment-1610 ] Hive QA commented on HIVE-20512: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945839/HIVE-20512.4.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 33 failed/errored test(s), 15026 tests executed *Failed tests:* {noformat} TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=196) [druidmini_masking.q,druidmini_test1.q,druidkafkamini_basic.q,druidmini_joins.q,druid_timestamptz.q] TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=154) [intersect_all.q,unionDistinct_1.q,table_nonprintable.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,cte_2.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q] TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=189) [infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join2.q,spark_explain_groupbyshuffle.q,spark_dynamic_partition_pruning.q,spark_combine_equivalent_work.q,orc_merge1.q,spark_use_op_stats.q,orc_merge_diff_fs.q,quotedid_smb.q,truncate_column_buckets.q,spark_vectorized_dynamic_partition_pruning.q,spark_in_process_launcher.q,orc_merge3.q] TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=190) [infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,spark_explainuser_1.q,spark_use_ts_stats_for_mapjoin.q,orc_merge6.q,orc_merge5.q,bucketmapjoin6.q,spark_opt_shuffle_serde.q,temp_table_external.q,spark_dynamic_partition_pruning_6.q,dynamic_rdd_cache.q,auto_sortmerge_join_16.q,vector_outer_join3.q,spark_dynamic_partition_pruning_7.q,schemeAuthority.q,parallel_orderby.q,vector_outer_join1.q,load_hdfs_file_with_space_in_the_name.q,spark_dynamic_partition_pruning_recursive_mapjoin.q,spark_dynamic_partition_pruning_mapjoin_only.q] TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=191) [insert_overwrite_directory2.q,spark_dynamic_partition_pruning_4.q,import_exported_table.q,vector_outer_join0.q,bucket4.q,orc_merge4.q,infer_bucket_sort_merge.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_partitioner.q,constprog_semijoin.q,external_table_with_space_in_location_path.q,spark_constprog_dpp.q,spark_dynamic_partition_pruning_3.q,load_fs2.q,infer_bucket_sort_map_operators.q,spark_dynamic_partition_pruning_2.q,vector_inner_join.q,spark_multi_insert_parallel_orderby.q,remote_script.q] TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=192) [scriptfile1.q,vector_outer_join5.q,file_with_header_footer.q,input16_cc.q,bucket5.q,orc_merge2.q,reduce_deduplicate.q,schemeAuthority2.q,spark_dynamic_partition_pruning_5.q,orc_merge8.q,orc_merge_incompat2.q,infer_bucket_sort_bucketed_table.q,vector_outer_join4.q,disable_merge_for_bucketing.q,orc_merge7.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=110) [bucketmapjoin4.q,bucket_map_join_spark4.q,union21.q,groupby2_noskew.q,timestamp_2.q,date_join1.q,mergejoins.q,smb_mapjoin_11.q,auto_sortmerge_join_3.q,mapjoin_test_outer.q,vectorization_9.q,merge2.q,groupby6_noskew.q,auto_join_without_localtask.q,multi_join_union.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=111) [join_cond_pushdown_unqual4.q,union_remove_7.q,join13.q,join_vc.q,groupby_cube1.q,parquet_vectorization_2.q,bucket_map_join_spark2.q,sample3.q,smb_mapjoin_19.q,union23.q,union.q,union31.q,cbo_udf_udaf.q,ptf_decimal.q,bucketmapjoin2.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=112) [parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,bucketsortoptimize_insert_8.q,stats16.q,union20.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=114) [groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,order2.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,parquet_vectorization_1.q,auto_join11.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=115) [skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_
[jira] [Commented] (HIVE-20486) Kafka: Use Row SerDe + vectorization
[ https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1611#comment-1611 ] Hive QA commented on HIVE-20486: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945942/HIVE-20486.4.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14673/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14673/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14673/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12945942/HIVE-20486.4.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12945942 - PreCommit-HIVE-Build > Kafka: Use Row SerDe + vectorization > > > Key: HIVE-20486 > URL: https://issues.apache.org/jira/browse/HIVE-20486 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Reporter: Gopal V >Assignee: slim bouguerra >Priority: Major > Labels: kafka, vectorization > Fix For: 4.0.0 > > Attachments: HIVE-20486.3.patch, HIVE-20486.3.patch, > HIVE-20486.4.patch, HIVE-20486.patch > > > KafkaHandler returns unvectorized rows which causes the operators downstream > to be slower and sub-optimal. > Hive has a vectorization shim which allows Kafka streams without complex > projections to be wrapped into a vectorized reader via > {{hive.vectorized.use.row.serde.deserialize}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[ https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1627#comment-1627 ] Hive QA commented on HIVE-20778: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 41s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 35s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14674/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14674/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Join reordering may not be triggered if all joins in plan are created by > decorrelation logic > > > Key: HIVE-20778 > URL: https://issues.apache.org/jira/browse/HIVE-20778 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, > HIVE-20778.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[ https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-20512: Attachment: (was: HIVE-20512.4.patch) > Improve record and memory usage logging in SparkRecordHandler > - > > Key: HIVE-20512 > URL: https://issues.apache.org/jira/browse/HIVE-20512 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch, > HIVE-20512.3.patch, HIVE-20512.4.patch > > > We currently log memory usage and # of records processed in Spark tasks, but > we should improve the methodology for how frequently we log this info. > Currently we use the following code: > {code:java} > private long getNextLogThreshold(long currentThreshold) { > // A very simple counter to keep track of number of rows processed by the > // reducer. It dumps > // every 1 million times, and quickly before that > if (currentThreshold >= 100) { > return currentThreshold + 100; > } > return 10 * currentThreshold; > } > {code} > The issue is that after a while, the increase by 10x factor means that you > have to process a huge # of records before this gets triggered. > A better approach would be to log this info at a given interval. This would > help in debugging tasks that are seemingly hung. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[ https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-20512: Attachment: HIVE-20512.4.patch > Improve record and memory usage logging in SparkRecordHandler > - > > Key: HIVE-20512 > URL: https://issues.apache.org/jira/browse/HIVE-20512 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch, > HIVE-20512.3.patch, HIVE-20512.4.patch > > > We currently log memory usage and # of records processed in Spark tasks, but > we should improve the methodology for how frequently we log this info. > Currently we use the following code: > {code:java} > private long getNextLogThreshold(long currentThreshold) { > // A very simple counter to keep track of number of rows processed by the > // reducer. It dumps > // every 1 million times, and quickly before that > if (currentThreshold >= 100) { > return currentThreshold + 100; > } > return 10 * currentThreshold; > } > {code} > The issue is that after a while, the increase by 10x factor means that you > have to process a huge # of records before this gets triggered. > A better approach would be to log this info at a given interval. This would > help in debugging tasks that are seemingly hung. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20778) Join reordering may not be triggered if all joins in plan are created by decorrelation logic
[ https://issues.apache.org/jira/browse/HIVE-20778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1641#comment-1641 ] Hive QA commented on HIVE-20778: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945943/HIVE-20778.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15508 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_notin] (batchId=177) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select] (batchId=169) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_dynamic_partition_pruning_3] (batchId=191) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1] (batchId=190) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in] (batchId=139) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi] (batchId=118) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_notin] (batchId=142) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_null_agg] (batchId=142) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_scalar] (batchId=128) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_select] (batchId=128) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_views] (batchId=116) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_mapjoin_reduce] (batchId=147) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14674/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14674/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14674/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945943 - PreCommit-HIVE-Build > Join reordering may not be triggered if all joins in plan are created by > decorrelation logic > > > Key: HIVE-20778 > URL: https://issues.apache.org/jira/browse/HIVE-20778 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-20778.1.patch, HIVE-20778.2.patch, > HIVE-20778.3.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1647#comment-1647 ] Hive QA commented on HIVE-20740: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 14s{color} | {color:red} /data/hiveptest/logs/PreCommit-HIVE-Build-14675/patches/PreCommit-HIVE-Build-14675.patch does not apply to master. Rebase required? Wrong Branch? See http://cwiki.apache.org/confluence/display/Hive/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14675/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very small amount of time. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20740) Remove global lock in ObjectStore.setConf method
[ https://issues.apache.org/jira/browse/HIVE-20740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1680#comment-1680 ] Hive QA commented on HIVE-20740: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945944/HIVE-20740.05.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 15509 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar.org.apache.hadoop.hive.metastore.TestHiveMetaStoreAlterColumnPar (batchId=240) org.apache.hadoop.hive.metastore.TestObjectStoreInitRetry.testObjStoreRetry (batchId=229) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarDataNucleusUnCaching (batchId=260) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/14675/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/14675/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-14675/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12945944 - PreCommit-HIVE-Build > Remove global lock in ObjectStore.setConf method > > > Key: HIVE-20740 > URL: https://issues.apache.org/jira/browse/HIVE-20740 > Project: Hive > Issue Type: Improvement >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-20740.01.patch, HIVE-20740.02.patch, > HIVE-20740.04.patch, HIVE-20740.05.patch > > > The ObjectStore#setConf method has a global lock which can block other > clients in concurrent workloads. > {code} > @Override > @SuppressWarnings("nls") > public void setConf(Configuration conf) { > // Although an instance of ObjectStore is accessed by one thread, there > may > // be many threads with ObjectStore instances. So the static variables > // pmf and prop need to be protected with locks. > pmfPropLock.lock(); > try { > isInitialized = false; > this.conf = conf; > this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, > ConfVars.HIVE_TXN_STATS_ENABLED); > configureSSL(conf); > Properties propsFromConf = getDataSourceProps(conf); > boolean propsChanged = !propsFromConf.equals(prop); > if (propsChanged) { > if (pmf != null){ > clearOutPmfClassLoaderCache(pmf); > if (!forTwoMetastoreTesting) { > // close the underlying connection pool to avoid leaks > pmf.close(); > } > } > pmf = null; > prop = null; > } > assert(!isActiveTransaction()); > shutdown(); > // Always want to re-create pm as we don't know if it were created by > the > // most recent instance of the pmf > pm = null; > directSql = null; > expressionProxy = null; > openTrasactionCalls = 0; > currentTransaction = null; > transactionStatus = TXN_STATUS.NO_STATE; > initialize(propsFromConf); > String partitionValidationRegex = > MetastoreConf.getVar(this.conf, > ConfVars.PARTITION_NAME_WHITELIST_PATTERN); > if (partitionValidationRegex != null && > !partitionValidationRegex.isEmpty()) { > partitionValidationPattern = > Pattern.compile(partitionValidationRegex); > } else { > partitionValidationPattern = null; > } > // Note, if metrics have not been initialized this will return null, > which means we aren't > // using metrics. Thus we should always check whether this is non-null > before using. > MetricRegistry registry = Metrics.getRegistry(); > if (registry != null) { > directSqlErrors = > Metrics.getOrCreateCounter(MetricsConstants.DIRECTSQL_ERRORS); > } > this.batchSize = MetastoreConf.getIntVar(conf, > ConfVars.RAWSTORE_PARTITION_BATCH_SIZE); > if (!isInitialized) { > throw new RuntimeException( > "Unable to create persistence manager. Check dss.log for details"); > } else { > LOG.debug("Initialized ObjectStore"); > } > } finally { > pmfPropLock.unlock(); > } > } > {code} > The {{pmfPropLock}} is a static object and it disallows any other new > connection to HMS which is trying to instantiate ObjectStore. We should > either remove the lock or reduce the scope of the lock so that it is held for > a very sm
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[ https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1701#comment-1701 ] Hive QA commented on HIVE-20512: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 52s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 42s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} ql: The patch generated 0 new + 4 unchanged - 5 fixed = 4 total (was 9) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 50s{color} | {color:red} ql generated 1 new + 2317 unchanged - 0 fixed = 2318 total (was 2317) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 23s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Increment of volatile field org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.rowNumber in org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber() At SparkRecordHandler.java:in org.apache.hadoop.hive.ql.exec.spark.SparkRecordHandler.incrementRowNumber() At SparkRecordHandler.java:[line 109] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14676/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-14676/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14676/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Improve record and memory usage logging in SparkRecordHandler > - > > Key: HIVE-20512 > URL: https://issues.apache.org/jira/browse/HIVE-20512 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-20512.1.patch, HIVE-20512.2.patch, > HIVE-20512.3.patch, HIVE-20512.4.patch > > > We currently log memory usage and # of records processed in Spark tasks, but > we should improve the methodology for how frequently we log this info. > Currently we use the following code: > {code:java} > private long getNextLogThreshold(long currentThreshold) { > // A very simple counter to keep track of number of rows processed by the > // reducer. It dumps > // every 1 million times, and quickly before that > if (currentThreshold >= 100) { > return currentThreshold + 100; > } > return 10 * currentThreshold; > } >
[jira] [Updated] (HIVE-20486) Kafka: Use Row SerDe + vectorization
[ https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-20486: -- Attachment: HIVE-20486.4.patch > Kafka: Use Row SerDe + vectorization > > > Key: HIVE-20486 > URL: https://issues.apache.org/jira/browse/HIVE-20486 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Reporter: Gopal V >Assignee: slim bouguerra >Priority: Major > Labels: kafka, vectorization > Fix For: 4.0.0 > > Attachments: HIVE-20486.3.patch, HIVE-20486.3.patch, > HIVE-20486.4.patch, HIVE-20486.4.patch, HIVE-20486.patch > > > KafkaHandler returns unvectorized rows which causes the operators downstream > to be slower and sub-optimal. > Hive has a vectorization shim which allows Kafka streams without complex > projections to be wrapped into a vectorized reader via > {{hive.vectorized.use.row.serde.deserialize}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20512) Improve record and memory usage logging in SparkRecordHandler
[ https://issues.apache.org/jira/browse/HIVE-20512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1751#comment-1751 ] Hive QA commented on HIVE-20512: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12945976/HIVE-20512.4.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 30 failed/errored test(s), 15043 tests executed *Failed tests:* {noformat} TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=189) [infer_bucket_sort_reducers_power_two.q,list_bucket_dml_10.q,orc_merge9.q,leftsemijoin_mr.q,bucket6.q,bucketmapjoin7.q,uber_reduce.q,empty_dir_in_table.q,vector_outer_join2.q,spark_explain_groupbyshuffle.q,spark_dynamic_partition_pruning.q,spark_combine_equivalent_work.q,orc_merge1.q,spark_use_op_stats.q,orc_merge_diff_fs.q,quotedid_smb.q,truncate_column_buckets.q,spark_vectorized_dynamic_partition_pruning.q,spark_in_process_launcher.q,orc_merge3.q] TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=190) [infer_bucket_sort_num_buckets.q,gen_udf_example_add10.q,spark_explainuser_1.q,spark_use_ts_stats_for_mapjoin.q,orc_merge6.q,orc_merge5.q,bucketmapjoin6.q,spark_opt_shuffle_serde.q,temp_table_external.q,spark_dynamic_partition_pruning_6.q,dynamic_rdd_cache.q,auto_sortmerge_join_16.q,vector_outer_join3.q,spark_dynamic_partition_pruning_7.q,schemeAuthority.q,parallel_orderby.q,vector_outer_join1.q,load_hdfs_file_with_space_in_the_name.q,spark_dynamic_partition_pruning_recursive_mapjoin.q,spark_dynamic_partition_pruning_mapjoin_only.q] TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=191) [insert_overwrite_directory2.q,spark_dynamic_partition_pruning_4.q,import_exported_table.q,vector_outer_join0.q,bucket4.q,orc_merge4.q,infer_bucket_sort_merge.q,orc_merge_incompat1.q,root_dir_external_table.q,constprog_partitioner.q,constprog_semijoin.q,external_table_with_space_in_location_path.q,spark_constprog_dpp.q,spark_dynamic_partition_pruning_3.q,load_fs2.q,infer_bucket_sort_map_operators.q,spark_dynamic_partition_pruning_2.q,vector_inner_join.q,spark_multi_insert_parallel_orderby.q,remote_script.q] TestMiniSparkOnYarnCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=192) [scriptfile1.q,vector_outer_join5.q,file_with_header_footer.q,input16_cc.q,bucket5.q,orc_merge2.q,reduce_deduplicate.q,schemeAuthority2.q,spark_dynamic_partition_pruning_5.q,orc_merge8.q,orc_merge_incompat2.q,infer_bucket_sort_bucketed_table.q,vector_outer_join4.q,disable_merge_for_bucketing.q,orc_merge7.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=110) [bucketmapjoin4.q,bucket_map_join_spark4.q,union21.q,groupby2_noskew.q,timestamp_2.q,date_join1.q,mergejoins.q,smb_mapjoin_11.q,auto_sortmerge_join_3.q,mapjoin_test_outer.q,vectorization_9.q,merge2.q,groupby6_noskew.q,auto_join_without_localtask.q,multi_join_union.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=111) [join_cond_pushdown_unqual4.q,union_remove_7.q,join13.q,join_vc.q,groupby_cube1.q,parquet_vectorization_2.q,bucket_map_join_spark2.q,sample3.q,smb_mapjoin_19.q,union23.q,union.q,union31.q,cbo_udf_udaf.q,ptf_decimal.q,bucketmapjoin2.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=112) [parallel_join1.q,union27.q,union12.q,groupby7_map_multi_single_reducer.q,varchar_join1.q,join7.q,join_reorder4.q,skewjoinopt2.q,bucketsortoptimize_insert_2.q,smb_mapjoin_17.q,script_env_var1.q,groupby7_map.q,bucketsortoptimize_insert_8.q,stats16.q,union20.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=114) [groupby_map_ppr.q,nullgroup4_multi_distinct.q,join_rc.q,union14.q,order2.q,smb_mapjoin_12.q,vector_cast_constant.q,union_remove_4.q,parquet_vectorization_1.q,auto_join11.q,udaf_collect_set.q,vectorization_12.q,groupby_sort_skew_1_23.q,smb_mapjoin_25.q,skewjoinopt12.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=115) [skewjoinopt15.q,auto_join18.q,list_bucket_dml_2.q,input1_limit.q,load_dyn_part3.q,union_remove_14.q,auto_sortmerge_join_14.q,auto_sortmerge_join_15.q,union10.q,bucket_map_join_tez2.q,groupby5_map_skew.q,load_dyn_part7.q,join_reorder.q,bucketmapjoin8.q,union34.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=116) [avro_joins.q,parquet_vectorization_8.q,auto_join14.q,vectorization_14.q,auto_join26.q,stats1.q,cbo_stats.q,union22.q,union_view.q,subquery_views.q,smb_mapjoin_22.q,stats15.q,ptf_matchpath.q,transform_ppr1.q,sample1.q] TestSparkCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=117
[jira] [Commented] (HIVE-20486) Kafka: Use Row SerDe + vectorization
[ https://issues.apache.org/jira/browse/HIVE-20486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1787#comment-1787 ] Hive QA commented on HIVE-20486: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 44s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 2s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 43s{color} | {color:blue} itests/util in master has 48 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 41s{color} | {color:blue} ql in master has 2317 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s{color} | {color:red} kafka-handler: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 29m 51s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-14677/dev-support/hive-personality.sh | | git revision | master / 54bba9c | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-14677/yetus/diff-checkstyle-kafka-handler.txt | | modules | C: itests/util kafka-handler ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-14677/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Kafka: Use Row SerDe + vectorization > > > Key: HIVE-20486 > URL: https://issues.apache.org/jira/browse/HIVE-20486 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Reporter: Gopal V >Assignee: slim bouguerra >Priority: Major > Labels: kafka, vectorization > Fix For: 4.0.0 > > Attachments: HIVE-20486.3.patch, HIVE-20486.3.patch, > HIVE-20486.4.patch, HIVE-20486.4.patch, HIVE-20486.patch > > > KafkaHandler returns unvectorized rows which causes the operators downstream > to be slower and sub-optimal. > Hive has a vectorization shim which allows Kafka streams without complex > projections to be wrapped into a vectorized reader via > {{hive.vectorized.use.row.serde.deserialize}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20815) JdbcRecordReader.next shall not eat exception
[ https://issues.apache.org/jira/browse/HIVE-20815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-20815: -- Attachment: HIVE-20815.2.patch > JdbcRecordReader.next shall not eat exception > - > > Key: HIVE-20815 > URL: https://issues.apache.org/jira/browse/HIVE-20815 > Project: Hive > Issue Type: Bug > Components: StorageHandler >Reporter: Daniel Dai >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-20815.1.patch, HIVE-20815.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)