[jira] [Commented] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755782#comment-16755782 ] Prasanth Jayachandran commented on HIVE-21177: -- Any reason why the #readOps jumped for test case testACIDReaderFooterSerializeWithDeltas (for 2 of the asserts)? Would be good to list those 2 new calls in the comment for reference. > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch, HIVE-21177.02.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755783#comment-16755783 ] Hive QA commented on HIVE-685: -- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956844/HIVE.685.PATCH {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15836/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15836/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15836/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-01-30 07:30:19.096 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-15836/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-01-30 07:30:19.100 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at dfc4b8e HIVE-21171: Skip creating scratch dirs for tez if RPC is on (Vineet Garg, reviewed by Ashutosh Chauhan) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at dfc4b8e HIVE-21171: Skip creating scratch dirs for tez if RPC is on (Vineet Garg, reviewed by Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-01-30 07:30:19.707 + rm -rf ../yetus_PreCommit-HIVE-Build-15836 + mkdir ../yetus_PreCommit-HIVE-Build-15836 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-15836 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15836/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: git apply -p0 + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc6738574775994778247.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc6738574775994778247.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] ANTLR Parser Generator Version 3.5.2 protoc-jar: executing: [/tmp/protoc7897253848702902013.exe, --version] libprotoc 2.5.0 ANTLR Parser Generator Version 3.5.2 Output file /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java does not exist: must build /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g org/apache/hadoop/hive/metastore/parser/Filter.g log4j:WARN No appenders could be found for logger (DataNucleus.Persistence). log4j:WARN Please initialize the log4j system properly. DataNucleus Enhancer (version 4.1.17) for API "JDO" DataNucleus Enhancer completed with success for 41 classes. ANTLR Parser Generator Version 3.5.2 Output file /data/hiveptest/working/apache-github-source-source/ql/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java does not exist: must build /data/hiveptest/working/apache-github-source-source/ql/src/java/org/apache/hadoop/hive/ql/parse
[jira] [Commented] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755778#comment-16755778 ] Hive QA commented on HIVE-21177: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956826/HIVE-21177.02.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15720 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefault (batchId=230) org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfDefaultEmptyString (batchId=230) org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfOverridden (batchId=230) org.apache.hadoop.hive.metastore.TestHiveMetaStoreGetMetaConf.testGetMetaConfUnknownPreperty (batchId=230) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15835/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15835/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15835/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12956826 - PreCommit-HIVE-Build > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch, HIVE-21177.02.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755740#comment-16755740 ] Hive QA commented on HIVE-21177: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 41s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 39s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 6 new + 518 unchanged - 5 fixed = 524 total (was 523) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 1s{color} | {color:red} ql generated 1 new + 2303 unchanged - 1 fixed = 2304 total (was 2304) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 36s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight defines compareTo(AcidUtils$ParsedDeltaLight) and uses Object.equals() At AcidUtils.java:Object.equals() At AcidUtils.java:[lines 915-943] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15835/dev-support/hive-personality.sh | | git revision | master / dfc4b8e | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15835/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-15835/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15835/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch, HIVE-21177.02.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20925) test
[ https://issues.apache.org/jira/browse/HIVE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755728#comment-16755728 ] Mani M commented on HIVE-20925: --- Due to lack of information, closing this ticket > test > > > Key: HIVE-20925 > URL: https://issues.apache.org/jira/browse/HIVE-20925 > Project: Hive > Issue Type: Bug >Reporter: Sudarshan Bhide >Priority: Minor > Labels: Gamma > > test -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21184: --- Attachment: HIVE-21184.03.patch > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21184.01.patch, HIVE-21184.03.patch > > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-20925) test
[ https://issues.apache.org/jira/browse/HIVE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M reassigned HIVE-20925: - Assignee: Mani M > test > > > Key: HIVE-20925 > URL: https://issues.apache.org/jira/browse/HIVE-20925 > Project: Hive > Issue Type: Bug >Reporter: Sudarshan Bhide >Assignee: Mani M >Priority: Minor > Labels: Gamma > > test -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20849) Review of ConstantPropagateProcFactory
[ https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755722#comment-16755722 ] Hive QA commented on HIVE-20849: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956821/HIVE-20849.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 15720 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.addNoSuchTable[Remote] (batchId=220) org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.createGetDrop[Remote] (batchId=220) org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.createTableWithConstraintsPkInOtherCatalog[Remote] (batchId=220) org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.createTableWithConstraintsPk[Remote] (batchId=220) org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.doubleAddUniqueConstraint[Remote] (batchId=220) org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.getNoSuchCatalog[Remote] (batchId=220) org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.getNoSuchDb[Remote] (batchId=220) org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.getNoSuchTable[Remote] (batchId=220) org.apache.hadoop.hive.metastore.client.TestUniqueConstraint.inOtherCatalog[Remote] (batchId=220) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15833/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15833/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15833/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 9 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12956821 - PreCommit-HIVE-Build > Review of ConstantPropagateProcFactory > -- > > Key: HIVE-20849 > URL: https://issues.apache.org/jira/browse/HIVE-20849 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.1.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch, > HIVE-20849.2.patch, HIVE-20849.3.patch > > > I was looking at this class because it blasts a lot of useless (to an admin) > information to the logs. Especially if the table has a lot of columns, I see > big blocks of logging that are meaningless to me. I request that the logging > is toned down to debug, and some other improvements to the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HIVE-20925) test
[ https://issues.apache.org/jira/browse/HIVE-20925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M resolved HIVE-20925. --- Resolution: Invalid > test > > > Key: HIVE-20925 > URL: https://issues.apache.org/jira/browse/HIVE-20925 > Project: Hive > Issue Type: Bug >Reporter: Sudarshan Bhide >Priority: Minor > Labels: Gamma > > test -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21184: --- Attachment: (was: HIVE-21184.02.patch) > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21184.01.patch, HIVE-21184.03.patch > > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21184: --- Attachment: HIVE-21184.02.patch > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21184.01.patch, HIVE-21184.02.patch > > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755724#comment-16755724 ] Hive QA commented on HIVE-21184: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956823/HIVE-21184.01.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15834/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15834/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15834/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-01-30 06:10:21.914 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-15834/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-01-30 06:10:21.917 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at dfc4b8e HIVE-21171: Skip creating scratch dirs for tez if RPC is on (Vineet Garg, reviewed by Ashutosh Chauhan) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at dfc4b8e HIVE-21171: Skip creating scratch dirs for tez if RPC is on (Vineet Garg, reviewed by Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-01-30 06:10:22.542 + rm -rf ../yetus_PreCommit-HIVE-Build-15834 + mkdir ../yetus_PreCommit-HIVE-Build-15834 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-15834 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15834/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: a/ql/src/java/org/apache/hadoop/hive/ql/Driver.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/hooks/ATSHook.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/hooks/HiveProtoLoggingHook.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelOptUtil.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java: does not exist in index error: a/ql/src/test/org/apache/hadoop/hive/ql/parse/TestUpdateDeleteSemanticAnalyzer.java: does not exist in index Going to apply patch with: git apply -p1 + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc4366829490944199103.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc4366829490944199103.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] ANTLR Parser Generator Version 3.5.2 protoc-jar: executing: [/tmp/protoc9064543571866509488.exe, --version] libprotoc 2.5.0 ANTLR Parser Generator Version 3.5.2 Output file /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parse
[jira] [Commented] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755720#comment-16755720 ] Gopal V commented on HIVE-21183: LGTM - +1 tests pending. > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Attachment: HIVE.685.PATCH > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Release Note: Added GenericUDFQuote as function Target Version/s: 4.0.0 Status: Patch Available (was: In Progress) Added the source > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > Fix For: 4.0.0 > > Attachments: HIVE.685.PATCH > > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-10773) MapJoinOperator times out on loading HashTable
[ https://issues.apache.org/jira/browse/HIVE-10773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755701#comment-16755701 ] Zhihua Deng commented on HIVE-10773: Another case is that if the mapjoin key has double type, a high map collision ratio will be seen when putting these into HashMapWrapper. this was fixed by HIVE-12354 > MapJoinOperator times out on loading HashTable > -- > > Key: HIVE-10773 > URL: https://issues.apache.org/jira/browse/HIVE-10773 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.14.0 >Reporter: frank luo >Priority: Major > > When running a map join, depends on data, it might timeout with last two > lines in the log as below. And when I do "set > mapreduce.task.timeout=60;", which is defaulted to 30, the query can > go through fine. The size of hashtable file is roughly 400M. > 2015-05-20 13:27:03,237 INFO [main] > org.apache.hadoop.hive.ql.exec.MapJoinOperator: *** Load from HashTable > for input file: > hdfs://nameservice1/tmp/hive/jluo/2ee8914d-1cef-4af4-aac6-51f64d630346/hive_2015-05-20_13-13-35_335_1565066409090716856-1/-mr-10007/00_0 > 2015-05-20 13:27:03,237 INFO [main] > org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load back 1 hashtable file > from tmp file > uri:file:/data/12/hadoop/yarn/local/usercache/xxy/appcache/application_1430337284339_2087 > /container_1430337284339_2087_01_03/Stage-3.tar.gz/MapJoin-mapfile31--.hashtable > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20849) Review of ConstantPropagateProcFactory
[ https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755698#comment-16755698 ] Hive QA commented on HIVE-20849: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 34s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 45s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 36s{color} | {color:red} ql: The patch generated 3 new + 91 unchanged - 3 fixed = 94 total (was 94) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 48s{color} | {color:green} ql generated 0 new + 2301 unchanged - 3 fixed = 2301 total (was 2304) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 12s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15833/dev-support/hive-personality.sh | | git revision | master / dfc4b8e | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15833/yetus/diff-checkstyle-ql.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15833/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Review of ConstantPropagateProcFactory > -- > > Key: HIVE-20849 > URL: https://issues.apache.org/jira/browse/HIVE-20849 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.1.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch, > HIVE-20849.2.patch, HIVE-20849.3.patch > > > I was looking at this class because it blasts a lot of useless (to an admin) > information to the logs. Especially if the table has a lot of columns, I see > big blocks of logging that are meaningless to me. I request that the logging > is toned down to debug, and some other improvements to the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755675#comment-16755675 ] Hive QA commented on HIVE-20255: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956816/HIVE-20255.16.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 15718 tests executed *Failed tests:* {noformat} TestReplicationScenariosIncrementalLoadAcidTables - did not produce a TEST-*.xml file (likely timed out) (batchId=251) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions (batchId=264) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=264) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomNonExistent (batchId=264) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighBytesRead (batchId=264) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=264) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryElapsedTime (batchId=264) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerSlowQueryExecutionTime (batchId=264) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15832/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15832/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15832/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 8 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12956816 - PreCommit-HIVE-Build > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.16.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755655#comment-16755655 ] Hive QA commented on HIVE-20255: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 36s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 37s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 2 fixed = 1 total (was 3) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 21m 50s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15832/dev-support/hive-personality.sh | | git revision | master / dfc4b8e | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15832/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.16.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755638#comment-16755638 ] Hive QA commented on HIVE-20484: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956817/HIVE-20484.4.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15720 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15831/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15831/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15831/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12956817 - PreCommit-HIVE-Build > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch, HIVE-20484.4.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >* This is true by default. When true, default settings of the table and >* family are used (this will never override caching blocks if the block >* cache is disabled for that family or entirely). >* >* @param cacheBlocks if false, default settings are overridden and blocks >* will not be cached >*/ > public Scan setCacheBlocks(boolean cacheBlocks) { > this.cacheBlocks = cacheBlocks; > return this; > } > {code} > Hive is doing full scans of the table with MapReduce/Spark and therefore, > according to the HBase docs, the default behavior here should be that blocks > are not cached. Hive should set this value to "false" by default unless the > table {{SERDEPROPERTIES}} override this. > {code:sql} > -- Commands for HBase > -- create 'test', 't' > CREATE EXTERNAL TABLE test(value map, row_key string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "t:,:key", > "hbase.scan.cacheblock" = "false" > ); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755600#comment-16755600 ] Eugene Koifman commented on HIVE-21177: --- fixed tests - all were test issues all TestTriggersTezSessionPoolManager pass locally - not sure what the issue is - there is some Infra issue where https://builds.apache.org/job/PreCommit-HIVE-Build/15828/testReport is blank [~prasanth_j]/[~gopalv] could you review please > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch, HIVE-21177.02.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-21177: -- Attachment: HIVE-21177.02.patch > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch, HIVE-21177.02.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755591#comment-16755591 ] Hive QA commented on HIVE-20484: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 10s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15831/dev-support/hive-personality.sh | | git revision | master / dfc4b8e | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: hbase-handler U: hbase-handler | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15831/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch, HIVE-20484.4.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for th
[jira] [Updated] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21184: --- Attachment: (was: HIVE-21184.patch) > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21184.01.patch > > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755579#comment-16755579 ] Hive QA commented on HIVE-21183: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956807/HIVE-21183.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15830/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15830/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15830/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-01-30 02:14:24.914 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-15830/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-01-30 02:14:24.918 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at dfc4b8e HIVE-21171: Skip creating scratch dirs for tez if RPC is on (Vineet Garg, reviewed by Ashutosh Chauhan) + git clean -f -d Removing ${project.basedir}/ Removing itests/${project.basedir}/ Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at dfc4b8e HIVE-21171: Skip creating scratch dirs for tez if RPC is on (Vineet Garg, reviewed by Ashutosh Chauhan) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-01-30 02:14:26.070 + rm -rf ../yetus_PreCommit-HIVE-Build-15830 + mkdir ../yetus_PreCommit-HIVE-Build-15830 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-15830 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-15830/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch fatal: corrupt patch at line 38 fatal: corrupt patch at line 38 fatal: corrupt patch at line 38 The patch does not appear to apply with p0, p1, or p2 + result=1 + '[' 1 -ne 0 ']' + rm -rf yetus_PreCommit-HIVE-Build-15830 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12956807 - PreCommit-HIVE-Build > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21184: --- Status: Patch Available (was: In Progress) > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21184.patch > > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21184: --- Attachment: HIVE-21184.01.patch > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21184.01.patch > > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21184 started by Jesus Camacho Rodriguez. -- > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21184.patch > > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-21184: --- Attachment: HIVE-21184.patch > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-21184.patch > > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17503) CBO: Add "Explain CBO" to print Calcite trees
[ https://issues.apache.org/jira/browse/HIVE-17503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-17503: --- Labels: todoc4.0 (was: ) > CBO: Add "Explain CBO" to print Calcite trees > - > > Key: HIVE-17503 > URL: https://issues.apache.org/jira/browse/HIVE-17503 > Project: Hive > Issue Type: Improvement > Components: CBO >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > > The calcite tree is only logged at debug level in Hive right now, which is > inconvenient to debug the CBO issues with selectivity and join rotations. > The Calcite plans, before being sent to the rest of the optimizers end up > looking like > {code} > HiveProject(s_store_name=[$0], s_company_id=[$1], s_street_number=[$2], > s_street_name=[$3], s_street_type=[$4], s_suite_number=[$5], s_city=[$6], > s_county=[$7], s_state=[$8], s_zip=[$9], 30days=[$10], 3160days=[$11], > 6190days=[$12], 91120days=[$13], 120days=[$14]) > HiveAggregate(group=[{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}], agg#0=[sum($10)], > agg#1=[sum($11)], agg#2=[sum($12)], agg#3=[sum($13)], agg#4=[sum($14)]) > HiveProject($f0=[$14], $f1=[$15], $f2=[$16], $f3=[$17], $f4=[$18], > $f5=[$19], $f6=[$20], $f7=[$21], $f8=[$22], $f9=[$23], $f10=[CASE(<=(-($8, > $4), CAST(30):BIGINT), 1, 0)], $f11=[CASE(AND(>(-($8, $4), CAST(30):BIGINT), > <=(-($8, $4), CAST(60):BIGINT)), 1, 0)], $f12=[CASE(AND(>(-($8, $4), > CAST(60):BIGINT), <=(-($8, $4), CAST(90):BIGINT)), 1, 0)], > $f13=[CASE(AND(>(-($8, $4), CAST(90):BIGINT), <=(-($8, $4), > CAST(120):BIGINT)), 1, 0)], $f14=[CASE(>(-($8, $4), CAST(120):BIGINT), 1, 0)]) > HiveJoin(condition=[=($2, $13)], joinType=[inner], algorithm=[none], > cost=[not available]) > HiveJoin(condition=[=($4, $12)], joinType=[inner], > algorithm=[none], cost=[not available]) > HiveJoin(condition=[AND(=($0, $5), =($1, $6), =($3, $7))], > joinType=[inner], algorithm=[none], cost=[not available]) > HiveProject(ss_item_sk=[$1], ss_customer_sk=[$2], > ss_store_sk=[$6], ss_ticket_number=[$8], ss_sold_date_sk=[$22]) > HiveFilter(condition=[AND(IS NOT NULL($1), IS NOT NULL($2), > IS NOT NULL($8), IS NOT NULL($6), IS NOT NULL($22))]) > > HiveTableScan(table=[[tpcds_bin_partitioned_orc_1.store_sales]], > table:alias=[store_sales]) > HiveJoin(condition=[=($3, $4)], joinType=[inner], > algorithm=[none], cost=[not available]) > HiveProject(sr_item_sk=[$1], sr_customer_sk=[$2], > sr_ticket_number=[$8], sr_returned_date_sk=[$19]) > HiveFilter(condition=[AND(IS NOT NULL($1), IS NOT NULL($2), > IS NOT NULL($8), IS NOT NULL($19))]) > > HiveTableScan(table=[[tpcds_bin_partitioned_orc_1.store_returns]], > table:alias=[store_returns]) > HiveProject(d_date_sk=[$0], d_year=[CAST(2000):INTEGER], > d_moy=[CAST(9):INTEGER]) > HiveFilter(condition=[AND(=($6, 2000), =($8, 9), IS NOT > NULL($0))]) > > HiveTableScan(table=[[tpcds_bin_partitioned_orc_1.date_dim]], > table:alias=[d2]) > HiveProject(d_date_sk=[$0]) > HiveFilter(condition=[IS NOT NULL($0)]) > > HiveTableScan(table=[[tpcds_bin_partitioned_orc_1.date_dim]], > table:alias=[d1]) > HiveProject(s_store_sk=[$0], s_store_name=[$5], s_company_id=[$16], > s_street_number=[$18], s_street_name=[$19], s_street_type=[$20], > s_suite_number=[$21], s_city=[$22], s_county=[$23], s_state=[$24], > s_zip=[$25]) > HiveFilter(condition=[IS NOT NULL($0)]) > HiveTableScan(table=[[tpcds_bin_partitioned_orc_1.store]], > table:alias=[store]) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21182) Skip setting up hive scratch dir during planning
[ https://issues.apache.org/jira/browse/HIVE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755575#comment-16755575 ] Hive QA commented on HIVE-21182: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956804/HIVE-21182.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15720 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15829/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15829/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15829/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12956804 - PreCommit-HIVE-Build > Skip setting up hive scratch dir during planning > > > Key: HIVE-21182 > URL: https://issues.apache.org/jira/browse/HIVE-21182 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21182.1.patch > > > During metadata gathering phase hive creates staging/scratch dir which is > further used by FS op (FS op sets up staging dir within this dir for tasks to > write to). > Since FS op do mkdirs to setup staging dir we can skip creating scratch dir > during metadata gathering phase. FS op will take care of setting up all the > dirs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21184) Add Calcite plan to QueryPlan object
[ https://issues.apache.org/jira/browse/HIVE-21184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-21184: -- > Add Calcite plan to QueryPlan object > > > Key: HIVE-21184 > URL: https://issues.apache.org/jira/browse/HIVE-21184 > Project: Hive > Issue Type: Improvement > Components: CBO >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > > Plan is more readable than full DAG. Explain formatted/extended will print > the plan. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20849) Review of ConstantPropagateProcFactory
[ https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20849: --- Attachment: HIVE-20849.3.patch > Review of ConstantPropagateProcFactory > -- > > Key: HIVE-20849 > URL: https://issues.apache.org/jira/browse/HIVE-20849 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.1.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch, > HIVE-20849.2.patch, HIVE-20849.3.patch > > > I was looking at this class because it blasts a lot of useless (to an admin) > information to the logs. Especially if the table has a lot of columns, I see > big blocks of logging that are meaningless to me. I request that the logging > is toned down to debug, and some other improvements to the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20849) Review of ConstantPropagateProcFactory
[ https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20849: --- Status: Patch Available (was: Open) > Review of ConstantPropagateProcFactory > -- > > Key: HIVE-20849 > URL: https://issues.apache.org/jira/browse/HIVE-20849 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.1.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch, > HIVE-20849.2.patch, HIVE-20849.3.patch > > > I was looking at this class because it blasts a lot of useless (to an admin) > information to the logs. Especially if the table has a lot of columns, I see > big blocks of logging that are meaningless to me. I request that the logging > is toned down to debug, and some other improvements to the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20849) Review of ConstantPropagateProcFactory
[ https://issues.apache.org/jira/browse/HIVE-20849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20849: --- Status: Open (was: Patch Available) > Review of ConstantPropagateProcFactory > -- > > Key: HIVE-20849 > URL: https://issues.apache.org/jira/browse/HIVE-20849 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Affects Versions: 3.1.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20849.1.patch, HIVE-20849.1.patch, > HIVE-20849.2.patch > > > I was looking at this class because it blasts a lot of useless (to an admin) > information to the logs. Especially if the table has a lot of columns, I see > big blocks of logging that are meaningless to me. I request that the logging > is toned down to debug, and some other improvements to the code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21182) Skip setting up hive scratch dir during planning
[ https://issues.apache.org/jira/browse/HIVE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755549#comment-16755549 ] Hive QA commented on HIVE-21182: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 45s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 15s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15829/dev-support/hive-personality.sh | | git revision | master / dfc4b8e | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15829/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Skip setting up hive scratch dir during planning > > > Key: HIVE-21182 > URL: https://issues.apache.org/jira/browse/HIVE-21182 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21182.1.patch > > > During metadata gathering phase hive creates staging/scratch dir which is > further used by FS op (FS op sets up staging dir within this dir for tasks to > write to). > Since FS op do mkdirs to setup staging dir we can skip creating scratch dir > during metadata gathering phase. FS op will take care of setting up all the > dirs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20484: --- Attachment: HIVE-20484.4.patch > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch, HIVE-20484.4.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >* This is true by default. When true, default settings of the table and >* family are used (this will never override caching blocks if the block >* cache is disabled for that family or entirely). >* >* @param cacheBlocks if false, default settings are overridden and blocks >* will not be cached >*/ > public Scan setCacheBlocks(boolean cacheBlocks) { > this.cacheBlocks = cacheBlocks; > return this; > } > {code} > Hive is doing full scans of the table with MapReduce/Spark and therefore, > according to the HBase docs, the default behavior here should be that blocks > are not cached. Hive should set this value to "false" by default unless the > table {{SERDEPROPERTIES}} override this. > {code:sql} > -- Commands for HBase > -- create 'test', 't' > CREATE EXTERNAL TABLE test(value map, row_key string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "t:,:key", > "hbase.scan.cacheblock" = "false" > ); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20255: --- Attachment: HIVE-20255.16.patch > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.16.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-21164: Summary: ACID: explore how we can avoid a move step during inserts/compaction (was: ACID: explore how we can avoid a move step during compaction) > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20484: --- Status: Open (was: Patch Available) > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch, HIVE-20484.4.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >* This is true by default. When true, default settings of the table and >* family are used (this will never override caching blocks if the block >* cache is disabled for that family or entirely). >* >* @param cacheBlocks if false, default settings are overridden and blocks >* will not be cached >*/ > public Scan setCacheBlocks(boolean cacheBlocks) { > this.cacheBlocks = cacheBlocks; > return this; > } > {code} > Hive is doing full scans of the table with MapReduce/Spark and therefore, > according to the HBase docs, the default behavior here should be that blocks > are not cached. Hive should set this value to "false" by default unless the > table {{SERDEPROPERTIES}} override this. > {code:sql} > -- Commands for HBase > -- create 'test', 't' > CREATE EXTERNAL TABLE test(value map, row_key string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "t:,:key", > "hbase.scan.cacheblock" = "false" > ); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20484: --- Status: Patch Available (was: Open) > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch, HIVE-20484.4.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >* This is true by default. When true, default settings of the table and >* family are used (this will never override caching blocks if the block >* cache is disabled for that family or entirely). >* >* @param cacheBlocks if false, default settings are overridden and blocks >* will not be cached >*/ > public Scan setCacheBlocks(boolean cacheBlocks) { > this.cacheBlocks = cacheBlocks; > return this; > } > {code} > Hive is doing full scans of the table with MapReduce/Spark and therefore, > according to the HBase docs, the default behavior here should be that blocks > are not cached. Hive should set this value to "false" by default unless the > table {{SERDEPROPERTIES}} override this. > {code:sql} > -- Commands for HBase > -- create 'test', 't' > CREATE EXTERNAL TABLE test(value map, row_key string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "t:,:key", > "hbase.scan.cacheblock" = "false" > ); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20255: --- Status: Patch Available (was: Open) > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.16.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20255: --- Status: Open (was: Patch Available) > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21164) ACID: explore how we can avoid a move step during compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755532#comment-16755532 ] Vaibhav Gumashta commented on HIVE-21164: - Looks like this can benefit general cloud ingest use cases and not just compaction. > ACID: explore how we can avoid a move step during compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755528#comment-16755528 ] Hive QA commented on HIVE-21177: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956802/HIVE-21177.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 15720 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited] (batchId=275) org.apache.hadoop.hive.ql.io.TestAcidUtils.testParsing (batchId=297) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas (batchId=300) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas (batchId=300) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps (batchId=300) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache (batchId=300) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeFileLen (batchId=300) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeModificationTime (batchId=300) org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testGetLogicalLength (batchId=301) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitions (batchId=264) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerCustomCreatedDynamicPartitionsUnionAll (batchId=264) org.apache.hive.jdbc.TestTriggersTezSessionPoolManager.testTriggerHighShuffleBytes (batchId=264) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15828/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15828/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15828/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 12 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12956802 - PreCommit-HIVE-Build > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755496#comment-16755496 ] Hive QA commented on HIVE-21177: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 42s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 5 new + 174 unchanged - 4 fixed = 179 total (was 178) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 57s{color} | {color:red} ql generated 1 new + 2303 unchanged - 1 fixed = 2304 total (was 2304) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 41s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | org.apache.hadoop.hive.ql.io.AcidUtils$ParsedDeltaLight defines compareTo(AcidUtils$ParsedDeltaLight) and uses Object.equals() At AcidUtils.java:Object.equals() At AcidUtils.java:[lines 915-943] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15828/dev-support/hive-personality.sh | | git revision | master / dfc4b8e | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15828/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-15828/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15828/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Status: Patch Available (was: Open) > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese updated HIVE-21183: - Attachment: HIVE-21183.patch > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > Attachments: HIVE-21183.patch > > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21183) Interrupt wait time for FileCacheCleanupThread
[ https://issues.apache.org/jira/browse/HIVE-21183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oliver Draese reassigned HIVE-21183: > Interrupt wait time for FileCacheCleanupThread > -- > > Key: HIVE-21183 > URL: https://issues.apache.org/jira/browse/HIVE-21183 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Oliver Draese >Assignee: Oliver Draese >Priority: Minor > > The FileCacheCleanupThread is waiting unnecessarily long for eviction counts > to increment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755453#comment-16755453 ] Eugene Koifman edited comment on HIVE-21177 at 1/29/19 10:54 PM: - I added checks so that we don't look for the side file if we don't have to. We have another issue. Operations like Load Data/Add Partition, create base/delta and place 'raw' (aka 'original' schema) files there. Split gen and read path need to know what schema to expect in a given file/split. There is nothing in the file path that indicates what it is so it opens one of the data files in base/delta to determine that: {{AcidUtils.isRawFormat()}}. This should be less of an issue, since it does a listing first to choose the file, so it should never be looking for a file that is not actually there. I optimized isRawFormat() some but it will do the checks a lot of the time. It could be changed to rely on the file name instead but that's rather fragile. was (Author: ekoifman): I added checks so that we don't look for the side file if we don't have to. We have another issue. Operations like Load Data/Add Partition, create base/delta and place 'raw' (aka 'original' schema) files there. Split gen and read path need to know what schema to expect in a given file/split. There is nothing in the file path that indicates what it is so it opens one of the data files in base/delta to determine that: {{AcidUtils.isRawFormat()}}. This should be less of an issue, since it does a listing first to choose the file, so it should never be looking for a file that is not actually there. I optimized isRawFormat() some but it will do the checks a lot of the time. It could be changed to rely of file name instead but that's rather fragile. > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21182) Skip setting up hive scratch dir during planning
[ https://issues.apache.org/jira/browse/HIVE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21182: --- Status: Patch Available (was: Open) > Skip setting up hive scratch dir during planning > > > Key: HIVE-21182 > URL: https://issues.apache.org/jira/browse/HIVE-21182 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21182.1.patch > > > During metadata gathering phase hive creates staging/scratch dir which is > further used by FS op (FS op sets up staging dir within this dir for tasks to > write to). > Since FS op do mkdirs to setup staging dir we can skip creating scratch dir > during metadata gathering phase. FS op will take care of setting up all the > dirs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-21177: -- Status: Patch Available (was: Open) I added checks so that we don't look for the side file if we don't have to. We have another issue. Operations like Load Data/Add Partition, create base/delta and place 'raw' (aka 'original' schema) files there. Split gen and read path need to know what schema to expect in a given file/split. There is nothing in the file path that indicates what it is so it opens one of the data files in base/delta to determine that: {{AcidUtils.isRawFormat()}}. This should be less of an issue, since it does a listing first to choose the file, so it should never be looking for a file that is not actually there. I optimized isRawFormat() some but it will do the checks a lot of the time. It could be changed to rely of file name instead but that's rather fragile. > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21182) Skip setting up hive scratch dir during planning
[ https://issues.apache.org/jira/browse/HIVE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21182: --- Attachment: HIVE-21182.1.patch > Skip setting up hive scratch dir during planning > > > Key: HIVE-21182 > URL: https://issues.apache.org/jira/browse/HIVE-21182 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-21182.1.patch > > > During metadata gathering phase hive creates staging/scratch dir which is > further used by FS op (FS op sets up staging dir within this dir for tasks to > write to). > Since FS op do mkdirs to setup staging dir we can skip creating scratch dir > during metadata gathering phase. FS op will take care of setting up all the > dirs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21182) Skip setting up hive scratch dir during planning
[ https://issues.apache.org/jira/browse/HIVE-21182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-21182: -- > Skip setting up hive scratch dir during planning > > > Key: HIVE-21182 > URL: https://issues.apache.org/jira/browse/HIVE-21182 > Project: Hive > Issue Type: Improvement >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > > During metadata gathering phase hive creates staging/scratch dir which is > further used by FS op (FS op sets up staging dir within this dir for tasks to > write to). > Since FS op do mkdirs to setup staging dir we can skip creating scratch dir > during metadata gathering phase. FS op will take care of setting up all the > dirs. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-21177: -- Attachment: HIVE-21177.01.patch > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-21177.01.patch > > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755379#comment-16755379 ] Hive QA commented on HIVE-20484: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956765/HIVE-20484.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15679 tests executed *Failed tests:* {noformat} TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestObjectStore - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestPartitionProjectionEvaluator - did not produce a TEST-*.xml file (likely timed out) (batchId=230) TestRetriesInRetryingHMSHandler - did not produce a TEST-*.xml file (likely timed out) (batchId=230) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15827/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15827/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15827/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12956765 - PreCommit-HIVE-Build > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >* This is true by default. When true, default settings of the table and >* family are used (this will never override caching blocks if the block >* cache is disabled for that family or entirely). >* >* @param cacheBlocks if false, default settings are overridden and blocks >* will not be cached >*/ > public Scan setCacheBlocks(boolean cacheBlocks) { > this.cacheBlocks = cacheBlocks; > return this; > } > {code} > Hive is doing full scans of the table with MapReduce/Spark and therefore, > according to the HBase docs, the default behavior here should be that blocks > are not cached. Hive should set this value to "false" by default unless the > table {{SERDEPROPERTIES}} override this. > {code:sql} > -- Commands for HBase > -- create 'test', 't' > CREATE EXTERNAL TABLE test(value map, row_key string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "t:,:key", > "hbase.scan.cacheblock" = "false" > ); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755340#comment-16755340 ] Hive QA commented on HIVE-20484: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 30s{color} | {color:blue} hbase-handler in master has 15 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15827/dev-support/hive-personality.sh | | git revision | master / dfc4b8e | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: hbase-handler U: hbase-handler | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15827/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755324#comment-16755324 ] Hive QA commented on HIVE-20255: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956763/HIVE-20255.15.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 15720 tests executed *Failed tests:* {noformat} org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=260) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15826/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15826/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15826/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12956763 - PreCommit-HIVE-Build > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21181) Hive pre-upgrade tool not working with HDFS HA, tries connecting to nameservice as it was a NameNode
[ https://issues.apache.org/jira/browse/HIVE-21181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Csaba Marosi updated HIVE-21181: --- Attachment: hdfs-site.xml > Hive pre-upgrade tool not working with HDFS HA, tries connecting to > nameservice as it was a NameNode > > > Key: HIVE-21181 > URL: https://issues.apache.org/jira/browse/HIVE-21181 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 > Environment: Centos 7.4.1708 > kernel 3.10.0-693.11.6.el7.x86_64 > Ambari 2.6.2.2 > HDP-2.6.5.0-292 > Hive 1.2.1000 > HDFS 2.7.3 >Reporter: Attila Csaba Marosi >Priority: Major > Attachments: core-site.xml, hdfs-site.xml > > > While preparing a production cluster HDP-2.6.5 -> HDP-3.1 upgrades, we've > noticed issues with the hive-pre-upgrade tool, when we tried running it, we > got the exception: > {{Found Acid table: default.hello_acid > 2019-01-28 15:54:20,331 ERROR [main] acid.PreUpgradeTool > (PreUpgradeTool.java:main(152)) - PreUpgradeTool failed > java.lang.IllegalArgumentException: java.net.UnknownHostException: > mytestcluster > at > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:439) > at > org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:321) > at > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:696) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:636) > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2796) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2830) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2812) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:417) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:384) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:374) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:235) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:149) > Caused by: java.net.UnknownHostException: mytestcluster > ... 17 more}} > We tried running it on a kerberized test cluster built based on the same > blueprint like the production clusters, with HDP-2.6.5.0-292, Hive 1.2.1000, > HDFS 2.7.3, with HDFS HA and without Hive HA. > We enabled Hive ACID, created the same example ACID table as shown in > https://hortonworks.com/tutorial/using-hive-acid-transactions-to-insert-update-and-delete-data/ > We followed the steps described at > https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/prepare_hive_for_upgrade.html > , kinit-ed, used the "-Djavax.security.auth.useSubjectCredsOnly=false" > parameter. > Without the ACID table there is no issue. > I'm attaching the hdfs-site.xml and core-site.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21181) Hive pre-upgrade tool not working with HDFS HA, tries connecting to nameservice as it was a NameNode
[ https://issues.apache.org/jira/browse/HIVE-21181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Csaba Marosi updated HIVE-21181: --- Attachment: core-site.xml > Hive pre-upgrade tool not working with HDFS HA, tries connecting to > nameservice as it was a NameNode > > > Key: HIVE-21181 > URL: https://issues.apache.org/jira/browse/HIVE-21181 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.1 > Environment: Centos 7.4.1708 > kernel 3.10.0-693.11.6.el7.x86_64 > Ambari 2.6.2.2 > HDP-2.6.5.0-292 > Hive 1.2.1000 > HDFS 2.7.3 >Reporter: Attila Csaba Marosi >Priority: Major > Attachments: core-site.xml, hdfs-site.xml > > > While preparing a production cluster HDP-2.6.5 -> HDP-3.1 upgrades, we've > noticed issues with the hive-pre-upgrade tool, when we tried running it, we > got the exception: > {{Found Acid table: default.hello_acid > 2019-01-28 15:54:20,331 ERROR [main] acid.PreUpgradeTool > (PreUpgradeTool.java:main(152)) - PreUpgradeTool failed > java.lang.IllegalArgumentException: java.net.UnknownHostException: > mytestcluster > at > org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:439) > at > org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:321) > at > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:696) > at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:636) > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160) > at > org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2796) > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2830) > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2812) > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390) > at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:417) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:384) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:374) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:235) > at > org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:149) > Caused by: java.net.UnknownHostException: mytestcluster > ... 17 more}} > We tried running it on a kerberized test cluster built based on the same > blueprint like the production clusters, with HDP-2.6.5.0-292, Hive 1.2.1000, > HDFS 2.7.3, with HDFS HA and without Hive HA. > We enabled Hive ACID, created the same example ACID table as shown in > https://hortonworks.com/tutorial/using-hive-acid-transactions-to-insert-update-and-delete-data/ > We followed the steps described at > https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/prepare_hive_for_upgrade.html > , kinit-ed, used the "-Djavax.security.auth.useSubjectCredsOnly=false" > parameter. > Without the ACID table there is no issue. > I'm attaching the hdfs-site.xml and core-site.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21181) Hive pre-upgrade tool not working with HDFS HA, tries connecting to nameservice as it was a NameNode
[ https://issues.apache.org/jira/browse/HIVE-21181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Csaba Marosi updated HIVE-21181: --- Description: While preparing a production cluster HDP-2.6.5 -> HDP-3.1 upgrades, we've noticed issues with the hive-pre-upgrade tool, when we tried running it, we got the exception: {{Found Acid table: default.hello_acid 2019-01-28 15:54:20,331 ERROR [main] acid.PreUpgradeTool (PreUpgradeTool.java:main(152)) - PreUpgradeTool failed java.lang.IllegalArgumentException: java.net.UnknownHostException: mytestcluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:439) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:321) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:696) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:636) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2796) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2830) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2812) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:417) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:384) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:374) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:235) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:149) Caused by: java.net.UnknownHostException: mytestcluster ... 17 more}} We tried running it on a kerberized test cluster built based on the same blueprint like the production clusters, with HDP-2.6.5.0-292, Hive 1.2.1000, HDFS 2.7.3, with HDFS HA and without Hive HA. We enabled Hive ACID, created the same example ACID table as shown in https://hortonworks.com/tutorial/using-hive-acid-transactions-to-insert-update-and-delete-data/ We followed the steps described at https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/prepare_hive_for_upgrade.html , kinit-ed, used the "-Djavax.security.auth.useSubjectCredsOnly=false" parameter. Without the ACID table there is no issue. I'm attaching the hdfs-site.xml and core-site.xml. was: While preparing a production cluster HDP-2.6.5 -> HDP-3.1 upgrades, we've noticed issues with the hive-pre-upgrade tool, when we tried running it, we got the exception: {{Found Acid table: default.hello_acid 2019-01-28 15:54:20,331 ERROR [main] acid.PreUpgradeTool (PreUpgradeTool.java:main(152)) - PreUpgradeTool failed java.lang.IllegalArgumentException: java.net.UnknownHostException: mytestcluster at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:439) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:321) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:696) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:636) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:160) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2796) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:99) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2830) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2812) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.needsCompaction(PreUpgradeTool.java:417) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:384) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.getCompactionCommands(PreUpgradeTool.java:374) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.prepareAcidUpgradeInternal(PreUpgradeTool.java:235) at org.apache.hadoop.hive.upgrade.acid.PreUpgradeTool.main(PreUpgradeTool.java:149) Caused by: java.net.UnknownHostException: mytestcluster ... 17 more}} We tried running it on a kerberized test cluster built based on the same blueprint like the production clusters, w
[jira] [Commented] (HIVE-21077) Database and catalogs should have creation time
[ https://issues.apache.org/jira/browse/HIVE-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755300#comment-16755300 ] Vihang Karajgaonkar commented on HIVE-21077: Thanks [~ngangam] merged the addendum to master and branch-3 > Database and catalogs should have creation time > --- > > Key: HIVE-21077 > URL: https://issues.apache.org/jira/browse/HIVE-21077 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-21077.01.patch, HIVE-21077.02.patch, > HIVE-21077.03.patch, HIVE-21077.04.patch, HIVE-21077.05.patch, > HIVE-21077.06.patch, HIVE-21077.07.patch, HIVE-21077.08.branch-3.patch, > HIVE-21077.09.patch, HIVE-21077.10.patch, HIVE-21077.addendum.patch > > > Currently, database do not have creation time like we have for tables and > partitions. > {noformat} > // namespace for tables > struct Database { > 1: string name, > 2: string description, > 3: string locationUri, > 4: map parameters, // properties associated with the > database > 5: optional PrincipalPrivilegeSet privileges, > 6: optional string ownerName, > 7: optional PrincipalType ownerType, > 8: optional string catalogName > } > {noformat} > Currently, without creationTime there is no way to identify if the copy of > Database which a client has is the same as the one on the server if the name > is same. Without object ids creationTime value is the only way currently to > identify uniquely a instance of metastore object. It would be good to have > Database creation time as well. > Same applies for catalogs as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21171) Skip creating scratch dirs for tez if RPC is on
[ https://issues.apache.org/jira/browse/HIVE-21171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-21171: --- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to master. > Skip creating scratch dirs for tez if RPC is on > --- > > Key: HIVE-21171 > URL: https://issues.apache.org/jira/browse/HIVE-21171 > Project: Hive > Issue Type: Improvement > Components: Tez >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21171.1.patch, HIVE-21171.2.patch > > > There are few places e.g. during creating DAG/Vertices where scratch > directories are created for each vertex even if plan is being sent using RPC. > This adds un-necessary overhead for cloud file system e.g. S3A. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-21180) Fix branch-3 metastore test timeouts
[ https://issues.apache.org/jira/browse/HIVE-21180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-21180: -- > Fix branch-3 metastore test timeouts > > > Key: HIVE-21180 > URL: https://issues.apache.org/jira/browse/HIVE-21180 > Project: Hive > Issue Type: Test >Affects Versions: 3.2.0 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > > The module name below is wrong since metastore-server doesn't exist on > branch-3. This is most likely the reason why test batches are timing out on > branch-3 > {noformat} > 2019-01-29 00:32:17,765 INFO [HostExecutor 3] > HostExecutor.executeTestBatch:262 Drone [user=hiveptest, > host=104.198.216.224, instance=0] executing UnitTestBatch > [name=228_UTBatch_standalone-metastore__metastore-server_20_tests, id=228, > moduleName=standalone-metastore/metastore-server, batchSize=20, > isParallel=true, testList=[TestPartitionManagement, > TestCatalogNonDefaultClient, TestCatalogOldClient, TestHiveAlterHandler, > TestTxnHandlerNegative, TestTxnUtils, TestFilterHooks, TestRawStoreProxy, > TestLockRequestBuilder, TestHiveMetastoreCli, TestCheckConstraint, > TestAddPartitions, TestListPartitions, TestFunctions, TestGetTableMeta, > TestTablesCreateDropAlterTruncate, TestRuntimeStats, TestDropPartitions, > TestTablesList, TestUniqueConstraint]] with bash > /home/hiveptest/104.198.216.224-hiveptest-0/scratch/hiveptest-228_UTBatch_standalone-metastore__metastore-server_20_tests.sh > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21128) hive.version.shortname should be 3.2 on branch-3
[ https://issues.apache.org/jira/browse/HIVE-21128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755297#comment-16755297 ] Vihang Karajgaonkar commented on HIVE-21128: Looks like metastore tests on branch-3 are broken since the module name is pointing to standalone-metastore/metastore-server but metastore-server module doesn't exist on branch-3 2019-01-29 00:32:17,765 INFO [HostExecutor 3] HostExecutor.executeTestBatch:262 Drone [user=hiveptest, host=104.198.216.224, instance=0] executing UnitTestBatch [name=228_UTBatch_standalone-metastore__metastore-server_20_tests, id=228, moduleName=standalone-metastore/metastore-server, batchSize=20, isParallel=true, testList=[TestPartitionManagement, TestCatalogNonDefaultClient, TestCatalogOldClient, TestHiveAlterHandler, TestTxnHandlerNegative, TestTxnUtils, TestFilterHooks, TestRawStoreProxy, TestLockRequestBuilder, TestHiveMetastoreCli, TestCheckConstraint, TestAddPartitions, TestListPartitions, TestFunctions, TestGetTableMeta, TestTablesCreateDropAlterTruncate, TestRuntimeStats, TestDropPartitions, TestTablesList, TestUniqueConstraint]] with bash /home/hiveptest/104.198.216.224-hiveptest-0/scratch/hiveptest-228_UTBatch_standalone-metastore__metastore-server_20_tests.sh > hive.version.shortname should be 3.2 on branch-3 > > > Key: HIVE-21128 > URL: https://issues.apache.org/jira/browse/HIVE-21128 > Project: Hive > Issue Type: Bug >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Attachments: HIVE-21128.01.branch-3.patch, > HIVE-21128.02.branch-3.patch, HIVE-21128.03.branch-3.patch > > > Since 3.1.0 is already release, the {{hive.version.shortname}} property in > the pom.xml of standalone-metastore should be 3.2.0. This version shortname > is used to generate the metastore schema version and used by Schematool to > initialize the schema using the correct script. Currently it using 3.1.0 > schema init script instead of 3.2.0 init script -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755287#comment-16755287 ] Hive QA commented on HIVE-20255: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 13s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 49s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 55s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} ql: The patch generated 0 new + 1 unchanged - 2 fixed = 1 total (was 3) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 22m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15826/dev-support/hive-personality.sh | | git revision | master / 71dfd1d | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15826/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20484: --- Status: Open (was: Patch Available) > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >* This is true by default. When true, default settings of the table and >* family are used (this will never override caching blocks if the block >* cache is disabled for that family or entirely). >* >* @param cacheBlocks if false, default settings are overridden and blocks >* will not be cached >*/ > public Scan setCacheBlocks(boolean cacheBlocks) { > this.cacheBlocks = cacheBlocks; > return this; > } > {code} > Hive is doing full scans of the table with MapReduce/Spark and therefore, > according to the HBase docs, the default behavior here should be that blocks > are not cached. Hive should set this value to "false" by default unless the > table {{SERDEPROPERTIES}} override this. > {code:sql} > -- Commands for HBase > -- create 'test', 't' > CREATE EXTERNAL TABLE test(value map, row_key string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "t:,:key", > "hbase.scan.cacheblock" = "false" > ); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20484: --- Status: Patch Available (was: Open) > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >* This is true by default. When true, default settings of the table and >* family are used (this will never override caching blocks if the block >* cache is disabled for that family or entirely). >* >* @param cacheBlocks if false, default settings are overridden and blocks >* will not be cached >*/ > public Scan setCacheBlocks(boolean cacheBlocks) { > this.cacheBlocks = cacheBlocks; > return this; > } > {code} > Hive is doing full scans of the table with MapReduce/Spark and therefore, > according to the HBase docs, the default behavior here should be that blocks > are not cached. Hive should set this value to "false" by default unless the > table {{SERDEPROPERTIES}} override this. > {code:sql} > -- Commands for HBase > -- create 'test', 't' > CREATE EXTERNAL TABLE test(value map, row_key string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "t:,:key", > "hbase.scan.cacheblock" = "false" > ); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20484) Disable Block Cache By Default With HBase SerDe
[ https://issues.apache.org/jira/browse/HIVE-20484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20484: --- Attachment: HIVE-20484.3.patch > Disable Block Cache By Default With HBase SerDe > --- > > Key: HIVE-20484 > URL: https://issues.apache.org/jira/browse/HIVE-20484 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 1.2.3, 2.4.0, 4.0.0, 3.2.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Major > Attachments: HIVE-20484.1.patch, HIVE-20484.2.patch, > HIVE-20484.3.patch > > > {quote} > Scan instances can be set to use the block cache in the RegionServer via the > setCacheBlocks method. For input Scans to MapReduce jobs, this should be > false. > https://hbase.apache.org/book.html#perf.hbase.client.blockcache > {quote} > However, from the Hive code, we can see that this is not the case. > {code} > public static final String HBASE_SCAN_CACHEBLOCKS = "hbase.scan.cacheblock"; > ... > String scanCacheBlocks = > tableProperties.getProperty(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > jobProperties.put(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS, scanCacheBlocks); > } > ... > String scanCacheBlocks = jobConf.get(HBaseSerDe.HBASE_SCAN_CACHEBLOCKS); > if (scanCacheBlocks != null) { > scan.setCacheBlocks(Boolean.parseBoolean(scanCacheBlocks)); > } > {code} > In the Hive code, we can see that if {{hbase.scan.cacheblock}} is not > specified in the {{SERDEPROPERTIES}} then {{setCacheBlocks}} is not called > and the default value of the HBase {{Scan}} class is used. > {code:java|title=Scan.java} > /** >* Set whether blocks should be cached for this Scan. >* >* This is true by default. When true, default settings of the table and >* family are used (this will never override caching blocks if the block >* cache is disabled for that family or entirely). >* >* @param cacheBlocks if false, default settings are overridden and blocks >* will not be cached >*/ > public Scan setCacheBlocks(boolean cacheBlocks) { > this.cacheBlocks = cacheBlocks; > return this; > } > {code} > Hive is doing full scans of the table with MapReduce/Spark and therefore, > according to the HBase docs, the default behavior here should be that blocks > are not cached. Hive should set this value to "false" by default unless the > table {{SERDEPROPERTIES}} override this. > {code:sql} > -- Commands for HBase > -- create 'test', 't' > CREATE EXTERNAL TABLE test(value map, row_key string) > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping" = "t:,:key", > "hbase.scan.cacheblock" = "false" > ); > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20255: --- Attachment: HIVE-20255.15.patch > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20255: --- Status: Patch Available (was: Open) > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-20255) Review LevelOrderWalker.java
[ https://issues.apache.org/jira/browse/HIVE-20255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BELUGA BEHR updated HIVE-20255: --- Status: Open (was: Patch Available) > Review LevelOrderWalker.java > > > Key: HIVE-20255 > URL: https://issues.apache.org/jira/browse/HIVE-20255 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 3.0.0, 4.0.0 >Reporter: BELUGA BEHR >Assignee: BELUGA BEHR >Priority: Minor > Attachments: HIVE-20255.10.patch, HIVE-20255.11.patch, > HIVE-20255.12.patch, HIVE-20255.13.patch, HIVE-20255.14.patch, > HIVE-20255.15.patch, HIVE-20255.9.patch > > > https://github.com/apache/hive/blob/6d890faf22fd1ede3658a5eed097476eab3c67e9/ql/src/java/org/apache/hadoop/hive/ql/lib/LevelOrderWalker.java > * Make code more concise > * Fix some check style issues > {code} > if (toWalk.get(index).getChildren() != null) { > for(Node child : toWalk.get(index).getChildren()) { > {code} > Actually, the underlying implementation of {{getChildren()}} has to do some > real work, so do not throw away the work after checking for null. Simply > call once and store the results. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21145) Enable cbo to use runtime statistics during re-optimization
[ https://issues.apache.org/jira/browse/HIVE-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755237#comment-16755237 ] Hive QA commented on HIVE-21145: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956701/HIVE-21145.01.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1047 failed/errored test(s), 15715 tests executed *Failed tests:* {noformat} TestJdbcWithMiniLlapArrow - did not produce a TEST-*.xml file (likely timed out) (batchId=261) TestServiceDiscovery - did not produce a TEST-*.xml file (likely timed out) (batchId=261) org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_single_sourced_multi_insert] (batchId=267) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[mapjoin2] (batchId=275) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[materialized_view_create_rewrite] (batchId=275) org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_10] (batchId=275) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_subquery] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf] (batchId=57) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join] (batchId=58) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join_pkfk] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_1] (batchId=24) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_2] (batchId=91) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_8] (batchId=15) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_9] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join0] (batchId=95) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join10] (batchId=38) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join11] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join12] (batchId=26) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=87) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join15] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join16] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join17] (batchId=89) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join1] (batchId=84) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join20] (batchId=96) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join21] (batchId=88) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join22] (batchId=61) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join24] (batchId=81) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join27] (batchId=97) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join28] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join29] (batchId=59) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join2] (batchId=69) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join33] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join3] (batchId=88) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join4] (batchId=77) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join5] (batchId=79) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join6] (batchId=93) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join7] (batchId=28) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join8] (batchId=94) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_reordering_values] (batchId=6) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats2] (batchId=94) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_stats] (batchId=52) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join_without_localtask] (batchId=1) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketmapjoin_negative3] (batchId=30) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_input26] (batchId=2) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join0] (batchId=16) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join17] (batchId=27) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_auto_join1] (batchId=4) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_cross_product_check_2] (batchId=22) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=40) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[complex_alias] (batchId=18) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_1] (batchId=43) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriv
[jira] [Assigned] (HIVE-21177) Optimize AcidUtils.getLogicalLength()
[ https://issues.apache.org/jira/browse/HIVE-21177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-21177: - > Optimize AcidUtils.getLogicalLength() > - > > Key: HIVE-21177 > URL: https://issues.apache.org/jira/browse/HIVE-21177 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > {{AcidUtils.getLogicalLength()}} - tries look for the side file > {{OrcAcidUtils.getSideFile()}} on the file system even when the file couldn't > possibly be there, e.g. when the path is delta_x_x or base_x. It could only > be there in delta_x_y, x != y. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21077) Database and catalogs should have creation time
[ https://issues.apache.org/jira/browse/HIVE-21077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755230#comment-16755230 ] Naveen Gangam commented on HIVE-21077: -- [~vihangk1] The add-on patch for the MSSQL datatype looks good to me. So +1 for me. Thanks > Database and catalogs should have creation time > --- > > Key: HIVE-21077 > URL: https://issues.apache.org/jira/browse/HIVE-21077 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar >Priority: Major > Fix For: 4.0.0, 3.2.0 > > Attachments: HIVE-21077.01.patch, HIVE-21077.02.patch, > HIVE-21077.03.patch, HIVE-21077.04.patch, HIVE-21077.05.patch, > HIVE-21077.06.patch, HIVE-21077.07.patch, HIVE-21077.08.branch-3.patch, > HIVE-21077.09.patch, HIVE-21077.10.patch, HIVE-21077.addendum.patch > > > Currently, database do not have creation time like we have for tables and > partitions. > {noformat} > // namespace for tables > struct Database { > 1: string name, > 2: string description, > 3: string locationUri, > 4: map parameters, // properties associated with the > database > 5: optional PrincipalPrivilegeSet privileges, > 6: optional string ownerName, > 7: optional PrincipalType ownerType, > 8: optional string catalogName > } > {noformat} > Currently, without creationTime there is no way to identify if the copy of > Database which a client has is the same as the one on the server if the name > is same. Without object ids creationTime value is the only way currently to > identify uniquely a instance of metastore object. It would be good to have > Database creation time as well. > Same applies for catalogs as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21145) Enable cbo to use runtime statistics during re-optimization
[ https://issues.apache.org/jira/browse/HIVE-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755216#comment-16755216 ] Hive QA commented on HIVE-21145: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 18s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 26s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 48s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 43s{color} | {color:red} ql: The patch generated 11 new + 785 unchanged - 39 fixed = 796 total (was 824) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 3m 48s{color} | {color:red} ql generated 2 new + 2303 unchanged - 1 fixed = 2305 total (was 2304) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 17s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Dead store to ss1 in org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(RelNode, RexNode) At HiveRelFactories.java:org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories$HiveFilterFactoryImpl.createFilter(RelNode, RexNode) At HiveRelFactories.java:[line 141] | | | Format-string method String.format(String, Object[]) called with format string "expected(stat-sig) 1-1, got {}-{} ;" wants 0 arguments but is given 2 in org.apache.hadoop.hive.ql.plan.mapper.StatsSources.extractStatMapFromPlanMapper(PlanMapper) At StatsSources.java:with format string "expected(stat-sig) 1-1, got {}-{} ;" wants 0 arguments but is given 2 in org.apache.hadoop.hive.ql.plan.mapper.StatsSources.extractStatMapFromPlanMapper(PlanMapper) At StatsSources.java:[line 88] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15825/dev-support/hive-personality.sh | | git revision | master / 71dfd1d | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15825/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-15825/yetus/new-findbugs-ql.html | | modules | C: ql itests U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15825/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Enable cbo to use runtime statistics during re-optimization > --- > > Key: HIVE-21145 > URL: https://issues.apache.org/jira/browse/HIVE-21145
[jira] [Updated] (HIVE-21145) Enable cbo to use runtime statistics during re-optimization
[ https://issues.apache.org/jira/browse/HIVE-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-21145: Status: Patch Available (was: Open) > Enable cbo to use runtime statistics during re-optimization > --- > > Key: HIVE-21145 > URL: https://issues.apache.org/jira/browse/HIVE-21145 > Project: Hive > Issue Type: Improvement > Components: CBO, Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21145.01.patch > > > This could enable to reorder joins according to runtime rowcounts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755181#comment-16755181 ] Hive QA commented on HIVE-21029: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956724/HIVE-21029.02.patch {color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15721 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15824/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15824/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15824/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12956724 - PreCommit-HIVE-Build > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch, HIVE-21029.02.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755155#comment-16755155 ] Hive QA commented on HIVE-21029: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 5s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 37s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s{color} | {color:red} ql: The patch generated 1 new + 266 unchanged - 2 fixed = 267 total (was 268) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 21 unchanged - 0 fixed = 22 total (was 21) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 1s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15824/dev-support/hive-personality.sh | | git revision | master / 71dfd1d | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15824/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15824/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: common ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15824/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.
[jira] [Commented] (HIVE-21079) Replicate column statistics for partitions of partitioned table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755123#comment-16755123 ] Ashutosh Bapat commented on HIVE-21079: --- Thanks [~sankarh] for your review comments, help and fixes. > Replicate column statistics for partitions of partitioned table. > > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch, HIVE-21079.04.patch, HIVE-21079.05.patch, > HIVE-21079.06.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21029: Component/s: (was: HiveServer2) > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: repl >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch, HIVE-21029.02.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21029: Attachment: HIVE-21029.02.patch > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch, HIVE-21029.02.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21029: Status: Patch Available (was: Open) 02.patch fixes check style issues and rebased with master. > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.1.1, 3.1.0, 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch, HIVE-21029.02.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21079) Replicate column statistics for partitions of partitioned table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21079: Resolution: Fixed Status: Resolved (was: Patch Available) 06.patch is committed to master. Thanks [~ashutosh.bapat] for the patch! > Replicate column statistics for partitions of partitioned table. > > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch, HIVE-21079.04.patch, HIVE-21079.05.patch, > HIVE-21079.06.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21079) Replicate column statistics for partitions of partitioned table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21079: Summary: Replicate column statistics for partitions of partitioned table. (was: Replicate column statistics for partitions of partitioned Hive table.) > Replicate column statistics for partitions of partitioned table. > > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch, HIVE-21079.04.patch, HIVE-21079.05.patch, > HIVE-21079.06.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755047#comment-16755047 ] Hive QA commented on HIVE-21079: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956700/HIVE-21079.06.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15720 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15821/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15821/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15821/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12956700 - PreCommit-HIVE-Build > Replicate column statistics for partitions of partitioned Hive table. > - > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch, HIVE-21079.04.patch, HIVE-21079.05.patch, > HIVE-21079.06.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21029: Status: Open (was: Patch Available) > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.1.1, 3.1.0, 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-21153) task run in parallel cause ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HIVE-21153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21153 started by Chris Bogan. -- > task run in parallel cause ArrayIndexOutOfBoundsException > - > > Key: HIVE-21153 > URL: https://issues.apache.org/jira/browse/HIVE-21153 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: yx91490 >Assignee: Chris Bogan >Priority: Major > > It happens occasionally when we run sql in parallel and the job is > blocked.the reason is HiveConf class is not thread safe, and > SessionState.createSessionDirs() is not Synchronized. > the error message: > java.lang.ArrayIndexOutOfBoundsException: 22 > at java.util.ArrayList.add(ArrayList.java:459) > at org.apache.hadoop.hive.conf.HiveConf.setupRestrictList(HiveConf.java:3216) > at org.apache.hadoop.hive.conf.HiveConf.addToRestrictList(HiveConf.java:3191) > at > org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:651) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547) > at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:78) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work stopped] (HIVE-21153) task run in parallel cause ArrayIndexOutOfBoundsException
[ https://issues.apache.org/jira/browse/HIVE-21153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-21153 stopped by Chris Bogan. -- > task run in parallel cause ArrayIndexOutOfBoundsException > - > > Key: HIVE-21153 > URL: https://issues.apache.org/jira/browse/HIVE-21153 > Project: Hive > Issue Type: Bug > Components: Testing Infrastructure >Reporter: yx91490 >Assignee: Chris Bogan >Priority: Major > > It happens occasionally when we run sql in parallel and the job is > blocked.the reason is HiveConf class is not thread safe, and > SessionState.createSessionDirs() is not Synchronized. > the error message: > java.lang.ArrayIndexOutOfBoundsException: 22 > at java.util.ArrayList.add(ArrayList.java:459) > at org.apache.hadoop.hive.conf.HiveConf.setupRestrictList(HiveConf.java:3216) > at org.apache.hadoop.hive.conf.HiveConf.addToRestrictList(HiveConf.java:3191) > at > org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:651) > at > org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:547) > at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:78) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21029: Status: Patch Available (was: Open) > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.1.1, 3.1.0, 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21029: Attachment: HIVE-21029.01.patch > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21029: Attachment: (was: HIVE-21029.01.patch) > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21029: Status: Open (was: Patch Available) > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.1.1, 3.1.0, 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16755029#comment-16755029 ] Hive QA commented on HIVE-21079: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 26s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 18s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 10s{color} | {color:blue} standalone-metastore/metastore-common in master has 29 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 7s{color} | {color:blue} standalone-metastore/metastore-server in master has 184 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 40s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 25s{color} | {color:blue} hcatalog/server-extensions in master has 3 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 32s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 49s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 38s{color} | {color:red} ql: The patch generated 5 new + 363 unchanged - 5 fixed = 368 total (was 368) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 20s{color} | {color:red} itests/hive-unit: The patch generated 1 new + 565 unchanged - 3 fixed = 566 total (was 568) {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 28 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 34s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 43m 3s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15821/dev-support/hive-personality.sh | | git revision | master / 9747083 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15821/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15821/yetus/diff-checkstyle-itests_hive-unit.txt | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-15821/yetus/whitespace-eol.txt | | modules | C: standalone-metastore/metastore-common standalone-metastore/metastore-server ql hcatalog/server-extensions itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15821/yetus.txt | |
[jira] [Commented] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754985#comment-16754985 ] Hive QA commented on HIVE-21029: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12956696/HIVE-21029.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 15719 tests executed *Failed tests:* {noformat} TestReplicationScenariosIncrementalLoadAcidTables - did not produce a TEST-*.xml file (likely timed out) (batchId=251) org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=230) org.apache.hive.hcatalog.mapreduce.TestHCatMutableNonPartitioned.testHCatNonPartitionedTable[7] (batchId=214) org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=260) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/15820/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/15820/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-15820/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 4 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12956696 - PreCommit-HIVE-Build > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 > Project: Hive > Issue Type: Bug > Components: HiveServer2, repl >Affects Versions: 3.0.0, 3.1.0, 3.1.1 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Critical > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21029.01.patch > > > Existing deployments using hive replication do not get external tables > replicated. For such deployments to enable external table replication they > will have to provide a specific switch to first bootstrap external tables as > part of hive incremental replication, following which the incremental > replication will take care of further changes in external tables. > The switch will be provided by an additional hive configuration (for ex: > hive.repl.bootstrap.external.tables) and is to be used in > {code} WITH {code} clause of > {code} REPL DUMP {code} command. > Additionally the existing hive config _hive.repl.include.external.tables_ > will always have to be set to "true" in the above clause. > Proposed usage for enabling external tables replication on existing > replication policy. > 1. Consider an ongoing repl policy in incremental phase. > Enable hive.repl.include.external.tables=true and > hive.repl.bootstrap.external.tables=true in next incremental REPL DUMP. > - Dumps all events but skips events related to external tables. > - Instead, combine bootstrap dump for all external tables under “_bootstrap” > directory. > - Also, includes the data locations file "_external_tables_info”. > - LIMIT or TO clause shouldn’t be there to ensure the latest events are > dumped before bootstrap dumping external tables. > 2. REPL LOAD on this dump applies all the events first, copies external > tables data and then bootstrap external tables (metadata). > - It is possible that the external tables (metadata) are not point-in time > consistent with rest of the tables. > - But, it would be eventually consistent when the next incremental load is > applied. > - This REPL LOAD is fault tolerant and can be retried if failed. > 3. All future REPL DUMPs on this repl policy should set > hive.repl.bootstrap.external.tables=false. > - If not set to false, then target might end up having inconsistent set of > external tables as bootstrap wouldn’t clean-up any dropped external tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21029) External table replication for existing deployments running incremental replication.
[ https://issues.apache.org/jira/browse/HIVE-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754944#comment-16754944 ] Hive QA commented on HIVE-21029: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 19s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 50s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 31s{color} | {color:blue} common in master has 65 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 35s{color} | {color:blue} ql in master has 2304 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 52s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s{color} | {color:red} common: The patch generated 3 new + 428 unchanged - 0 fixed = 431 total (was 428) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s{color} | {color:red} ql: The patch generated 4 new + 266 unchanged - 2 fixed = 270 total (was 268) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 15 unchanged - 0 fixed = 19 total (was 15) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-15820/dev-support/hive-personality.sh | | git revision | master / 9747083 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15820/yetus/diff-checkstyle-common.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15820/yetus/diff-checkstyle-ql.txt | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-15820/yetus/diff-checkstyle-itests_hive-unit.txt | | modules | C: common ql itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-15820/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > External table replication for existing deployments running incremental > replication. > > > Key: HIVE-21029 > URL: https://issues.apache.org/jira/browse/HIVE-21029 >
[jira] [Commented] (HIVE-21145) Enable cbo to use runtime statistics during re-optimization
[ https://issues.apache.org/jira/browse/HIVE-21145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16754932#comment-16754932 ] Zoltan Haindrich commented on HIVE-21145: - #01) wip patch; see if it break anything.. > Enable cbo to use runtime statistics during re-optimization > --- > > Key: HIVE-21145 > URL: https://issues.apache.org/jira/browse/HIVE-21145 > Project: Hive > Issue Type: Improvement > Components: CBO, Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21145.01.patch > > > This could enable to reorder joins according to runtime rowcounts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-685) add UDFquote
[ https://issues.apache.org/jira/browse/HIVE-685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mani M updated HIVE-685: Labels: todoc4.0 udf (was: ) > add UDFquote > > > Key: HIVE-685 > URL: https://issues.apache.org/jira/browse/HIVE-685 > Project: Hive > Issue Type: New Feature > Components: Query Processor, UDF >Reporter: Namit Jain >Assignee: Mani M >Priority: Major > Labels: todoc4.0, udf > > add UDFquote > look at > http://dev.mysql.com/doc/refman/5.0/en/func-op-summary-ref.html > for details -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21079: -- Attachment: HIVE-21079.06.patch Status: Patch Available (was: In Progress) Attaching the same patch as .05 only renamed again to trigger ptest. The test failed in the last run is flaky and passes for me when run individually. > Replicate column statistics for partitions of partitioned Hive table. > - > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch, HIVE-21079.04.patch, HIVE-21079.05.patch, > HIVE-21079.06.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-21079) Replicate column statistics for partitions of partitioned Hive table.
[ https://issues.apache.org/jira/browse/HIVE-21079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Bapat updated HIVE-21079: -- Status: In Progress (was: Patch Available) > Replicate column statistics for partitions of partitioned Hive table. > - > > Key: HIVE-21079 > URL: https://issues.apache.org/jira/browse/HIVE-21079 > Project: Hive > Issue Type: Sub-task >Affects Versions: 4.0.0 >Reporter: Ashutosh Bapat >Assignee: Ashutosh Bapat >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21079.01.patch, HIVE-21079.02.patch, > HIVE-21079.03.patch, HIVE-21079.04.patch, HIVE-21079.05.patch > > > This task is for replicating statistics for partitions of a partitioned Hive > table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)