[jira] [Commented] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934958#comment-16934958 ] Hive QA commented on HIVE-5: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980937/HIVE-5.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16829 tests executed *Failed tests:* {noformat} TestStatsReplicationScenariosACIDNoAutogather - did not produce a TEST-*.xml file (likely timed out) (batchId=254) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18675/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18675/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18675/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12980937 - PreCommit-HIVE-Build > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934939#comment-16934939 ] Hive QA commented on HIVE-5: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 6s{color} | {color:blue} standalone-metastore/metastore-server in master has 170 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 14m 5s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18675/dev-support/hive-personality.sh | | git revision | master / 1278bf1 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: standalone-metastore/metastore-server U: standalone-metastore/metastore-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18675/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22229) Backport HIVE-8472 to branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned HIVE-9: -- Assignee: (was: Yuming Wang) > Backport HIVE-8472 to branch-2.3 > > > Key: HIVE-9 > URL: https://issues.apache.org/jira/browse/HIVE-9 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.3.6 >Reporter: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934917#comment-16934917 ] Hive QA commented on HIVE-2: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980926/HIVE-2.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16833 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18674/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18674/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18674/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12980926 - PreCommit-HIVE-Build > Clean up the error handling in Driver - get rid of global variables > --- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-2.01.patch > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934906#comment-16934906 ] Hive QA commented on HIVE-2: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 33s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 6s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s{color} | {color:red} ql: The patch generated 5 new + 87 unchanged - 46 fixed = 92 total (was 133) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 26s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18674/dev-support/hive-personality.sh | | git revision | master / 1278bf1 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18674/yetus/diff-checkstyle-ql.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18674/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Clean up the error handling in Driver - get rid of global variables > --- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-2.01.patch > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22229) Backport HIVE-8472 to branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934902#comment-16934902 ] Yuming Wang commented on HIVE-9: [* DDL – Alter Database|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DDL#LanguageManualDDL-AlterDatabase] Please note, Fix Version/s should include 2.3.7 if we backport HIVE-8472 to branch-2.3. > Backport HIVE-8472 to branch-2.3 > > > Key: HIVE-9 > URL: https://issues.apache.org/jira/browse/HIVE-9 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.3.6 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22229) Backport HIVE-8472 to branch-2.3
[ https://issues.apache.org/jira/browse/HIVE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned HIVE-9: -- > Backport HIVE-8472 to branch-2.3 > > > Key: HIVE-9 > URL: https://issues.apache.org/jira/browse/HIVE-9 > Project: Hive > Issue Type: Improvement > Components: Database/Schema >Affects Versions: 2.3.6 >Reporter: Yuming Wang >Assignee: Yuming Wang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21975) Fix incremental compilation
[ https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934899#comment-16934899 ] Hive QA commented on HIVE-21975: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980924/HIVE-21975.1.patch {color:green}SUCCESS:{color} +1 due to 13 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16833 tests executed *Failed tests:* {noformat} org.apache.hive.jdbc.TestSSL.testMetastoreWithSSL (batchId=282) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18673/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18673/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18673/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12980924 - PreCommit-HIVE-Build > Fix incremental compilation > --- > > Key: HIVE-21975 > URL: https://issues.apache.org/jira/browse/HIVE-21975 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Steve Carlin >Priority: Major > Attachments: HIVE-21975.1.patch, HIVE-21975.patch > > > we have an incremental compilation issue around SA ; mostly because of "? > extends Serializable" > it could be reproduced with: > {code} > git clean -dfx > mvn install -pl ql -am -DskipTests > touch `find . -name Sema*A*java` `find . -name Task*Factory.java` > mvn install -pl ql -DskipTests > {code} > error is: > {code} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile > (default-compile) on project hive-exec: Compilation failure: Compilation > failure: > [ERROR] > /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[12573,60] > incompatible types: java.util.List extends java.io.Serializable>> cannot be converted to > java.util.List> > [ERROR] > /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[15187,49] > incompatible types: java.util.List> > cannot be converted to java.util.List extends java.io.Serializable>> > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-1: - Attachment: HIVE-1.2.patch > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-1.1.patch, HIVE-1.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21975) Fix incremental compilation
[ https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934896#comment-16934896 ] Hive QA commented on HIVE-21975: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 23s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 53s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 49s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 55s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} service in master has 49 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} hcatalog/core in master has 36 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 44s{color} | {color:blue} itests/util in master has 51 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 53s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 55s{color} | {color:red} ql: The patch generated 188 new + 1675 unchanged - 95 fixed = 1863 total (was 1770) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} service: The patch generated 2 new + 54 unchanged - 1 fixed = 56 total (was 55) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} hcatalog/core: The patch generated 4 new + 43 unchanged - 0 fixed = 47 total (was 43) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 586 unchanged - 1 fixed = 590 total (was 587) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s{color} | {color:red} itests/util: The patch generated 4 new + 14 unchanged - 0 fixed = 18 total (was 14) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 11s{color} | {color:red} ql generated 2 new + 1571 unchanged - 0 fixed = 1573 total (was 1571) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 41m 39s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Inconsistent synchronization of org.apache.hadoop.hive.ql.QueryDisplay$TaskDisplay.beginTime; locked 57% of time Unsynchronized access at QueryDisplay.java:57% of time Unsynchronized access at QueryDisplay.java:[line 243] | | | Inconsistent synchronization of org.apache.hadoop.hive.ql.QueryDisplay$TaskDisplay.endTime; locked 50% of time Unsynchronized access at QueryDisplay.java:50% of time Unsynchronized access at QueryDisplay.java:[line 248] | \\ \\ || Subs
[jira] [Commented] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934885#comment-16934885 ] Hive QA commented on HIVE-1: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980909/HIVE-1.1.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16840 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_functions] (batchId=81) org.apache.hadoop.hive.metastore.TestMetaStoreEventListenerOnlyOnCommit.testEventStatus (batchId=233) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18672/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18672/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18672/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12980909 - PreCommit-HIVE-Build > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-1.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Status: Patch Available (was: Open) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Attachment: HIVE-22197.05.patch > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Status: Open (was: Patch Available) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-22197: --- Attachment: (was: HIVE-22197.05.patch) > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934875#comment-16934875 ] Hive QA commented on HIVE-1: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 48s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 22s{color} | {color:blue} llap-client in master has 26 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 4s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 24s{color} | {color:blue} llap-ext-client in master has 1 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 37s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 10s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 40s{color} | {color:red} ql: The patch generated 49 new + 96 unchanged - 4 fixed = 145 total (was 100) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 10s{color} | {color:red} llap-ext-client: The patch generated 3 new + 36 unchanged - 2 fixed = 39 total (was 38) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s{color} | {color:red} itests/hive-unit: The patch generated 9 new + 53 unchanged - 5 fixed = 62 total (was 58) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s{color} | {color:red} llap-client generated 1 new + 26 unchanged - 0 fixed = 27 total (was 26) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 12s{color} | {color:red} ql generated 1 new + 1570 unchanged - 1 fixed = 1571 total (was 1571) {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 33s{color} | {color:red} llap-ext-client generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 34m 54s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:llap-client | | | org.apache.hadoop.hive.llap.LlapInputSplit.setPlanBytes(byte[]) may expose internal representation by storing an externally mutable object into LlapInputSplit.planBytes At LlapInputSplit.java:by storing an externally mutable object into LlapInputSplit.planBytes At LlapInputSplit.java:[line 95] | | FindBugs | module:ql | | | Redundant nullcheck of driverCleanup, which is known to be non-null in org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlanFragment(String, ApplicationId) Redundant null check at GenericUDTFGetSplits.java:is known to be non-null in org.apache.hadoop.hive.ql.udf.generic.GenericUDTFGetSplits.createPlan
[jira] [Commented] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[ https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934863#comment-16934863 ] Hive QA commented on HIVE-22209: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980896/HIVE-22209.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16833 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18671/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18671/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18671/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12980896 - PreCommit-HIVE-Build > Creating a materialized view with no tables should be handled more gracefully > - > > Key: HIVE-22209 > URL: https://issues.apache.org/jira/browse/HIVE-22209 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Steve Carlin >Priority: Minor > Attachments: HIVE-22209.1.patch, HIVE-22209.patch > > > Currently, materialized views without a table reference are not supported. > However, instead of printing a clear message about it, when a materialized > view is created without a table reference, we fail with an unclear message. > {code} > > create materialized view mv_test1 as select 5; > (...) > ERROR : FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request > failed : > INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) > INFO : Completed executing > command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a); > Time taken: 10.469 seconds > Error: Error while processing statement: FAILED: Execution Error, return code > 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep > tion(message:Add request failed : INSERT INTO MV_TABLES_USED > (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[ https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934851#comment-16934851 ] Hive QA commented on HIVE-22209: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 51s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 4s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 24m 47s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18671/dev-support/hive-personality.sh | | git revision | master / 1278bf1 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18671/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Creating a materialized view with no tables should be handled more gracefully > - > > Key: HIVE-22209 > URL: https://issues.apache.org/jira/browse/HIVE-22209 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Steve Carlin >Priority: Minor > Attachments: HIVE-22209.1.patch, HIVE-22209.patch > > > Currently, materialized views without a table reference are not supported. > However, instead of printing a clear message about it, when a materialized > view is created without a table reference, we fail with an unclear message. > {code} > > create materialized view mv_test1 as select 5; > (...) > ERROR : FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request > failed : > INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) > INFO : Completed executing > command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a); > Time taken: 10.469 seconds > Error: Error while processing statement: FAILED: Execution Error, return code > 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep > tion(message:Add request failed : INSERT INTO MV_TABLES_USED > (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22211) Change maven phase to generate test sources
[ https://issues.apache.org/jira/browse/HIVE-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934840#comment-16934840 ] Hive QA commented on HIVE-22211: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980890/HIVE-22211.01.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16833 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18670/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18670/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18670/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12980890 - PreCommit-HIVE-Build > Change maven phase to generate test sources > --- > > Key: HIVE-22211 > URL: https://issues.apache.org/jira/browse/HIVE-22211 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22211.01.patch, HIVE-22211.01.patch, > HIVE-22211.01.patch > > > Some protobuf files are generated in the wrong phase; so I get compile errors > because they are not there for eclipse... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934824#comment-16934824 ] Hive QA commented on HIVE-22205: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 1s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 10s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m 23s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 22s{color} | {color:blue} storage-api in master has 48 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 29s{color} | {color:blue} standalone-metastore/metastore-common in master has 32 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 22s{color} | {color:blue} llap-client in master has 26 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m 10s{color} | {color:blue} standalone-metastore/metastore-server in master has 170 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 59s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 43s{color} | {color:blue} llap-server in master has 90 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 25s{color} | {color:blue} standalone-metastore/metastore-tools/metastore-benchmarks in master has 3 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 21s{color} | {color:blue} upgrade-acid/pre-upgrade in master has 1 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 39s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 50s{color} | {color:blue} itests/util in master has 51 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 12m 24s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 16m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 32s{color} | {color:green} The patch storage-api passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green} The patch standalone-metastore passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s{color} | {color:green} The patch metastore-common passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} The patch llap-client passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 32s{color} | {color:green} The patch metastore-server passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 12s{color} | {color:green} ql: The patch generated 0 new + 6 unchanged - 2 fixed = 6 total (was 8) {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 20s{color} | {color:green} The patch llap-server passed checkstyle {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} The patch kafka-handler passed checkstyle {color} | | {color:green}+1{color} | {color:
[jira] [Updated] (HIVE-22227) Tez bucket pruning produces wrong result with shared work optimization
[ https://issues.apache.org/jira/browse/HIVE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-7: --- Status: Patch Available (was: Open) > Tez bucket pruning produces wrong result with shared work optimization > -- > > Key: HIVE-7 > URL: https://issues.apache.org/jira/browse/HIVE-7 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-7.1.patch > > > *Reproducer* > {code:sql} > set hive.tez.bucket.pruning=true; > set hive.optimize.shared.work=true; > CREATE TABLE srcbucket_mapjoin_n16(key int, value string) partitioned by (ds > string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE; > CREATE TABLE tab_part_n10 (key int, value string) PARTITIONED BY(ds STRING) > CLUSTERED BY (key) SORTED BY (key) INTO 4 BUCKETS STORED AS ORCFILE; > CREATE TABLE srcbucket_mapjoin_part_n17 (key int, value string) partitioned > by (ds string) CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE; > load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE > srcbucket_mapjoin_n16 partition(ds='2008-04-08'); > load data local inpath '.$HIVE_SRC/data/files/bmj1/01_0' INTO TABLE > srcbucket_mapjoin_n16 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/01_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/02_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > set hive.optimize.bucketingsorting=false; > insert overwrite table tab_part_n10 partition (ds='2008-04-08') > select key,value from srcbucket_mapjoin_part_n17; > CREATE TABLE tab_n9(key int, value string) PARTITIONED BY(ds STRING) > CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORCFILE; > insert overwrite table tab_n9 partition (ds='2008-04-08') > select key,value from srcbucket_mapjoin_n16; > select * from > (select * from tab_n9 where tab_n9.key = 0)a > join > (select * from tab_part_n10 where tab_part_n10.key = 98)b full outer join > tab_part_n10 c on a.key = b.key and b.key = c.key > order by 1,2,3,4,5,6,7,8,9; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22227) Tez bucket pruning produces wrong result with shared work optimization
[ https://issues.apache.org/jira/browse/HIVE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-7: --- Attachment: HIVE-7.1.patch > Tez bucket pruning produces wrong result with shared work optimization > -- > > Key: HIVE-7 > URL: https://issues.apache.org/jira/browse/HIVE-7 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-7.1.patch > > > *Reproducer* > {code:sql} > set hive.tez.bucket.pruning=true; > set hive.optimize.shared.work=true; > CREATE TABLE srcbucket_mapjoin_n16(key int, value string) partitioned by (ds > string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE; > CREATE TABLE tab_part_n10 (key int, value string) PARTITIONED BY(ds STRING) > CLUSTERED BY (key) SORTED BY (key) INTO 4 BUCKETS STORED AS ORCFILE; > CREATE TABLE srcbucket_mapjoin_part_n17 (key int, value string) partitioned > by (ds string) CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE; > load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE > srcbucket_mapjoin_n16 partition(ds='2008-04-08'); > load data local inpath '.$HIVE_SRC/data/files/bmj1/01_0' INTO TABLE > srcbucket_mapjoin_n16 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/01_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/02_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > set hive.optimize.bucketingsorting=false; > insert overwrite table tab_part_n10 partition (ds='2008-04-08') > select key,value from srcbucket_mapjoin_part_n17; > CREATE TABLE tab_n9(key int, value string) PARTITIONED BY(ds STRING) > CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORCFILE; > insert overwrite table tab_n9 partition (ds='2008-04-08') > select key,value from srcbucket_mapjoin_n16; > select * from > (select * from tab_n9 where tab_n9.key = 0)a > join > (select * from tab_part_n10 where tab_part_n10.key = 98)b full outer join > tab_part_n10 c on a.key = b.key and b.key = c.key > order by 1,2,3,4,5,6,7,8,9; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22211) Change maven phase to generate test sources
[ https://issues.apache.org/jira/browse/HIVE-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934808#comment-16934808 ] Hive QA commented on HIVE-22211: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m 11s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 45s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 41s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 21m 18s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc xml compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18670/dev-support/hive-personality.sh | | git revision | master / 1278bf1 | | Default Java | 1.8.0_111 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18670/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Change maven phase to generate test sources > --- > > Key: HIVE-22211 > URL: https://issues.apache.org/jira/browse/HIVE-22211 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22211.01.patch, HIVE-22211.01.patch, > HIVE-22211.01.patch > > > Some protobuf files are generated in the wrong phase; so I get compile errors > because they are not there for eclipse... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22228) SemanticAnalyzer cleanup - visibility + types
[ https://issues.apache.org/jira/browse/HIVE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-8: -- Status: Patch Available (was: Open) > SemanticAnalyzer cleanup - visibility + types > - > > Key: HIVE-8 > URL: https://issues.apache.org/jira/browse/HIVE-8 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-8.01.patch > > > Cleaning up SemanticAnalyzer: > * reduce the visibility of those functions/variables that are too wide, so > their scope is clearer > * modify the type of data structures, use interface instead of actual > implementation (e.g. HashMap -> Map in variable declaration) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22228) SemanticAnalyzer cleanup - visibility + types
[ https://issues.apache.org/jira/browse/HIVE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-8: -- Attachment: HIVE-8.01.patch > SemanticAnalyzer cleanup - visibility + types > - > > Key: HIVE-8 > URL: https://issues.apache.org/jira/browse/HIVE-8 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-8.01.patch > > > Cleaning up SemanticAnalyzer: > * reduce the visibility of those functions/variables that are too wide, so > their scope is clearer > * modify the type of data structures, use interface instead of actual > implementation (e.g. HashMap -> Map in variable declaration) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934794#comment-16934794 ] Hive QA commented on HIVE-22205: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980849/HIVE-22205.4.patch {color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16828 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.org.apache.hadoop.hive.cli.TestAccumuloCliDriver (batchId=289) org.apache.hadoop.hive.ql.TestTxnCommands.testParallelInsertAnalyzeStats (batchId=351) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18669/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18669/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18669/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12980849 - PreCommit-HIVE-Build > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, > HIVE-22205.4.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22228) SemanticAnalyzer cleanup - visibility + types
[ https://issues.apache.org/jira/browse/HIVE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-8: - > SemanticAnalyzer cleanup - visibility + types > - > > Key: HIVE-8 > URL: https://issues.apache.org/jira/browse/HIVE-8 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Cleaning up SemanticAnalyzer: > * reduce the visibility of those functions/variables that are too wide, so > their scope is clearer > * modify the type of data structures, use interface instead of actual > implementation (e.g. HashMap -> Map in variable declaration) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22227) Tez bucket pruning produces wrong result with shared work optimization
[ https://issues.apache.org/jira/browse/HIVE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-7: --- Description: *Reproducer* {code:sql} set hive.tez.bucket.pruning=true; set hive.optimize.shared.work=true; CREATE TABLE srcbucket_mapjoin_n16(key int, value string) partitioned by (ds string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE; CREATE TABLE tab_part_n10 (key int, value string) PARTITIONED BY(ds STRING) CLUSTERED BY (key) SORTED BY (key) INTO 4 BUCKETS STORED AS ORCFILE; CREATE TABLE srcbucket_mapjoin_part_n17 (key int, value string) partitioned by (ds string) CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE; load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE srcbucket_mapjoin_n16 partition(ds='2008-04-08'); load data local inpath '.$HIVE_SRC/data/files/bmj1/01_0' INTO TABLE srcbucket_mapjoin_n16 partition(ds='2008-04-08'); load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); load data local inpath '$HIVE_SRC/data/files/bmj/01_0' INTO TABLE srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); load data local inpath '$HIVE_SRC/data/files/bmj/02_0' INTO TABLE srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); set hive.optimize.bucketingsorting=false; insert overwrite table tab_part_n10 partition (ds='2008-04-08') select key,value from srcbucket_mapjoin_part_n17; CREATE TABLE tab_n9(key int, value string) PARTITIONED BY(ds STRING) CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORCFILE; insert overwrite table tab_n9 partition (ds='2008-04-08') select key,value from srcbucket_mapjoin_n16; select * from (select * from tab_n9 where tab_n9.key = 0)a join (select * from tab_part_n10 where tab_part_n10.key = 98)b full outer join tab_part_n10 c on a.key = b.key and b.key = c.key order by 1,2,3,4,5,6,7,8,9; {code} > Tez bucket pruning produces wrong result with shared work optimization > -- > > Key: HIVE-7 > URL: https://issues.apache.org/jira/browse/HIVE-7 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > > *Reproducer* > {code:sql} > set hive.tez.bucket.pruning=true; > set hive.optimize.shared.work=true; > CREATE TABLE srcbucket_mapjoin_n16(key int, value string) partitioned by (ds > string) CLUSTERED BY (key) INTO 2 BUCKETS STORED AS TEXTFILE; > CREATE TABLE tab_part_n10 (key int, value string) PARTITIONED BY(ds STRING) > CLUSTERED BY (key) SORTED BY (key) INTO 4 BUCKETS STORED AS ORCFILE; > CREATE TABLE srcbucket_mapjoin_part_n17 (key int, value string) partitioned > by (ds string) CLUSTERED BY (key) INTO 4 BUCKETS STORED AS TEXTFILE; > load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE > srcbucket_mapjoin_n16 partition(ds='2008-04-08'); > load data local inpath '.$HIVE_SRC/data/files/bmj1/01_0' INTO TABLE > srcbucket_mapjoin_n16 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/00_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/01_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > load data local inpath '$HIVE_SRC/data/files/bmj/02_0' INTO TABLE > srcbucket_mapjoin_part_n17 partition(ds='2008-04-08'); > set hive.optimize.bucketingsorting=false; > insert overwrite table tab_part_n10 partition (ds='2008-04-08') > select key,value from srcbucket_mapjoin_part_n17; > CREATE TABLE tab_n9(key int, value string) PARTITIONED BY(ds STRING) > CLUSTERED BY (key) SORTED BY (key) INTO 2 BUCKETS STORED AS ORCFILE; > insert overwrite table tab_n9 partition (ds='2008-04-08') > select key,value from srcbucket_mapjoin_n16; > select * from > (select * from tab_n9 where tab_n9.key = 0)a > join > (select * from tab_part_n10 where tab_part_n10.key = 98)b full outer join > tab_part_n10 c on a.key = b.key and b.key = c.key > order by 1,2,3,4,5,6,7,8,9; > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22227) Tez bucket pruning produces wrong result with shared work optimization
[ https://issues.apache.org/jira/browse/HIVE-7?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-7: -- > Tez bucket pruning produces wrong result with shared work optimization > -- > > Key: HIVE-7 > URL: https://issues.apache.org/jira/browse/HIVE-7 > Project: Hive > Issue Type: Bug > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22219) Bringing a node manager down blocks restart of LLAP service
[ https://issues.apache.org/jira/browse/HIVE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22219: --- Attachment: HIVE-22219.patch > Bringing a node manager down blocks restart of LLAP service > --- > > Key: HIVE-22219 > URL: https://issues.apache.org/jira/browse/HIVE-22219 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22219.patch, HIVE-22219.patch > > > For YARN service, when number of running instances != number of desired > instances, the service state may be in STARTED or FLEX (instead of STABLE). > For Hive LLAP side, there is a config to control the threshold of service > health check. The Hive LLAP code misses checking these states, which can > result in the service not coming up even if the threshold is met. > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusServiceDriver.java#L382 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22219) Bringing a node manager down blocks restart of LLAP service
[ https://issues.apache.org/jira/browse/HIVE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934734#comment-16934734 ] Hive QA commented on HIVE-22219: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980840/HIVE-22219.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16801 tests executed *Failed tests:* {noformat} TestDataSourceProviderFactory - did not produce a TEST-*.xml file (likely timed out) (batchId=233) TestObjectStore - did not produce a TEST-*.xml file (likely timed out) (batchId=233) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18666/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18666/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18666/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12980840 - PreCommit-HIVE-Build > Bringing a node manager down blocks restart of LLAP service > --- > > Key: HIVE-22219 > URL: https://issues.apache.org/jira/browse/HIVE-22219 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22219.patch > > > For YARN service, when number of running instances != number of desired > instances, the service state may be in STARTED or FLEX (instead of STABLE). > For Hive LLAP side, there is a config to control the threshold of service > health check. The Hive LLAP code misses checking these states, which can > result in the service not coming up even if the threshold is met. > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusServiceDriver.java#L382 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22145) Avoid optimizations for analyze compute statistics
[ https://issues.apache.org/jira/browse/HIVE-22145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22145: --- Attachment: HIVE-22145.3.patch > Avoid optimizations for analyze compute statistics > -- > > Key: HIVE-22145 > URL: https://issues.apache.org/jira/browse/HIVE-22145 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22145.1.patch, HIVE-22145.2.patch, > HIVE-22145.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22145) Avoid optimizations for analyze compute statistics
[ https://issues.apache.org/jira/browse/HIVE-22145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22145: --- Status: Open (was: Patch Available) > Avoid optimizations for analyze compute statistics > -- > > Key: HIVE-22145 > URL: https://issues.apache.org/jira/browse/HIVE-22145 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22145.1.patch, HIVE-22145.2.patch, > HIVE-22145.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22145) Avoid optimizations for analyze compute statistics
[ https://issues.apache.org/jira/browse/HIVE-22145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22145: --- Status: Patch Available (was: Open) > Avoid optimizations for analyze compute statistics > -- > > Key: HIVE-22145 > URL: https://issues.apache.org/jira/browse/HIVE-22145 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22145.1.patch, HIVE-22145.2.patch, > HIVE-22145.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Attachment: HIVE-20113.10.patch > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Vineet Garg >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.10.patch, HIVE-20113.2.patch, HIVE-20113.3.patch, > HIVE-20113.4.patch, HIVE-20113.4.patch, HIVE-20113.5.patch, > HIVE-20113.6.patch, HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Status: Patch Available (was: Open) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Vineet Garg >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.10.patch, HIVE-20113.2.patch, HIVE-20113.3.patch, > HIVE-20113.4.patch, HIVE-20113.4.patch, HIVE-20113.5.patch, > HIVE-20113.6.patch, HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-20113: --- Status: Open (was: Patch Available) > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Vineet Garg >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.10.patch, HIVE-20113.2.patch, HIVE-20113.3.patch, > HIVE-20113.4.patch, HIVE-20113.4.patch, HIVE-20113.5.patch, > HIVE-20113.6.patch, HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22217) Better Logging for Hive JAR Reload
[ https://issues.apache.org/jira/browse/HIVE-22217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934695#comment-16934695 ] Naveen Gangam commented on HIVE-22217: -- Patch looks good to me. So +1. Thanks [~belugabehr] > Better Logging for Hive JAR Reload > -- > > Key: HIVE-22217 > URL: https://issues.apache.org/jira/browse/HIVE-22217 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.2.0, 2.3.6 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22217.1.patch > > > Troubleshooting Hive Reloadable Auxiliary JARs has always been difficult. > Add logging to at least confirm which JAR files are being loaded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22219) Bringing a node manager down blocks restart of LLAP service
[ https://issues.apache.org/jira/browse/HIVE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934691#comment-16934691 ] Hive QA commented on HIVE-22219: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 40s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 43s{color} | {color:blue} llap-server in master has 90 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 56s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18666/dev-support/hive-personality.sh | | git revision | master / 1278bf1 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: llap-server U: llap-server | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18666/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Bringing a node manager down blocks restart of LLAP service > --- > > Key: HIVE-22219 > URL: https://issues.apache.org/jira/browse/HIVE-22219 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22219.patch > > > For YARN service, when number of running instances != number of desired > instances, the service state may be in STARTED or FLEX (instead of STABLE). > For Hive LLAP side, there is a config to control the threshold of service > health check. The Hive LLAP code misses checking these states, which can > result in the service not coming up even if the threshold is met. > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusServiceDriver.java#L382 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934673#comment-16934673 ] Hive QA commented on HIVE-22197: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980839/HIVE-22197.05.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16834 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=361) org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable (batchId=223) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18665/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18665/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18665/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12980839 - PreCommit-HIVE-Build > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[ https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22079: --- Status: Patch Available (was: Open) > Post order walker for iterating over expression tree > > > Key: HIVE-22079 > URL: https://issues.apache.org/jira/browse/HIVE-22079 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer, Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch, > HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, > HIVE-22079.6.patch, HIVE-22079.7.patch > > > Current {{DefaultGraphWalker}} is used to iterate over an expression tree. > This walker uses hash map to keep track of visited/processed nodes. If an > expression tree is large this adds significant overhead due to map lookup. > For an expression trees we can instead use post order traversal and avoid > using map. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[ https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22079: --- Status: Open (was: Patch Available) > Post order walker for iterating over expression tree > > > Key: HIVE-22079 > URL: https://issues.apache.org/jira/browse/HIVE-22079 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer, Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch, > HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, > HIVE-22079.6.patch, HIVE-22079.7.patch > > > Current {{DefaultGraphWalker}} is used to iterate over an expression tree. > This walker uses hash map to keep track of visited/processed nodes. If an > expression tree is large this adds significant overhead due to map lookup. > For an expression trees we can instead use post order traversal and avoid > using map. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22079) Post order walker for iterating over expression tree
[ https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-22079: --- Attachment: HIVE-22079.7.patch > Post order walker for iterating over expression tree > > > Key: HIVE-22079 > URL: https://issues.apache.org/jira/browse/HIVE-22079 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer, Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch, > HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, > HIVE-22079.6.patch, HIVE-22079.7.patch > > > Current {{DefaultGraphWalker}} is used to iterate over an expression tree. > This walker uses hash map to keep track of visited/processed nodes. If an > expression tree is large this adds significant overhead due to map lookup. > For an expression trees we can instead use post order traversal and avoid > using map. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An updated HIVE-5: -- Attachment: HIVE-5.1.patch Status: Patch Available (was: Open) > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Attachments: HIVE-5.1.patch > > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22197) Common Merge join throwing class cast exception
[ https://issues.apache.org/jira/browse/HIVE-22197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934627#comment-16934627 ] Hive QA commented on HIVE-22197: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 46s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 59s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 22s{color} | {color:blue} shims/0.23 in master has 7 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 54s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 29 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 26m 45s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18665/dev-support/hive-personality.sh | | git revision | master / 1278bf1 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-18665/yetus/whitespace-tabs.txt | | modules | C: shims/0.23 ql U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18665/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Common Merge join throwing class cast exception > > > Key: HIVE-22197 > URL: https://issues.apache.org/jira/browse/HIVE-22197 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22197.01.patch, HIVE-22197.05.patch > > > In DummyStoreOperator the row is cached to fix HIVE-5973. The row is copyed > and stored in the writable format, but the object inspector is initialized to > default. So when join operator is fetching the data from dummy store > operator, its getting the OI is Long and the row as LongWritable. This is > causing the class cast exception. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20456) Query fails with FNFException using MR with skewjoin enabled and auto convert join disabled
[ https://issues.apache.org/jira/browse/HIVE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934599#comment-16934599 ] Hive QA commented on HIVE-20456: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12936968/HIVE-20456.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16833 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint] (batchId=172) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18664/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18664/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18664/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12936968 - PreCommit-HIVE-Build > Query fails with FNFException using MR with skewjoin enabled and auto convert > join disabled > --- > > Key: HIVE-20456 > URL: https://issues.apache.org/jira/browse/HIVE-20456 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.0, 2.1.1, 3.1.0 >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-20456.patch > > > When skew join is enabled and auto convert join is disabled the query fails > with file not found exception. The following query reproduces the error: > > {code:java} > set hive.optimize.skewjoin = true; > set hive.auto.convert.join = false; > set hive.groupby.orderby.position.alias = true; > set hive.on.master=true; > set hive.execution.engine=mr; > set hive.exec.parallel=false; > drop database if exists test cascade; > create database if not exists test; > use test; > CREATE EXTERNAL TABLE test_table1 > ( `a` int , `b` int, `c` int) > PARTITIONED BY ( > `d` int) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > ; > CREATE EXTERNAL TABLE test_table2 > ( `a` int , `b` int, `c` int) > PARTITIONED BY ( > `d` int) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'; > CREATE EXTERNAL TABLE test_table3 > ( `a` int , `b` int, `c` int) > PARTITIONED BY ( > `e` int) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' > WITH SERDEPROPERTIES ( > 'field.delim'='\u0001', > 'serialization.format'='\u0001') > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'; > CREATE EXTERNAL TABLE test_table4 (`a` int , `b` int, `c` int) > PARTITIONED BY ( > `e` string) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' > WITH SERDEPROPERTIES ( > 'field.delim'='\u0001', > 'serialization.format'='\u0001') > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'; > with > temp1 as ( > select > g.a, > n.b, > u.c > from > test_table2 g > inner join test_table4 u on g.a = u.a > inner join test_table3 n on u.b = n.b > ), > temp2 as ( > select > n.a, > n.b, > u.c > from > test_table4 u > inner join test_table3 n on u.b = n.b > ), > temp21 as ( > select > g.b, > n.c, > u.a > from > temp2 g > inner join test_table3 u on g.b = u.b > inner join test_table2 n on u.c = n.c > group by g.b, n.c, u.a > ), > stack as ( > select * from temp1 > union all > select * from temp21 > ) > select * from stack; > inner join test_table2 n on u.c = n.c > group by g.b, n.c, u.a > ), > stack as ( > select * from temp1 > union all > select * from temp21 > ) > select * from stack; > {code} > The query runs perfectly fine when tez is used or other combinations of skew > join and auto convert join are set. On diagnosing the issue, the problem was > when a conditional task resolves tasks it puts the resolved task directly in > the runnable state without checking the parental dependencies as well as > whether the task is already queued. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934520#comment-16934520 ] Dongjoon Hyun commented on HIVE-21237: -- My bad, [~dawood.m] and guys. I removed my previous comment. Sorry again. > [JDK 11] SessionState can't be initialized due to classloader problem > - > > Key: HIVE-21237 > URL: https://issues.apache.org/jira/browse/HIVE-21237 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 > Environment: JDK11, Hadoop-3, Hive 3.1.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > When I start Hive with JDK11 > {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID = > cecd9c34-d61a-44d0-9e52-a0a7d6413e49 > Exception in thread "main" java.lang.ClassCastException: class > jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class > java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and > java.net.URLClassLoader are in module java.base of loader 'bootstrap') > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410) > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386) > at > org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at org.apache.hadoop.util.RunJar.run(RunJar.java:323) > at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated HIVE-21237: - Comment: was deleted (was: [~dawood.m]. The following is `Hive` command. If that fails, there is nothing for Apache Spark community to help you. Please report to Apache Hive community. {code} [root@apachehive hive]# bin/schematool -dbType derby -initSchema --verbose {code}) > [JDK 11] SessionState can't be initialized due to classloader problem > - > > Key: HIVE-21237 > URL: https://issues.apache.org/jira/browse/HIVE-21237 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 > Environment: JDK11, Hadoop-3, Hive 3.1.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > When I start Hive with JDK11 > {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID = > cecd9c34-d61a-44d0-9e52-a0a7d6413e49 > Exception in thread "main" java.lang.ClassCastException: class > jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class > java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and > java.net.URLClassLoader are in module java.base of loader 'bootstrap') > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410) > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386) > at > org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at org.apache.hadoop.util.RunJar.run(RunJar.java:323) > at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934519#comment-16934519 ] Dongjoon Hyun commented on HIVE-21237: -- [~dawood.m]. The following is `Hive` command. If that fails, there is nothing for Apache Spark community to help you. Please report to Apache Hive community. {code} [root@apachehive hive]# bin/schematool -dbType derby -initSchema --verbose {code} > [JDK 11] SessionState can't be initialized due to classloader problem > - > > Key: HIVE-21237 > URL: https://issues.apache.org/jira/browse/HIVE-21237 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 > Environment: JDK11, Hadoop-3, Hive 3.1.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > When I start Hive with JDK11 > {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID = > cecd9c34-d61a-44d0-9e52-a0a7d6413e49 > Exception in thread "main" java.lang.ClassCastException: class > jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class > java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and > java.net.URLClassLoader are in module java.base of loader 'bootstrap') > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410) > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386) > at > org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at org.apache.hadoop.util.RunJar.run(RunJar.java:323) > at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22225) add owner name for create database if missing
[ https://issues.apache.org/jira/browse/HIVE-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam An reassigned HIVE-5: - > add owner name for create database if missing > - > > Key: HIVE-5 > URL: https://issues.apache.org/jira/browse/HIVE-5 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Sam An >Assignee: Sam An >Priority: Minor > > When Spark connects to HMS, the database owner name is not filled. This could > happen to other clients as well. We shall fill this in HMS. Ownership info is > useful for authorizer component in Ranger, etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20456) Query fails with FNFException using MR with skewjoin enabled and auto convert join disabled
[ https://issues.apache.org/jira/browse/HIVE-20456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934548#comment-16934548 ] Hive QA commented on HIVE-20456: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 59s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 47s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18664/dev-support/hive-personality.sh | | git revision | master / 1278bf1 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18664/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Query fails with FNFException using MR with skewjoin enabled and auto convert > join disabled > --- > > Key: HIVE-20456 > URL: https://issues.apache.org/jira/browse/HIVE-20456 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.2.0, 2.1.1, 3.1.0 >Reporter: Aditya Shah >Assignee: Aditya Shah >Priority: Major > Attachments: HIVE-20456.patch > > > When skew join is enabled and auto convert join is disabled the query fails > with file not found exception. The following query reproduces the error: > > {code:java} > set hive.optimize.skewjoin = true; > set hive.auto.convert.join = false; > set hive.groupby.orderby.position.alias = true; > set hive.on.master=true; > set hive.execution.engine=mr; > set hive.exec.parallel=false; > drop database if exists test cascade; > create database if not exists test; > use test; > CREATE EXTERNAL TABLE test_table1 > ( `a` int , `b` int, `c` int) > PARTITIONED BY ( > `d` int) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' > STORED AS INPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' > ; > CREATE EXTERNAL TABLE test_table2 > ( `a` int , `b` int, `c` int) > PARTITIONED BY ( > `d` int) > ROW FORMAT SERDE > 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' > STORED AS INPUTFORMAT > 'org.apache.hadoop.mapred.TextInputFormat' > OUTPUTFORMAT > 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'; > CREATE EXTERNAL TABLE test_ta
[jira] [Updated] (HIVE-22224) Support Parquet-Avro Timestamp Type
[ https://issues.apache.org/jira/browse/HIVE-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cdmikechen updated HIVE-4: -- Labels: parquet (was: ) > Support Parquet-Avro Timestamp Type > --- > > Key: HIVE-4 > URL: https://issues.apache.org/jira/browse/HIVE-4 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 2.3.5, 2.3.6 >Reporter: cdmikechen >Priority: Major > Labels: parquet > Fix For: 2.3.7 > > > When user create an external table and import a parquet-avro data with 1.8.2 > version which supported logical_type in Hive2.3 or before version, Hive can > not read timestamp type column data correctly. > Hive will read it as LongWritable which it actually stores as > long(logical_type=timestamp-millis).So we may add some codes in > org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java > to let Hive cast long type to timestamp type. > Some code like below: > > public Timestamp getPrimitiveJavaObject(Object o) { > if (o instanceof LongWritable) { > return new Timestamp(((LongWritable) o).get()); > } > return o == null ? null : ((TimestampWritable) o).getTimestamp(); > } > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22224) Support Parquet-Avro Timestamp Type
[ https://issues.apache.org/jira/browse/HIVE-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cdmikechen reassigned HIVE-4: - Assignee: cdmikechen > Support Parquet-Avro Timestamp Type > --- > > Key: HIVE-4 > URL: https://issues.apache.org/jira/browse/HIVE-4 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 2.3.5, 2.3.6 >Reporter: cdmikechen >Assignee: cdmikechen >Priority: Major > Labels: parquet > Fix For: 2.3.7 > > > When user create an external table and import a parquet-avro data with 1.8.2 > version which supported logical_type in Hive2.3 or before version, Hive can > not read timestamp type column data correctly. > Hive will read it as LongWritable which it actually stores as > long(logical_type=timestamp-millis).So we may add some codes in > org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java > to let Hive cast long type to timestamp type. > Some code like below: > > public Timestamp getPrimitiveJavaObject(Object o) { > if (o instanceof LongWritable) { > return new Timestamp(((LongWritable) o).get()); > } > return o == null ? null : ((TimestampWritable) o).getTimestamp(); > } > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22224) Support Parquet-Avro Timestamp Type
[ https://issues.apache.org/jira/browse/HIVE-4?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] cdmikechen updated HIVE-4: -- Description: When user create an external table and import a parquet-avro data with 1.8.2 version which supported logical_type in Hive2.3 or before version, Hive can not read timestamp type column data correctly. Hive will read it as LongWritable which it actually stores as long(logical_type=timestamp-millis).So we may add some codes in org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java to let Hive cast long type to timestamp type. Some code like below: public Timestamp getPrimitiveJavaObject(Object o) { if (o instanceof LongWritable) { return new Timestamp(((LongWritable) o).get()); } return o == null ? null : ((TimestampWritable) o).getTimestamp(); } was: When user create an external table and import a parquet-avro data with 1.8.2 version which supported logical_type in Hive2.3 or before version, Hive can not read timestamp type column data correctly. Hive will read it as LongWritable which it actually stores as long(logical_type=timestamp-millis).So we may add some codes in org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java to let Hive cast long type to timestamp type. Some code like below: public Timestamp getPrimitiveJavaObject(Object o) { if (o instanceof LongWritable) { return new Timestamp(((LongWritable) o).get()); } return o == null ? null : ((TimestampWritable) o).getTimestamp(); } > Support Parquet-Avro Timestamp Type > --- > > Key: HIVE-4 > URL: https://issues.apache.org/jira/browse/HIVE-4 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 2.3.5, 2.3.6 >Reporter: cdmikechen >Priority: Major > Labels: parquet > Fix For: 2.3.7 > > > When user create an external table and import a parquet-avro data with 1.8.2 > version which supported logical_type in Hive2.3 or before version, Hive can > not read timestamp type column data correctly. > Hive will read it as LongWritable which it actually stores as > long(logical_type=timestamp-millis).So we may add some codes in > org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableTimestampObjectInspector.java > to let Hive cast long type to timestamp type. > Some code like below: > > public Timestamp getPrimitiveJavaObject(Object o) { > if (o instanceof LongWritable) { > return new Timestamp(((LongWritable) o).get()); > } > return o == null ? null : ((TimestampWritable) o).getTimestamp(); > } > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-2: -- Attachment: HIVE-2.01.patch > Clean up the error handling in Driver - get rid of global variables > --- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-2.01.patch > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-2: -- Status: Patch Available (was: Open) > Clean up the error handling in Driver - get rid of global variables > --- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-2.01.patch > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22222) Clean up the error handling in Driver - get rid of global variables
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-2: -- Summary: Clean up the error handling in Driver - get rid of global variables (was: Clean up the error handling in Driver.) > Clean up the error handling in Driver - get rid of global variables > --- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22222) Clean up the error handling in Driver.
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-2: - > Clean up the error handling in Driver. > -- > > Key: HIVE-2 > URL: https://issues.apache.org/jira/browse/HIVE-2 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > The error handling in Hive is done with some global variables for no apparent > reason, as all the data that is gathered to described an exception are > produced and used at the point where the exception occurred. Thus having > global variables is misleading. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21975) Fix incremental compilation
[ https://issues.apache.org/jira/browse/HIVE-21975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Carlin updated HIVE-21975: Attachment: HIVE-21975.1.patch > Fix incremental compilation > --- > > Key: HIVE-21975 > URL: https://issues.apache.org/jira/browse/HIVE-21975 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Steve Carlin >Priority: Major > Attachments: HIVE-21975.1.patch, HIVE-21975.patch > > > we have an incremental compilation issue around SA ; mostly because of "? > extends Serializable" > it could be reproduced with: > {code} > git clean -dfx > mvn install -pl ql -am -DskipTests > touch `find . -name Sema*A*java` `find . -name Task*Factory.java` > mvn install -pl ql -DskipTests > {code} > error is: > {code} > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile > (default-compile) on project hive-exec: Compilation failure: Compilation > failure: > [ERROR] > /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[12573,60] > incompatible types: java.util.List extends java.io.Serializable>> cannot be converted to > java.util.List> > [ERROR] > /mnt/work/hwx/hive/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:[15187,49] > incompatible types: java.util.List> > cannot be converted to java.util.List extends java.io.Serializable>> > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22219) Bringing a node manager down blocks restart of LLAP service
[ https://issues.apache.org/jira/browse/HIVE-22219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934474#comment-16934474 ] slim bouguerra commented on HIVE-22219: --- +1 > Bringing a node manager down blocks restart of LLAP service > --- > > Key: HIVE-22219 > URL: https://issues.apache.org/jira/browse/HIVE-22219 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22219.patch > > > For YARN service, when number of running instances != number of desired > instances, the service state may be in STARTED or FLEX (instead of STABLE). > For Hive LLAP side, there is a config to control the threshold of service > health check. The Hive LLAP code misses checking these states, which can > result in the service not coming up even if the threshold is met. > https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/cli/status/LlapStatusServiceDriver.java#L382 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934475#comment-16934475 ] Dawood commented on HIVE-21237: --- [~yumwang]on my other setup it fails with below error, {noformat} [root@apachehive hive]# bin/schematool -dbType derby -initSchema --verbose SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Metastore connection URL: jdbc:derby://apachehive:1527/metastore_db;create=true Metastore Connection Driver :org.apache.derby.jdbc.EmbeddedDriver Metastore connection User: APP org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version. Underlying cause: java.sql.SQLNonTransientConnectionException : java.net.ConnectException : Error connecting to server apachehive on port 1,527 with message Connection refused (Connection refused). SQL Error code: 4 org.apache.hadoop.hive.metastore.HiveMetaException: Failed to get schema version. at org.apache.hive.beeline.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:77) at org.apache.hive.beeline.HiveSchemaTool.getConnectionToMetastore(HiveSchemaTool.java:144) at org.apache.hive.beeline.HiveSchemaTool.testConnectionToMetastore(HiveSchemaTool.java:473) at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:577) at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:563) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1145) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) Caused by: java.sql.SQLNonTransientConnectionException: java.net.ConnectException : Error connecting to server apachehive on port 1,527 with message Connection refused (Connection refused). at org.apache.derby.client.am.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.client.am.SqlException.getSQLException(Unknown Source) at org.apache.derby.jdbc.ClientDriver.connect(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228) at org.apache.hive.beeline.HiveSchemaHelper.getConnectionToMetastore(HiveSchemaHelper.java:73) ... 11 more Caused by: ERROR 08001: java.net.ConnectException : Error connecting to server apachehive on port 1,527 with message Connection refused (Connection refused). at org.apache.derby.client.net.NetAgent.(Unknown Source) at org.apache.derby.client.net.NetConnection.newAgent_(Unknown Source) at org.apache.derby.client.am.ClientConnection.(Unknown Source) at org.apache.derby.client.net.NetConnection.(Unknown Source) at org.apache.derby.client.net.ClientJDBCObjectFactoryImpl.newNetConnection(Unknown Source) ... 15 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) at java.base/java.net.Socket.connect(Socket.java:591) at java.base/java.net.Socket.connect(Socket.java:540) at java.base/java.net.Socket.(Socket.java:436) at java.base/java.net.Socket.(Socket.java:213) at java.base/javax.net.DefaultSocketFactory.createSocket(SocketFactory.java:271) at org.apache.derby.client.net.OpenSocketAction.run(Unknown Source) at org.apache.derby.client.net.OpenSocketAction.run(Unknown Source) at java.base/java.security.AccessController.doPrivileged(Native Method) ... 20 more *** schemaTool failed *** {noformat} > [JDK 11] SessionState can't be initialized d
[jira] [Commented] (HIVE-21508) ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer
[ https://issues.apache.org/jira/browse/HIVE-21508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934466#comment-16934466 ] Ana Jalba commented on HIVE-21508: -- [~pvary]: This looks like it's similar to the patch for branch-3.1. Would the same check as before need to be done for ptests to see if the changes are unrelated to the failing tests? > ClassCastException when initializing HiveMetaStoreClient on JDK10 or newer > -- > > Key: HIVE-21508 > URL: https://issues.apache.org/jira/browse/HIVE-21508 > Project: Hive > Issue Type: Bug > Components: Clients >Affects Versions: 3.2.0, 2.3.4 >Reporter: Adar Dembo >Assignee: Ana Jalba >Priority: Major > Fix For: 2.4.0, 4.0.0, 2.3.7 > > Attachments: HIVE-21508.1.patch, HIVE-21508.2.branch-2.3.patch, > HIVE-21508.3.branch-2.patch, HIVE-21508.4.branch-3.1.patch, > HIVE-21508.5.branch-3.1.patch, HIVE-21508.6.branch-3.patch, HIVE-21508.patch > > > There's this block of code in {{HiveMetaStoreClient:resolveUris}} (called > from the constructor) on master: > {noformat} > private URI metastoreUris[]; > ... > if (MetastoreConf.getVar(conf, > ConfVars.THRIFT_URI_SELECTION).equalsIgnoreCase("RANDOM")) { > List uriList = Arrays.asList(metastoreUris); > Collections.shuffle(uriList); > metastoreUris = (URI[]) uriList.toArray(); > } > {noformat} > The cast to {{URI[]}} throws a {{ClassCastException}} beginning with JDK 10, > possibly with JDK 9 as well. Note that {{THRIFT_URI_SELECTION}} defaults to > {{RANDOM}} so this should affect anyone who creates a > {{HiveMetaStoreClient}}. On master this can be overridden with {{SEQUENTIAL}} > to avoid the broken case; I'm working against 2.3.4 where there's no such > workaround. > [Here's|https://stackoverflow.com/questions/51372788/array-cast-java-8-vs-java-9] > a StackOverflow post that explains the issue in more detail. Interestingly, > the author described the issue in the context of the HMS; not sure why there > was no follow up with a Hive bug report. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-1: - Attachment: HIVE-1.1.patch Status: Patch Available (was: Open) > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-1.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?focusedWorklogId=315706&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315706 ] ASF GitHub Bot logged work on HIVE-1: - Author: ASF GitHub Bot Created on: 20/Sep/19 14:04 Start Date: 20/Sep/19 14:04 Worklog Time Spent: 10m Work Description: ShubhamChaurasia commented on pull request #778: HIVE-1: Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint URL: https://github.com/apache/hive/pull/778 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 315706) Remaining Estimate: 0h Time Spent: 10m > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-1: -- Labels: pull-request-available (was: ) > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22221) Llap external client - Need to reduce LlapBaseInputFormat#getSplits() footprint
[ https://issues.apache.org/jira/browse/HIVE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia reassigned HIVE-1: > Llap external client - Need to reduce LlapBaseInputFormat#getSplits() > footprint > - > > Key: HIVE-1 > URL: https://issues.apache.org/jira/browse/HIVE-1 > Project: Hive > Issue Type: Bug > Components: llap, UDF >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > > While querying through llap external client, LlapBaseInputFormat#getSplits() > invokes get_splits() (GenericUDTFGetSplits) udtf under the hoods. > GenericUDTFGetSplits returns LlapInputSplit in which planBytes[] occupies > around 90% of the split size. > Depending on data size/partitions and plan, LlapInputSplit can grow upto 1mb > with planBytes[] being common to all the splits and occupying more than 850 > kb. Also, it sometimes causes OOM on HS2 depending on HS2 heap size. > This can be resolved by separating out common parts from actual splits and > reassembling them at client side. > We can also provide an option where client can say it does not want to > reassemble them and can take the control of reassembling in it's hands. > Splits can be broken like: > 1) schema split > 2) plan split > 3) actual split 1 > 4) actual split 2and so on. > This greatly reduces the memory(in my case from 5GB(~5000 splits) to around > 15MB) on server side and hence the data transfer. And this eliminates OOM on > HS2 side. > cc [~jdere] [~sankarh] [~thejas] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21164) ACID: explore how we can avoid a move step during inserts/compaction
[ https://issues.apache.org/jira/browse/HIVE-21164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang reassigned HIVE-21164: Assignee: Laszlo Bodor (was: Vaibhav Gumashta) > ACID: explore how we can avoid a move step during inserts/compaction > > > Key: HIVE-21164 > URL: https://issues.apache.org/jira/browse/HIVE-21164 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.1 >Reporter: Vaibhav Gumashta >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-21164.1.patch, HIVE-21164.10.patch, > HIVE-21164.2.patch, HIVE-21164.3.patch, HIVE-21164.4.patch, > HIVE-21164.5.patch, HIVE-21164.6.patch, HIVE-21164.7.patch, > HIVE-21164.8.patch, HIVE-21164.9.patch > > > Currently, we write compacted data to a temporary location and then move the > files to a final location, which is an expensive operation on some cloud file > systems. Since HIVE-20823 is already in, it can control the visibility of > compacted data for the readers. Therefore, we can perhaps avoid writing data > to a temporary location and directly write compacted data to the intended > final path. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22191) Simplify SemanticAnalyzer by removing unused code
[ https://issues.apache.org/jira/browse/HIVE-22191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akos Dombi updated HIVE-22191: -- Status: In Progress (was: Patch Available) > Simplify SemanticAnalyzer by removing unused code > - > > Key: HIVE-22191 > URL: https://issues.apache.org/jira/browse/HIVE-22191 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Akos Dombi >Assignee: Akos Dombi >Priority: Major > Attachments: HIVE-22191.2.patch, HIVE-22191.patch > > > Simplify {{SemanticAnalyzer}} by: > - Remove dead code > - Simplify returning statements > - Use interfaces types for parameters/fields/variables where it is > straightforward to migrate > - Make visibility stricter where it is possible > - Check logging to use parametrised logging > - Removing unnecessary keywords (e.g.: {{static}}) > - Some code part could be simplified by using Java 8 features > I think this is crucial step as this class already contains 15000+ lines of > code which is screaming for splitting into more reasonable classes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22209) Creating a materialized view with no tables should be handled more gracefully
[ https://issues.apache.org/jira/browse/HIVE-22209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Carlin updated HIVE-22209: Attachment: HIVE-22209.1.patch > Creating a materialized view with no tables should be handled more gracefully > - > > Key: HIVE-22209 > URL: https://issues.apache.org/jira/browse/HIVE-22209 > Project: Hive > Issue Type: Bug > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Steve Carlin >Priority: Minor > Attachments: HIVE-22209.1.patch, HIVE-22209.patch > > > Currently, materialized views without a table reference are not supported. > However, instead of printing a clear message about it, when a materialized > view is created without a table reference, we fail with an unclear message. > {code} > > create materialized view mv_test1 as select 5; > (...) > ERROR : FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Add request > failed : > INSERT INTO MV_TABLES_USED (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) > INFO : Completed executing > command(queryId=hive_20190916203511_b609cccf-f5e3-45dd-abfd-6e869d94e39a); > Time taken: 10.469 seconds > Error: Error while processing statement: FAILED: Execution Error, return code > 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaExcep > tion(message:Add request failed : INSERT INTO MV_TABLES_USED > (MV_CREATION_METADATA_ID,TBL_ID) VALUES (?,?) ) (state=08S01,code=1) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934329#comment-16934329 ] Ivan Suller commented on HIVE-21000: Created the Accumulo upgrade ticket: HIVE-0 > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22211) Change maven phase to generate test sources
[ https://issues.apache.org/jira/browse/HIVE-22211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22211: Attachment: HIVE-22211.01.patch > Change maven phase to generate test sources > --- > > Key: HIVE-22211 > URL: https://issues.apache.org/jira/browse/HIVE-22211 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-22211.01.patch, HIVE-22211.01.patch, > HIVE-22211.01.patch > > > Some protobuf files are generated in the wrong phase; so I get compile errors > because they are not there for eclipse... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934296#comment-16934296 ] Dawood commented on HIVE-21237: --- Failed with below error again, {noformat} [root@hive apache-hive-2.3.6-bin]# bin/schematool -dbType derby -initSchema --verbose SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/apache-hive-2.3.6-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/hadoop-3.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Metastore connection URL:jdbc:derby://hive:1527/metastore_db;create=true Metastore Connection Driver :org.apache.derby.jdbc.ClientDriver Metastore connection User: APP Starting metastore schema initialization to 2.3.0 Initialization script hive-schema-2.3.0.derby.sql Connecting to jdbc:derby://hive:1527/metastore_db;create=true Connected to: Apache Derby (version 10.15.1.3 - (1853019)) Driver: Apache Derby Network Client JDBC Driver (version 10.11.1.1 - (1616546)) Transaction isolation: TRANSACTION_READ_COMMITTED 0: jdbc:derby://hive:1527/metastore_db> !autocommit on Autocommit status: true 0: jdbc:derby://hive:1527/metastore_db> CREATE FUNCTION "APP"."NUCLEUS_ASCII" (C CHAR(1)) RETURNS INTEGER LANGUAGE JAVA PARAMETER STYLE JAVA READS SQL DATA CALLED ON NULL INPUT EXTERNAL NAME 'org.datanucleus.store.rdbms.adapter.DerbySQLFunction.ascii' Error: FUNCTION 'NUCLEUS_ASCII' already exists. (state=X0Y68,code=3) Closing: 0: jdbc:derby://hive:1527/metastore_db;create=true org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! Underlying cause: java.io.IOException : Schema script failed, errorcode 2 org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization FAILED! Metastore state would be inconsistent !! at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:590) at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:563) at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1145) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) Caused by: java.io.IOException: Schema script failed, errorcode 2 at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:980) at org.apache.hive.beeline.HiveSchemaTool.runBeeLine(HiveSchemaTool.java:959) at org.apache.hive.beeline.HiveSchemaTool.doInit(HiveSchemaTool.java:586) ... 8 more *** schemaTool failed *** {noformat} > [JDK 11] SessionState can't be initialized due to classloader problem > - > > Key: HIVE-21237 > URL: https://issues.apache.org/jira/browse/HIVE-21237 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 > Environment: JDK11, Hadoop-3, Hive 3.1.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > When I start Hive with JDK11 > {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID = > cecd9c34-d61a-44d0-9e52-a0a7d6413e49 > Exception in thread "main" java.lang.ClassCastException: class > jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class > java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and > java.net.URLClassLoader are in module java.base of loader 'bootstrap') > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410) > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386) > at > org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect
[jira] [Commented] (HIVE-22079) Post order walker for iterating over expression tree
[ https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934284#comment-16934284 ] Hive QA commented on HIVE-22079: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980800/HIVE-22079.6.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16833 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1] (batchId=171) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18662/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18662/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18662/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12980800 - PreCommit-HIVE-Build > Post order walker for iterating over expression tree > > > Key: HIVE-22079 > URL: https://issues.apache.org/jira/browse/HIVE-22079 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer, Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch, > HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, HIVE-22079.6.patch > > > Current {{DefaultGraphWalker}} is used to iterate over an expression tree. > This walker uses hash map to keep track of visited/processed nodes. If an > expression tree is large this adds significant overhead due to map lookup. > For an expression trees we can instead use post order traversal and avoid > using map. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934274#comment-16934274 ] Yuming Wang edited comment on HIVE-21237 at 9/20/19 10:17 AM: -- Please try to initialize metastore: {code:sh} bin/schematool -dbType derby -initSchema --verbose {code} was (Author: q79969786): Please try to initialize metartore: {code:sh} bin/schematool -dbType derby -initSchema --verbose {code} > [JDK 11] SessionState can't be initialized due to classloader problem > - > > Key: HIVE-21237 > URL: https://issues.apache.org/jira/browse/HIVE-21237 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 > Environment: JDK11, Hadoop-3, Hive 3.1.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > When I start Hive with JDK11 > {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID = > cecd9c34-d61a-44d0-9e52-a0a7d6413e49 > Exception in thread "main" java.lang.ClassCastException: class > jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class > java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and > java.net.URLClassLoader are in module java.base of loader 'bootstrap') > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410) > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386) > at > org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at org.apache.hadoop.util.RunJar.run(RunJar.java:323) > at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934274#comment-16934274 ] Yuming Wang commented on HIVE-21237: Please try to initialize metartore: {code:sh} bin/schematool -dbType derby -initSchema --verbose {code} > [JDK 11] SessionState can't be initialized due to classloader problem > - > > Key: HIVE-21237 > URL: https://issues.apache.org/jira/browse/HIVE-21237 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 > Environment: JDK11, Hadoop-3, Hive 3.1.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > When I start Hive with JDK11 > {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID = > cecd9c34-d61a-44d0-9e52-a0a7d6413e49 > Exception in thread "main" java.lang.ClassCastException: class > jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class > java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and > java.net.URLClassLoader are in module java.base of loader 'bootstrap') > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410) > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386) > at > org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at org.apache.hadoop.util.RunJar.run(RunJar.java:323) > at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-20508) Hive does not support user names of type "user@realm"
[ https://issues.apache.org/jira/browse/HIVE-20508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-20508: -- Fix Version/s: 4.0.0 > Hive does not support user names of type "user@realm" > - > > Key: HIVE-20508 > URL: https://issues.apache.org/jira/browse/HIVE-20508 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-20508.1.patch, HIVE-20508.2.patch, > HIVE-20508.3.patch, HIVE-20508.4.patch, HIVE-20508.5.patch, > HIVE-20508.6.patch, HIVE-20508.7.patch > > > Hive does not support user names of type "user@realm". This causes > authorization problems with Ranger for user names containing realms in > Kerberos environment. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22079) Post order walker for iterating over expression tree
[ https://issues.apache.org/jira/browse/HIVE-22079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934252#comment-16934252 ] Hive QA commented on HIVE-22079: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 26s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 50s{color} | {color:blue} ql in master has 1571 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 39s{color} | {color:red} ql: The patch generated 4 new + 104 unchanged - 1 fixed = 108 total (was 105) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 7s{color} | {color:red} ql generated 1 new + 1571 unchanged - 0 fixed = 1572 total (was 1571) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 40s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Should org.apache.hadoop.hive.ql.lib.ExpressionWalker$NodeLabeled be a _static_ inner class? At ExpressionWalker.java:inner class? At ExpressionWalker.java:[lines 47-61] | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18662/dev-support/hive-personality.sh | | git revision | master / 1278bf1 | | Default Java | 1.8.0_111 | | findbugs | v3.0.1 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18662/yetus/diff-checkstyle-ql.txt | | findbugs | http://104.198.109.242/logs//PreCommit-HIVE-Build-18662/yetus/new-findbugs-ql.html | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18662/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Post order walker for iterating over expression tree > > > Key: HIVE-22079 > URL: https://issues.apache.org/jira/browse/HIVE-22079 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer, Physical Optimizer >Affects Versions: 4.0.0 >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-22079.1.patch, HIVE-22079.2.patch, > HIVE-22079.3.patch, HIVE-22079.4.patch, HIVE-22079.5.patch, HIVE-22079.6.patch > > > Current {{DefaultGraphWalker}} is used to iterate over an expression tree. > This walker uses hash map to keep track of visited/processed nodes. If an > expression tree is large this adds significant overhead due to map lookup. > For an expression trees we can instead use post order traversal and avoid > using map. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934250#comment-16934250 ] Dawood commented on HIVE-21237: --- >From the Hive logs, {noformat} 2019-09-20T05:43:18,483 INFO [main] conf.HiveConf: Found configuration file file:/opt/apache-hive-2.3.6-bin/conf/hive-site.xml 2019-09-20T05:43:19,231 INFO [main] SessionState: Logging initialized using configuration in jar:file:/opt/apache-hive-2.3.6-bin/lib/hive-common-2.3.6.jar!/hive-log4j2.properties Async: true 2019-09-20T05:43:19,412 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2019-09-20T05:43:20,466 INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/root/3515e631-50a7-429e-9875-e1bcc777bb2e 2019-09-20T05:43:20,485 INFO [main] session.SessionState: Created local directory: /tmp/root/3515e631-50a7-429e-9875-e1bcc777bb2e 2019-09-20T05:43:20,494 INFO [main] session.SessionState: Created HDFS directory: /tmp/hive/root/3515e631-50a7-429e-9875-e1bcc777bb2e/_tmp_space.db 2019-09-20T05:43:20,512 INFO [main] conf.HiveConf: Using the default value passed in for log id: 3515e631-50a7-429e-9875-e1bcc777bb2e 2019-09-20T05:43:20,514 INFO [main] session.SessionState: Updating thread name to 3515e631-50a7-429e-9875-e1bcc777bb2e main 2019-09-20T05:43:20,516 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main] CliDriver: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. 2019-09-20T05:43:44,667 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main] conf.HiveConf: Using the default value passed in for log id: 3515e631-50a7-429e-9875-e1bcc777bb2e 2019-09-20T05:43:44,732 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main] ql.Driver: Compiling command(queryId=root_20190920054344_d4447492-7063-4d08-a7f3-9ba47b1dc636): show tables 2019-09-20T05:43:45,175 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main] metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2019-09-20T05:43:45,210 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main] metastore.ObjectStore: ObjectStore, initialize called 2019-09-20T05:43:46,221 INFO [3515e631-50a7-429e-9875-e1bcc777bb2e main] metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 2019-09-20T05:43:48,045 WARN [3515e631-50a7-429e-9875-e1bcc777bb2e main] metastore.MetaStoreDirectSql: Self-test query [select "DB_ID" from "DBS"] failed; direct SQL is disabled javax.jdo.JDODataStoreException: Error executing SQL query "select "DB_ID" from "DBS"". at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543) ~[datanucleus-api-jdo-4.2.4.jar:?] at org.datanucleus.api.jdo.JDOQuery.executeInternal(JDOQuery.java:391) ~[datanucleus-api-jdo-4.2.4.jar:?] at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:216) ~[datanucleus-api-jdo-4.2.4.jar:?] at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.runTestQuery(MetaStoreDirectSql.java:230) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.(MetaStoreDirectSql.java:144) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:410) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:342) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:303) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77) ~[hadoop-common-3.2.0.jar:?] at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137) ~[hadoop-common-3.2.0.jar:?] at org.apache.hadoop.hive.metastore.RawStoreProxy.(RawStoreProxy.java:58) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:628) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:594) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:588) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:655) ~[hive-exec-2.3.6.jar:2.3.6] at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(Hiv
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934213#comment-16934213 ] Yuming Wang commented on HIVE-21237: Please check the Hive log: \{{/tmp/${USER}/hive.log}} > [JDK 11] SessionState can't be initialized due to classloader problem > - > > Key: HIVE-21237 > URL: https://issues.apache.org/jira/browse/HIVE-21237 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 > Environment: JDK11, Hadoop-3, Hive 3.1.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > When I start Hive with JDK11 > {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID = > cecd9c34-d61a-44d0-9e52-a0a7d6413e49 > Exception in thread "main" java.lang.ClassCastException: class > jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class > java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and > java.net.URLClassLoader are in module java.base of loader 'bootstrap') > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410) > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386) > at > org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at org.apache.hadoop.util.RunJar.run(RunJar.java:323) > at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21237) [JDK 11] SessionState can't be initialized due to classloader problem
[ https://issues.apache.org/jira/browse/HIVE-21237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934203#comment-16934203 ] Dawood commented on HIVE-21237: --- Hello [~yumwang] Now I get the below error on my hive setup, {noformat} hive> show tables; FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient {noformat} Could you please help here, thx > [JDK 11] SessionState can't be initialized due to classloader problem > - > > Key: HIVE-21237 > URL: https://issues.apache.org/jira/browse/HIVE-21237 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 > Environment: JDK11, Hadoop-3, Hive 3.1.1 >Reporter: Uma Maheswara Rao G >Priority: Major > > When I start Hive with JDK11 > {{2019-02-08 22:29:51,500 INFO SessionState: Hive Session ID = > cecd9c34-d61a-44d0-9e52-a0a7d6413e49 > Exception in thread "main" java.lang.ClassCastException: class > jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class > java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and > java.net.URLClassLoader are in module java.base of loader 'bootstrap') > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:410) > at > org.apache.hadoop.hive.ql.session.SessionState.(SessionState.java:386) > at > org.apache.hadoop.hive.cli.CliSessionState.(CliSessionState.java:60) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:705) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at org.apache.hadoop.util.RunJar.run(RunJar.java:323) > at org.apache.hadoop.util.RunJar.main(RunJar.java:236)}} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Component/s: Standalone Metastore Metastore > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive, Metastore, Standalone Metastore >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Fix Version/s: 4.0.0 > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Fix For: 4.0.0 > > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-22202: Resolution: Fixed Status: Resolved (was: Patch Available) > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22202) Remove unnecessary code from ObjectStore.java
[ https://issues.apache.org/jira/browse/HIVE-22202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934200#comment-16934200 ] Laszlo Bodor commented on HIVE-22202: - pushed to master, thanks [~ghanko] for the patch! > Remove unnecessary code from ObjectStore.java > - > > Key: HIVE-22202 > URL: https://issues.apache.org/jira/browse/HIVE-22202 > Project: Hive > Issue Type: Task > Components: Hive >Reporter: Hankó Gergely >Assignee: Hankó Gergely >Priority: Minor > Attachments: HIVE-22202.1.patch, HIVE-22202.2.patch, > HIVE-22202.3.patch > > > Remove unnecessary code > * remove unnecessary/unused variables > * remove unused private methods > * remove unnecessary throws declarations > * remove unnecessary conditionals that are guaranteed to always give the > same result > * simplify collection handling > ** use member sort() instead of Collections.sort() > ** use addAll() instead of adding one-by-one in a loop > * remove Javadoc that doesn't give additional information > * use try with resource where possible > * remove unnecessary explicit type arguments > * remove redundant warning suppression -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22218) Break up DDLSemanticAnalyzer - extract Workload Management related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22218?focusedWorklogId=315518&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-315518 ] ASF GitHub Bot logged work on HIVE-22218: - Author: ASF GitHub Bot Created on: 20/Sep/19 08:06 Start Date: 20/Sep/19 08:06 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #777: HIVE-22218 Break up DDLSemanticAnalyzer - extract Workload Management… URL: https://github.com/apache/hive/pull/777 … related analyzers This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 315518) Remaining Estimate: 0h Time Spent: 10m > Break up DDLSemanticAnalyzer - extract Workload Management related analyzers > > > Key: HIVE-22218 > URL: https://issues.apache.org/jira/browse/HIVE-22218 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22218.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #5: extract all the workload management related analyzers from > DDLSemanticAnalyzer, and move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934166#comment-16934166 ] Ivan Suller commented on HIVE-21000: I don't know if I need to worry or not, but I do. It doesn't sound promising, but we will see how it'll turn out. > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934161#comment-16934161 ] David Lavati commented on HIVE-21000: - {quote}Another thing that might need investigation, is the fact that the fb303 wrapper according to THRIFT-4506 is no longer maintained in the Thrift ASF project. A high number of handlers and metastore-related scripts rely on this. {quote} Is this something we need to worry about? I'm not sure how it will play along with a newer thrift version. > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934160#comment-16934160 ] Ivan Suller commented on HIVE-21000: Not yet, but will open a ticket for Accumulo upgrade. > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-20113) Shuffle avoidance: Disable 1-1 edges for sorted shuffle
[ https://issues.apache.org/jira/browse/HIVE-20113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934158#comment-16934158 ] Hive QA commented on HIVE-20113: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12980777/HIVE-20113.10.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16803 tests executed *Failed tests:* {noformat} TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=177) [groupby_rollup_empty.q,schema_evol_orc_acidvec_table_llap_io.q,parquet_types.q,vectorization_5.q,schema_evol_stats.q,subquery_corr.q,jdbc_handler.q,column_access_stats.q,vector_between_in.q,merge1.q,cbo_rp_join.q,subquery_ANY.q,vector_coalesce_3.q,sample10_mm.q,stats_noscan_1.q,vector_partition_diff_num_cols.q,vector_decimal_10_0.q,orc_merge5.q,schema_evol_text_vec_part.q,vector_groupby_reduce.q,union8.q,lineage2.q,auto_join29.q,insert_values_orig_table.q,correlationoptimizer1.q,vector_decimal_mapjoin.q,schema_evol_undecorated.q,vectorized_case.q,vector_windowing_rank.q,materialized_view_rewrite_8.q] {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18661/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18661/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18661/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12980777 - PreCommit-HIVE-Build > Shuffle avoidance: Disable 1-1 edges for sorted shuffle > > > Key: HIVE-20113 > URL: https://issues.apache.org/jira/browse/HIVE-20113 > Project: Hive > Issue Type: Bug > Components: Tez >Reporter: Gopal V >Assignee: Vineet Garg >Priority: Major > Labels: Branch3Candidate > Attachments: HIVE-20113.1.patch, HIVE-20113.10.patch, > HIVE-20113.2.patch, HIVE-20113.3.patch, HIVE-20113.4.patch, > HIVE-20113.4.patch, HIVE-20113.5.patch, HIVE-20113.6.patch, > HIVE-20113.7.patch, HIVE-20113.8.patch, HIVE-20113.9.patch > > > The sorted shuffle avoidance can have some issues when the shuffle data gets > broken up into multiple chunks on disk. > The 1-1 edge cannot skip the tez final merge - there's no reason for 1-1 to > have a final merge at all, it should open a single compressed file and write > a single index entry. > Until the shuffle issue is resolved & a lot more testing, it is prudent to > disable the optimization for sorted shuffle edges and stop rewriting the > RS(sorted) = = = RS(sorted) into RS(sorted) = = = RS(FORWARD). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934150#comment-16934150 ] Zoltan Haindrich edited comment on HIVE-21000 at 9/20/19 7:19 AM: -- awesome :D do you have a separate ticket for the accumulo-2 upgrade? if not I think we would need one - just to make it more clear what is happening :) was (Author: kgyrtkirk): awesome :D do you have a separate ticket for the accumulo-2 upgrade? > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21000) Upgrade thrift to at least 0.10.0
[ https://issues.apache.org/jira/browse/HIVE-21000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16934150#comment-16934150 ] Zoltan Haindrich commented on HIVE-21000: - awesome :D do you have a separate ticket for the accumulo-2 upgrade? > Upgrade thrift to at least 0.10.0 > - > > Key: HIVE-21000 > URL: https://issues.apache.org/jira/browse/HIVE-21000 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Ivan Suller >Priority: Major > Attachments: HIVE-21000.01.patch, HIVE-21000.02.patch, > HIVE-21000.03.patch, HIVE-21000.04.patch, HIVE-21000.05.patch, > HIVE-21000.06.patch, HIVE-21000.07.patch, HIVE-21000.08.patch, > sampler_before.png > > > I was looking into some compile profiles for tables with lots of columns; and > it turned out that [thrift 0.9.3 is allocating a > List|https://github.com/apache/hive/blob/8e30b5e029570407d8a1db67d322a95db705750e/standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/FieldSchema.java#L348] > during every hashcode calculation; but luckily THRIFT-2877 is improving on > that - so I propose to upgrade to at least 0.10.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22205: - Status: Patch Available (was: Open) The failed tests are passing locally with hbase-2.0.0-alpha4. The prior patch was using 2.1.0 of hbase. Perhaps related to these failures. > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, > HIVE-22205.4.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22205) Upgrade zookeeper and curator versions
[ https://issues.apache.org/jira/browse/HIVE-22205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22205: - Attachment: HIVE-22205.4.patch > Upgrade zookeeper and curator versions > -- > > Key: HIVE-22205 > URL: https://issues.apache.org/jira/browse/HIVE-22205 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Minor > Attachments: HIVE-22205.2.patch, HIVE-22205.3.patch, > HIVE-22205.4.patch, HIVE-22205.patch > > > Other components like hadoop have switched to using new ZK versions. So these > jars end up in classpath for hive services and could cause issues due to > in-compatible curator versions that hive uses. > So it makes sense for hive to upgrade the ZK and curator versions to try to > keep up. -- This message was sent by Atlassian Jira (v8.3.4#803005)