[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16374512#comment-16374512 ] Zoltan Haindrich commented on HIVE-15680: - [~erwaman], [~prasanth_j]: this ticket have broken a test more than a week ago; and HIVE-18695 is not yet addressed; if the test failure will still persist on *monday*; I will *revert* this patch from the master branch > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, > HIVE-15680.6.patch, HIVE-15680.7.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341987#comment-16341987 ] Anthony Hsu commented on HIVE-15680: [~prasanth_j], [~niklaus.xiao]: thanks for the comments on the review. You guys are right – there seem to be correctness bugs in my patch; I think [~prasanth_j]'s patch to disable pushing filters for MR is the better approach for now. Thanks for fixing, [~prasanth_j]! > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Fix For: 3.0.0, 2.4.0 > > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, > HIVE-15680.6.patch, HIVE-15680.7.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341813#comment-16341813 ] Prasanth Jayachandran commented on HIVE-15680: -- Test failures are unrelated and are already happening in master > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, > HIVE-15680.6.patch, HIVE-15680.7.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340812#comment-16340812 ] Hive QA commented on HIVE-15680: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12907613/HIVE-15680.7.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 11662 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries] (batchId=238) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook] (batchId=12) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=173) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb] (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1] (batchId=170) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid] (batchId=169) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes] (batchId=162) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part] (batchId=94) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5] (batchId=121) org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=254) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=229) org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=186) org.apache.hive.hcatalog.common.TestHiveClientCache.testCloseAllClients (batchId=198) org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=232) org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=232) org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=232) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8846/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8846/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8846/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 21 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12907613 - PreCommit-HIVE-Build > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, > HIVE-15680.6.patch, HIVE-15680.7.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340726#comment-16340726 ] Hive QA commented on HIVE-15680: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Findbugs executables are not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s{color} | {color:red} ql: The patch generated 2 new + 37 unchanged - 0 fixed = 39 total (was 37) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 14s{color} | {color:red} The patch generated 6 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 12m 50s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh | | git revision | master / 26be99d | | Default Java | 1.8.0_111 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-8846/yetus/diff-checkstyle-ql.txt | | asflicense | http://104.198.109.242/logs//PreCommit-HIVE-Build-8846/yetus/patch-asflicense-problems.txt | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-8846/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, > HIVE-15680.6.patch, HIVE-15680.7.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338893#comment-16338893 ] Thejas M Nair commented on HIVE-15680: -- +1 > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, > HIVE-15680.6.patch, HIVE-15680.7.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338611#comment-16338611 ] Prasanth Jayachandran commented on HIVE-15680: -- Uploaded the correct patch. > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, > HIVE-15680.6.patch, HIVE-15680.7.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338606#comment-16338606 ] Prasanth Jayachandran commented on HIVE-15680: -- Sorry wrong patch. Will reupload again. > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, HIVE-15680.6.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338605#comment-16338605 ] Prasanth Jayachandran commented on HIVE-15680: -- Building on the test case provided by [~erwaman] in the previous patches, .7 patch disables pushing filters when execution engine is MR and when there are >1 aliases. This is safer choice for now since using MR will throw deprecation warning anyway. [~thejas]/[~gopalv] Can you please take a look? > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, > HIVE-15680.6.patch, HIVE-15680.7.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252634#comment-16252634 ] Hive QA commented on HIVE-15680: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12850112/HIVE-15680.6.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7811/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7811/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7811/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2017-11-14 22:50:57.198 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-7811/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2017-11-14 22:50:57.201 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at d295849 HIVE-18002 : add group support for pool mappings (Sergey Shelukhin, reviewed by Prasanth Jayachandran) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at d295849 HIVE-18002 : add group support for pool mappings (Sergey Shelukhin, reviewed by Prasanth Jayachandran) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2017-11-14 22:50:57.727 + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java:23 error: ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java: patch does not apply error: patch failed: serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java:117 error: serde/src/java/org/apache/hadoop/hive/serde2/ColumnProjectionUtils.java: patch does not apply The patch does not appear to apply with p0, p1, or p2 + exit 1 ' {noformat} This message is automatically generated. ATTACHMENT ID: 12850112 - PreCommit-HIVE-Build > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, HIVE-15680.6.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16251834#comment-16251834 ] Prasanth Jayachandran commented on HIVE-15680: -- [~niklaus.xiao] made some valid points in RB. Set will change order of needed columns/types. Also the case where aliases require different needed columns does not look correct. Added one more comment to RB. > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, HIVE-15680.6.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847891#comment-15847891 ] Sergey Shelukhin commented on HIVE-15680: - Submitted a patch there. You can try your patch with that patch to see if that makes tests pass here. Not sure why it triggers that path though... hopefully it doesn't somehow break projection. Although in this case it's a text table. > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, HIVE-15680.6.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847877#comment-15847877 ] Sergey Shelukhin commented on HIVE-15680: - The issue at least in some tests seems to be the continuation of HIVE-15649 that I've fixed recently. I will fix this shortly, probably today. > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, HIVE-15680.6.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847866#comment-15847866 ] Anthony Hsu commented on HIVE-15680: [~gopalv], [~sershe], [~xuefuz]: Is it possible to run the LLAP tests all in one process, so you can step through the code easily? If so, could you provide some pointers? > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch, HIVE-15680.6.patch > > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15844218#comment-15844218 ] Hive QA commented on HIVE-15680: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12849830/HIVE-15680.5.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 10468 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=135) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=136) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=137) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=138) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=139) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=140) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=141) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=142) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=145) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=148) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=155) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithAcid (batchId=255) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3244/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3244/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3244/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 27 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12849830 - PreCommit-HIVE-Build > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch, HIVE-15680.4.patch, HIVE-15680.5.patch > > > To repro: > {noformat} > set
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15842485#comment-15842485 ] Hive QA commented on HIVE-15680: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12849592/HIVE-15680.4.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 11004 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_concatenate_indexed_table] (batchId=42) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge] (batchId=25) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_2] (batchId=37) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_3] (batchId=72) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_merge_stats] (batchId=55) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[combine3] (batchId=7) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_merge_compressed] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[input22] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_4] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_6] (batchId=30) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_7] (batchId=50) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_8] (batchId=66) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_9] (batchId=76) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_16] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[truncate_column] (batchId=75) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[truncate_column_buckets] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[truncate_column_list_bucket] (batchId=78) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[truncate_column_merge] (batchId=56) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_not] (batchId=39) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[create_merge_compressed] (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1] (batchId=145) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_char_simple] (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_varchar_simple] (batchId=153) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[truncate_column_buckets] (batchId=160) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93) org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver[infer_bucket_sort_dyn_part] (batchId=83) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=223) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[create_merge_compressed] (batchId=113) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[having] (batchId=113) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[smb_mapjoin_16] (batchId=113) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_12] (batchId=113) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_not] (batchId=113) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testCombinationInputFormat (batchId=255) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorization (batchId=255) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithAcid (batchId=255) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithBuckets (batchId=255) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3210/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3210/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3210/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 38 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12849592 - PreCommit-HIVE-Build > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL:
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840217#comment-15840217 ] Hive QA commented on HIVE-15680: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12849540/HIVE-15680.3.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 27 failed/errored test(s), 10461 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=135) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=136) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=137) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=138) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=139) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=140) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=141) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=142) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=145) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=146) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=147) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=148) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=149) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=150) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=151) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=152) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=153) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=154) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=155) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_3] (batchId=93) org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver (batchId=230) org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorizationWithAcid (batchId=255) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/3199/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/3199/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-3199/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 27 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12849540 - PreCommit-HIVE-Build > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > Attachments: HIVE-15680.1.patch, HIVE-15680.2.patch, > HIVE-15680.3.patch > > > To repro: > {noformat} > set
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833117#comment-15833117 ] Hive QA commented on HIVE-15680: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12848689/HIVE-15680.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 59 failed/errored test(s), 10444 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=235) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join13] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_2] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_when_case] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_udaf] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[crtseltbl_serdeprops] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_1] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynamic_partition_skip_default] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_numeric] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4_map_skew] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join38] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_onview] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_acid] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin46] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nested_column_pruning] (batchId=31) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_decode_name] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_constant_expr] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_unquote_not] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_7] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats13] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats4] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_ppr_all] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_publisher_error_1] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_in_having] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf9] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udf_round_2] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_1] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_decimal_date] (batchId=73) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_case] (batchId=53) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_string_funcs] (batchId=53) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=134) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=135) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=136) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=137) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapCliDriver (batchId=138) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=139) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=140) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=141) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=142) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=143) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=144) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=145) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver (batchId=146)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832786#comment-15832786 ] Anthony Hsu commented on HIVE-15680: Same issue, even with explicit aliases: {noformat} hive (default)> set hive.optimize.index.filter=true; hive (default)> select * from test_table x where number = 1 > union all > select * from test_table y where number = 2; WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. Query ID = ahsu_20170120193810_ffa4adbb-e408-4505-82aa-5abeb7a5dd1c Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Job running in-process (local Hadoop) 2017-01-20 19:38:11,937 Stage-1 map = 100%, reduce = 0% Ended Job = job_local876667430_0002 MapReduce Jobs Launched: Stage-Stage-1: HDFS Read: 0 HDFS Write: 0 SUCCESS Total MapReduce CPU Time Spent: 0 msec OK 2 Time taken: 1.711 seconds, Fetched: 1 row(s) {noformat} > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832395#comment-15832395 ] Anthony Hsu commented on HIVE-15680: [~gopalv]: I only tested with MRv2. Not sure about other execution engines but I will test. > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15680) Incorrect results when hive.optimize.index.filter=true and same ORC table is referenced twice in query
[ https://issues.apache.org/jira/browse/HIVE-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832171#comment-15832171 ] Gopal V commented on HIVE-15680: [~erwaman]: is this only happening for MRv2? > Incorrect results when hive.optimize.index.filter=true and same ORC table is > referenced twice in query > -- > > Key: HIVE-15680 > URL: https://issues.apache.org/jira/browse/HIVE-15680 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu > > To repro: > {noformat} > set hive.optimize.index.filter=true; > create table test_table(number int) stored as ORC; > -- Two insertions will create two files, with one stripe each > insert into table test_table VALUES (1); > insert into table test_table VALUES (2); > -- This should and does return 2 records > select * from test_table; > -- These should and do each return 1 record > select * from test_table where number = 1; > select * from test_table where number = 2; > -- This should return 2 records but only returns 1 record > select * from test_table where number = 1 > union all > select * from test_table where number = 2; > {noformat} > What's happening is only the last predicate is being pushed down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)