[jira] [Commented] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919994#comment-16919994 ] Hive QA commented on HIVE-22164: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12979029/HIVE-22164.3.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 16745 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestKuduCliDriver.org.apache.hadoop.hive.cli.TestKuduCliDriver (batchId=288) org.apache.hadoop.hive.cli.TestKuduNegativeCliDriver.org.apache.hadoop.hive.cli.TestKuduNegativeCliDriver (batchId=288) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_offset_limit] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_order_null] (batchId=171) org.apache.hadoop.hive.llap.cache.TestBuddyAllocator.testMTT[2] (batchId=361) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18433/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18433/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18433/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 5 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12979029 - PreCommit-HIVE-Build > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch, HIVE-22164.2.patch, > HIVE-22164.3.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919992#comment-16919992 ] Hive QA commented on HIVE-22164: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 58s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 31s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 38s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 18s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 42s{color} | {color:blue} ql in master has 2248 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 46s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 11s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 57m 12s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18433/dev-support/hive-personality.sh | | git revision | master / 04397e5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql . itests U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18433/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch, HIVE-22164.2.patch, > HIVE-22164.3.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22164: Attachment: HIVE-22164.3.patch Status: Patch Available (was: Open) > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch, HIVE-22164.2.patch, > HIVE-22164.3.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22164: Status: Open (was: Patch Available) > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch, HIVE-22164.2.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919969#comment-16919969 ] Hive QA commented on HIVE-22164: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12979022/HIVE-22164.2.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16746 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_offset_limit] (batchId=157) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_order_null] (batchId=171) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18432/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18432/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18432/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12979022 - PreCommit-HIVE-Build > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch, HIVE-22164.2.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919953#comment-16919953 ] Hive QA commented on HIVE-22164: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 14s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 46s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 24s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 43s{color} | {color:blue} ql in master has 2248 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 50s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 23s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 15s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 62 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 56m 53s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18432/dev-support/hive-personality.sh | | git revision | master / 04397e5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | whitespace | http://104.198.109.242/logs//PreCommit-HIVE-Build-18432/yetus/whitespace-eol.txt | | modules | C: ql . itests U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18432/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch, HIVE-22164.2.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22164: Attachment: HIVE-22164.2.patch Status: Patch Available (was: In Progress) > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch, HIVE-22164.2.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22164: Status: Open (was: Patch Available) > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work started] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22164 started by Ramesh Kumar Thangarajan. --- > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919863#comment-16919863 ] Hive QA commented on HIVE-22164: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12979014/HIVE-22164.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18431/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18431/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18431/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-08-30 20:13:50.981 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-18431/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-08-30 20:13:50.984 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 04397e5 HIVE-22151 : Turn off hybrid grace hash join by default (Ashutosh Chauhan via Vineet Garg) + git clean -f -d Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 04397e5 HIVE-22151 : Turn off hybrid grace hash join by default (Ashutosh Chauhan via Vineet Garg) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-08-30 20:13:52.168 + rm -rf ../yetus_PreCommit-HIVE-Build-18431 + mkdir ../yetus_PreCommit-HIVE-Build-18431 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-18431 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-18431/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch fatal: git diff header lacks filename information when removing 0 leading pathname components (line 4) error: cannot apply binary patch to 'data/files/process_tag_meta/source_site=BIR/part-0-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000' without full index line Falling back to three-way merge... error: cannot apply binary patch to 'data/files/process_tag_meta/source_site=BIR/part-0-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000' without full index line error: data/files/process_tag_meta/source_site=BIR/part-0-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000: patch does not apply error: cannot apply binary patch to 'data/files/process_tag_meta/source_site=BIR/part-1-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000' without full index line Falling back to three-way merge... error: cannot apply binary patch to 'data/files/process_tag_meta/source_site=BIR/part-1-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000' without full index line error: data/files/process_tag_meta/source_site=BIR/part-1-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000: patch does not apply error: cannot apply binary patch to 'data/files/process_tag_meta/source_site=BIR/part-2-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000' without full index line Falling back to three-way merge... error: cannot apply binary patch to 'data/files/process_tag_meta/source_site=BIR/part-2-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000' without full index line error: data/files/process_tag_meta/source_site=BIR/part-2-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000: patch does not apply error: cannot apply binary patch to 'data/files/process_tag_meta/source_site=BIR/part-3-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000' without full index line Falling back to three-way merge... error: cannot apply binary patch to 'data/files/process_tag_meta/source_site=BIR/part-3-516c1ef3-a2a9-4ef1-9e42-6a7d1311e666.c000' without full index line error:
[jira] [Commented] (HIVE-22161) UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919862#comment-16919862 ] Hive QA commented on HIVE-22161: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12979008/HIVE-22161.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16745 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18430/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18430/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18430/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12979008 - PreCommit-HIVE-Build > UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType > class > - > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-22161.1.patch > > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. > https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22164: Attachment: HIVE-22164.1.patch Status: Patch Available (was: In Progress) > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22164.1.patch > > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Work started] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22164 started by Ramesh Kumar Thangarajan. --- > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HIVE-22164) Vectorized Limit operator returns wrong number of results with offset
[ https://issues.apache.org/jira/browse/HIVE-22164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan reassigned HIVE-22164: --- Assignee: Ramesh Kumar Thangarajan > Vectorized Limit operator returns wrong number of results with offset > - > > Key: HIVE-22164 > URL: https://issues.apache.org/jira/browse/HIVE-22164 > Project: Hive > Issue Type: Bug > Components: Hive, llap, Vectorization >Affects Versions: 4.0.0 >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > Vectorized Limit operator returns wrong number of results with offset -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22161) UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919834#comment-16919834 ] Hive QA commented on HIVE-22161: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 30s{color} | {color:blue} common in master has 61 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 11m 46s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18430/dev-support/hive-personality.sh | | git revision | master / 04397e5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: common U: common | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18430/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType > class > - > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-22161.1.patch > > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. > https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22163) CBO: Enabling CBO turns on stats estimation, even when the estimation is disabled
[ https://issues.apache.org/jira/browse/HIVE-22163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-22163: --- Component/s: CBO > CBO: Enabling CBO turns on stats estimation, even when the estimation is > disabled > - > > Key: HIVE-22163 > URL: https://issues.apache.org/jira/browse/HIVE-22163 > Project: Hive > Issue Type: Bug > Components: CBO >Reporter: Gopal V >Priority: Major > > {code} > create table claims(claim_rec_id bigint, claim_invoice_num string, typ_c int); > alter table claims update statistics set > ('numRows'='1154941534','rawDataSize'='1135307527922'); > set hive.stats.estimate=false; > explain extended select count(1) from claims where typ_c=3; > set hive.stats.ndv.estimate.percent=5e-7; > explain extended select count(1) from claims where typ_c=3; > {code} > Expecting the standard /2 for the single filter, but we instead get 5 rows. > {code} > 'Map Operator Tree:' > 'TableScan' > ' alias: claims' > ' filterExpr: (typ_c = 3) (type: boolean)' > ' Statistics: Num rows: 1154941534 Data size: 4388777832 > Basic stats: COMPLETE Column stats: NONE' > ' GatherStats: false' > ' Filter Operator' > 'isSamplingPred: false' > 'predicate: (typ_c = 3) (type: boolean)' > 'Statistics: Num rows: 5 Data size: 19 Basic stats: > COMPLETE Column stats: NONE' > {code} > The estimation is in effect, as changing the estimate.percent changes this. > {code} > ' filterExpr: (typ_c = 3) (type: boolean)' > ' Statistics: Num rows: 1154941534 Data size: 4388777832 > Basic stats: COMPLETE Column stats: NONE' > ' GatherStats: false' > ' Filter Operator' > 'isSamplingPred: false' > 'predicate: (typ_c = 3) (type: boolean)' > 'Statistics: Num rows: 230988307 Data size: 877755567 > Basic stats: COMPLETE Column stats: NONE' > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919818#comment-16919818 ] Gopal V commented on HIVE-22162: LGTM - +1 tests-pending > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22161) UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919814#comment-16919814 ] Ashutosh Chauhan commented on HIVE-22161: - +1 pending tests. > UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType > class > - > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-22161.1.patch > > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. > https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22161) UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919810#comment-16919810 ] Gopal V commented on HIVE-22161: This was a tiny bit worse than I thought - every single UDF was synchronizing around the same class objet, UDFType.class > UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType > class > - > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-22161.1.patch > > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. > https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22161) UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-22161: --- Status: Patch Available (was: Open) > UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType > class > - > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-22161.1.patch > > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. > https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22161) UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-22161: --- Attachment: HIVE-22161.1.patch > UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType > class > - > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > Attachments: HIVE-22161.1.patch > > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. > https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22161) UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-22161: --- Summary: UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType class (was: UDF: FunctionRegistry should cache statefulness of UDF) > UDF: FunctionRegistry synchronizes on org.apache.hadoop.hive.ql.udf.UDFType > class > - > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. > https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HIVE-22161) UDF: FunctionRegistry should cache statefulness of UDF
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V reassigned HIVE-22161: -- Assignee: Gopal V > UDF: FunctionRegistry should cache statefulness of UDF > -- > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22161) UDF: FunctionRegistry should cache statefulness of UDF
[ https://issues.apache.org/jira/browse/HIVE-22161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-22161: --- Description: There's a hidden synchronization across threads when looking up isStateful and isDeterministic. https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 {code} // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 public static T getAnnotation(Class clazz, Class annotationClass) { synchronized (annotationClass) { return clazz.getAnnotation(annotationClass); } } {code} This is serializing multiple threads initializing UDFs (or checking them during compilation) & also being locked across threads for each instance of GenericUDFOpEqual in the specific scenario. https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ was: There's a hidden synchronization across threads when looking up isStateful and isDeterministic. https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 {code} // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 public static T getAnnotation(Class clazz, Class annotationClass) { synchronized (annotationClass) { return clazz.getAnnotation(annotationClass); } } {code} This is serializing multiple threads initializing UDFs (or checking them during compilation) & also being locked across threads for each instance of GenericUDFOpEqual in the specific scenario. > UDF: FunctionRegistry should cache statefulness of UDF > -- > > Key: HIVE-22161 > URL: https://issues.apache.org/jira/browse/HIVE-22161 > Project: Hive > Issue Type: Bug > Components: UDF >Reporter: Gopal V >Assignee: Gopal V >Priority: Major > > There's a hidden synchronization across threads when looking up isStateful > and isDeterministic. > https://github.com/apache/hive/blob/master/common/src/java/org/apache/hive/common/util/AnnotationUtils.java#L27 > {code} > // to avoid https://bugs.openjdk.java.net/browse/JDK-7122142 > public static T getAnnotation(Class clazz, > Class annotationClass) { > synchronized (annotationClass) { > return clazz.getAnnotation(annotationClass); > } > } > {code} > This is serializing multiple threads initializing UDFs (or checking them > during compilation) & also being locked across threads for each instance of > GenericUDFOpEqual in the specific scenario. > https://bugs.openjdk.java.net/browse/JDK-7122142 is fixed in jdk8+ -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919603#comment-16919603 ] Hive QA commented on HIVE-22162: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12978949/HIVE-22162.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16746 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[materialized_view_create_acid] (batchId=72) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18429/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18429/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18429/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12978949 - PreCommit-HIVE-Build > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919565#comment-16919565 ] Hive QA commented on HIVE-22162: | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 30s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 50s{color} | {color:blue} ql in master has 2248 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 51s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18429/dev-support/hive-personality.sh | | git revision | master / 04397e5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | modules | C: ql U: ql | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18429/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction
[jira] [Commented] (HIVE-21750) INSERT OVERWRITE with empty result set does not clear transactional table
[ https://issues.apache.org/jira/browse/HIVE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919529#comment-16919529 ] Zoltan Haindrich commented on HIVE-21750: - I think this was fixed by [~isuller]'s patches - HIVE-21784 and the other 2 > INSERT OVERWRITE with empty result set does not clear transactional table > - > > Key: HIVE-21750 > URL: https://issues.apache.org/jira/browse/HIVE-21750 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 4.0.0 >Reporter: Todd Lipcon >Priority: Critical > > The following query: > {code} > INSERT OVERWRITE TABLE t SELECT 1 WHERE FALSE > {code} > should serve to truncate a table by producing an empty base data directory. > In fact no new base directory is created, so the table is not cleared. (at > least with an insert_only table, I didn't test full-ACID) > This bug does not seem to happen with non-transactional tables. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22150) HS2 allows setting system properties
[ https://issues.apache.org/jira/browse/HIVE-22150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919444#comment-16919444 ] Hive QA commented on HIVE-22150: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12978932/HIVE-22150.patch.2 {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 16745 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18428/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18428/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18428/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12978932 - PreCommit-HIVE-Build > HS2 allows setting system properties > > > Key: HIVE-22150 > URL: https://issues.apache.org/jira/browse/HIVE-22150 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Craig Condit >Assignee: Hui An >Priority: Major > Attachments: HIVE-22150.patch.1, HIVE-22150.patch.2 > > > HiveServer2 currently allows setting system properties, which is a problem > when used in a multi-user environment. > Connecting via beeline and executing the following demonstrates the issue: > {noformat} > 0: jdbc:hive2://serv1000.example.com:2181,serv> SET system:java.io.tmpdir; > +-+ > | set | > +-+ > | system:java.io.tmpdir=/tmp | > +-+ > 1 row selected (0.018 seconds) > 0: jdbc:hive2://serv1000.example.com:2181,serv> SET > system:java.io.tmpdir=/tmp/attacker-dir; > No rows affected (0.013 seconds) > 0: jdbc:hive2://serv1000.example.com:2181,serv> SET system:java.io.tmpdir; > +--+ > | set| > +--+ > | system:java.io.tmpdir=/tmp/attacker-dir | > +--+ > 1 row selected (0.019 seconds) > {noformat} > Any changes persist until HS2 is restarted, and affect all connected users. > At the very least, this is a denial-of-service vector (verified by setting > line.separator to a random string). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22150) HS2 allows setting system properties
[ https://issues.apache.org/jira/browse/HIVE-22150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919405#comment-16919405 ] Hive QA commented on HIVE-22150: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 58s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 17s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 19s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 31s{color} | {color:blue} common in master has 61 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 53s{color} | {color:blue} ql in master has 2248 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 29s{color} | {color:blue} beeline in master has 44 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 26s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 37s{color} | {color:red} ql: The patch generated 1 new + 30 unchanged - 0 fixed = 31 total (was 30) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 48s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 35m 39s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-18428/dev-support/hive-personality.sh | | git revision | master / 04397e5 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-18428/yetus/diff-checkstyle-ql.txt | | modules | C: common ql beeline itests/hive-unit U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-18428/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > HS2 allows setting system properties > > > Key: HIVE-22150 > URL: https://issues.apache.org/jira/browse/HIVE-22150 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Affects Versions: 3.1.1 >Reporter: Craig Condit >Assignee: Hui An >Priority: Major > Attachments: HIVE-22150.patch.1, HIVE-22150.patch.2 > > > HiveServer2 currently allows setting system properties, which is a problem > when used in a multi-user environment. > Connecting via beeline and executing the following demonstrates the issue: > {noformat} > 0: jdbc:hive2://serv1000.example.com:2181,serv> SET system:java.io.tmpdir; >
[jira] [Commented] (HIVE-21002) TIMESTAMP - Backwards incompatible change: Hive 3.1 reads back Avro and Parquet timestamps written by Hive 2.x incorrectly
[ https://issues.apache.org/jira/browse/HIVE-21002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919389#comment-16919389 ] Piotr Findeisen commented on HIVE-21002: [~klcopp] [~zi] this issue explicitly talks about Avro and Parquet, whereas the same problem applies also to "RCBinary" ({{ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe' STORED AS RCFILE;}}). Has this been addressed too, or should I create a new issue? > TIMESTAMP - Backwards incompatible change: Hive 3.1 reads back Avro and > Parquet timestamps written by Hive 2.x incorrectly > -- > > Key: HIVE-21002 > URL: https://issues.apache.org/jira/browse/HIVE-21002 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.0, 3.1.1 >Reporter: Zoltan Ivanfi >Priority: Major > > Hive 3.1 reads back Avro and Parquet timestamps written by Hive 2.x > incorrectly. As an example session to demonstrate this problem, create a > dataset using Hive version 2.x in America/Los_Angeles: > {code:sql} > hive> create table ts_‹format› (ts timestamp) stored as ‹format›; > hive> insert into ts_‹format› values (*‘2018-01-01 00:00:00.000’*); > {code} > Querying this table by issuing > {code:sql} > hive> select * from ts_‹format›; > {code} > from different time zones using different versions of Hive and different > storage formats gives the following results: > |‹format›|Writer time zone (in Hive 2.x)|Reader time zone|Result in Hive 2.x > reader|Result in Hive 3.1 reader| > |Avro and Parquet|America/Los_Angeles|America/Los_Angeles|2018-01-01 > *00*:00:00.0|2018-01-01 *08*:00:00.0| > |Avro and Parquet|America/Los_Angeles|Europe/Paris|2018-01-01 > *09*:00:00.0|2018-01-01 *08*:00:00.0| > |Textfile and ORC|America/Los_Angeles|America/Los_Angeles|2018-01-01 > 00:00:00.0|2018-01-01 00:00:00.0| > |Textfile and ORC|America/Los_Angeles|Europe/Paris|2018-01-01 > 00:00:00.0|2018-01-01 00:00:00.0| > *Hive 3.1 clearly gives different results than Hive 2.x for timestamps stored > in Avro and Parquet formats.* Apache ORC behaviour has not changed because it > was modified to adjust timestamps to retain backwards compatibility. Textfile > behaviour has not changed, because its processing involves parsing and > formatting instead of proper serializing and deserializing, so they > inherently had LocalDateTime semantics even in Hive 2.x. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Attachment: HIVE-22162.1.patch > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Status: Patch Available (was: Open) > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22162.1.patch > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22088) Dynamic partition insert problem on table with "=" in location path spec
[ https://issues.apache.org/jira/browse/HIVE-22088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919293#comment-16919293 ] Hui An commented on HIVE-22088: --- What does these two failed tests mean? I opened test results and there is no failed. > Dynamic partition insert problem on table with "=" in location path spec > > > Key: HIVE-22088 > URL: https://issues.apache.org/jira/browse/HIVE-22088 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0, 2.3.4 > Environment: Hive 2.6.0.10-2 Executing on Tez. > OS: Ubuntu 16.04.4 LTS > Config settings used: > SET hive.exec.dynamic.partition=true; > SET hive.exec.dynamic.partition.mode=nonstrict; >Reporter: Puneet Khatod >Assignee: Hui An >Priority: Major > Attachments: HIVE-22088.patch.1, HIVE-22088.patch.2 > > > If external table location spec has a '=' sign (coincidentally partition > specifier) in it, then dynamic partition loading fails. > *Use cases:* > Quite often the same data is used in different contexts by creating different > external tables on top of the data. Many times the tables have different > partition depths depending on how data is organized. > Like in below example, there are individual customer specific tables and > queries/jobs to insert data partitioned by type. And there is another table > to give the consolidated data view of all the customers, thus have two level > partition customer and type. > The job to insert customer specific data into customer specific table fails > if we use dynamic partitioning. Static partition insert on same table works > fine though. > *Replication:* > To replicate following simple setup could be done. Below execution is on > 'Tez'. > *Source table**-* > CREATE EXTERNAL TABLE temp_dummy_table > (id STRING, type STRING) > ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS TEXTFILE > LOCATION '/home/source/'; > > *Destination Table-* > CREATE EXTERNAL TABLE temp_dummy_dest_table > (id STRING) > PARTITIONED BY (type string) > ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' LINES TERMINATED BY '\n' > STORED AS TEXTFILE > LOCATION '/home/destination/{color:#ff}customer=abc{color}/'; > > *Insert into destination-* > insert overwrite table temp_dummy_dest_table partition (type) > select i.id as id, i.type as type > from temp_dummy_table i > where i.type in ('type1','type2'); > > *Log and Error Msgs on CLI*- > Loading data to table temp_dummy_dest_table partition (type=null) > Failed with exception Partition spec \{type=type1, customer=abc} contains > non-partition columns > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.MoveTask > > *Possible resolution:* > The dynamic partitioning should consider only those partition specs which are > under the defined table root/base path. If the path itself has partition > style format (customer=abc in above example) then that should not be > considered as partition as it is outside the scope of the table. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa updated HIVE-22162: -- Description: {code} SET hive.support.concurrency=true; SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; SET metastore.strict.managed.tables=true; SET hive.default.fileformat=textfile; SET hive.default.fileformat.managed=orc; SET metastore.create.as.acid=true; CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite AS SELECT a, b, c FROM cmv_basetable_n4; DESCRIBE FORMATTED cmv_mat_view_n4; {code} {code} POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 ... Table Type: MATERIALIZED_VIEW Table Parameters: COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} bucketing_version 2 numFiles1 numRows 5 rawDataSize 1025 totalSize 509 {code} Missing table parameter {code} transaction = true {code} cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] was: {code} SET hive.support.concurrency=true; SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; SET metastore.strict.managed.tables=true; SET hive.default.fileformat=textfile; SET hive.default.fileformat.managed=orc; SET metastore.create.as.acid=true; CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite AS SELECT a, b, c FROM cmv_basetable_n4; DESCRIBE FORMATTED cmv_mat_view_n4; {code} {code} POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 ... Table Type: MATERIALIZED_VIEW Table Parameters: COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} bucketing_version 2 numFiles1 numRows 5 rawDataSize 1025 totalSize 509 {code} Missing table parameter {code} transaction = true {code} > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} > cc.: [~ashutoshc], [~gopalv], [~jcamachorodriguez] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HIVE-22162) MVs are not using ACID tables.
[ https://issues.apache.org/jira/browse/HIVE-22162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-22162: - > MVs are not using ACID tables. > -- > > Key: HIVE-22162 > URL: https://issues.apache.org/jira/browse/HIVE-22162 > Project: Hive > Issue Type: Bug > Components: Materialized views >Affects Versions: 3.1.2 >Reporter: Krisztian Kasa >Assignee: Krisztian Kasa >Priority: Major > Fix For: 4.0.0 > > > {code} > SET hive.support.concurrency=true; > SET hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager; > SET metastore.strict.managed.tables=true; > SET hive.default.fileformat=textfile; > SET hive.default.fileformat.managed=orc; > SET metastore.create.as.acid=true; > CREATE TABLE cmv_basetable_n4 (a int, b varchar(256), c decimal(10,2)); > INSERT INTO cmv_basetable_n4 VALUES (1, 'alfred', 10.30),(2, 'bob', 3.14),(2, > 'bonnie', 172342.2),(3, 'calvin', 978.76),(3, 'charlie', 9.8); > CREATE MATERIALIZED VIEW cmv_mat_view_n4 disable rewrite > AS SELECT a, b, c FROM cmv_basetable_n4; > DESCRIBE FORMATTED cmv_mat_view_n4; > {code} > {code} > POSTHOOK: query: DESCRIBE FORMATTED cmv_mat_view_n4 > ... > Table Type: MATERIALIZED_VIEW > Table Parameters: > COLUMN_STATS_ACCURATE > {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\",\"b\":\"true\",\"c\":\"true\"}} > bucketing_version 2 > numFiles1 > numRows 5 > rawDataSize 1025 > totalSize 509 > {code} > Missing table parameter > {code} > transaction = true > {code} -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.
[ https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919268#comment-16919268 ] Hive QA commented on HIVE-22158: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12978922/HIVE-22158.1.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18427/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18427/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18427/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Tests exited with: Exception: Patch URL https://issues.apache.org/jira/secure/attachment/12978922/HIVE-22158.1.patch was found in seen patch url's cache and a test was probably run already on it. Aborting... {noformat} This message is automatically generated. ATTACHMENT ID: 12978922 - PreCommit-HIVE-Build > HMS Translation layer - Disallow non-ACID MANAGED tables. > - > > Key: HIVE-22158 > URL: https://issues.apache.org/jira/browse/HIVE-22158 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22158.1.patch > > > In the recent commits, we have allowed non-ACID MANAGED tables to be created > by clients that have some form of ACID WRITE capabilities. > I think it would make sense to disallow this entirely. MANAGED tables should > be ACID tables only. -- This message was sent by Atlassian Jira (v8.3.2#803003)