[jira] [Commented] (HIVE-21771) Support partition filter (where clause) in REPL dump command (Bootstrap Dump)
[ https://issues.apache.org/jira/browse/HIVE-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128230#comment-17128230 ] Hive QA commented on HIVE-21771: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12975123/HIVE-21771.02.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/22774/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22774/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22774/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2020-06-08 12:02:28.152 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-22774/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2020-06-08 12:02:28.155 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at d5fb71f HIVE-23605: 'Wrong FS' error during _external_tables_info creation when staging location is remote(Pravin Kumar Sinha, reviewed by Aasha Medhi) + git clean -f -d + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at d5fb71f HIVE-23605: 'Wrong FS' error during _external_tables_info creation when staging location is remote(Pravin Kumar Sinha, reviewed by Aasha Medhi) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2020-06-08 12:02:28.853 + rm -rf ../yetus_PreCommit-HIVE-Build-22774 + mkdir ../yetus_PreCommit-HIVE-Build-22774 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-22774 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22774/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Trying to apply the patch with -p0 error: a/itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/ExportTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/PartitionIterable.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ExportSemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: does not exist in index error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/repl/dump/TableExport.java: does not exist in index error: a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/common/repl/ReplScope.java: does not exist in index Trying to apply the patch with -p1 error: patch failed: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java:255 Falling back to three-way merge... Applied patch to 'itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/WarehouseInstance.java' with conflicts. error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/ExportTask.java:53 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/ExportTask.java' with conflicts. error: patch failed: ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java:580 Falling back to three-way merge... Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/exec/repl/ReplDumpTask.java' with conflicts. error: patch failed:
[jira] [Commented] (HIVE-21771) Support partition filter (where clause) in REPL dump command (Bootstrap Dump)
[ https://issues.apache.org/jira/browse/HIVE-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887819#comment-16887819 ] Hive QA commented on HIVE-21771: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12975123/HIVE-21771.02.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 16680 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeader.testHouseKeepingThreadExistence (batchId=241) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18081/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18081/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18081/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12975123 - PreCommit-HIVE-Build > Support partition filter (where clause) in REPL dump command (Bootstrap Dump) > - > > Key: HIVE-21771 > URL: https://issues.apache.org/jira/browse/HIVE-21771 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-21771.01.patch, HIVE-21771.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > *Bootstrap for managed table* > User should be allowed to execute REPL DUMP with where clause. The where > clause should support filtering out partition from dump. Format of the where > clause should be similar to *"REPL DUMP dbname from 10 where 't0' where key < > 10,'t1'* where key = 3, '(t2*)|'t3' where key > 3".* For initial version, > very basic filter condition will be supported and later the complexity will > be increased as and when required. > * From the AST generated for the where clause, extract the table information. > * Generate AST for each table. > * List the partition for each table using the AST generated for each table > using the same metastore API used by select query. > * During bootstrap load use the partition list to dump the partitions. > * During incremental dump, use the list to filter out the event. > In case of bootstrap load, all the tables of the database will be scanned and > * If table is not partitioned, then it will be dumped. > * If key provided in the filter condition for the table is not a partition > column, then dump will fail. > * If table is not mentioned in the where clause, then all partitions of the > table will be dumped. > * All the partitioned of the table satisfying the where clause will be > dumped. > *Incremental for managed table (Not part of this patch)* > In case of Incremental Dump, the events from the notification log will be > scanned and once the partition spec is extracted from the event, the > partition spec will be filtered against the condition. > * If table is not partitioned then the event will be added to the dump. > * If key mentioned is not a partition column, then dump will fail. > * If the table is not mentioned in the filter then event will be added to > the dump. > * If the event is multi partitioned, then the event will be added to the > dump. (Filtering out redundant partitions from message will be done as part > of separate task). > * If the partition spec matches the filter, then the event will be added to > the dump*.* > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21771) Support partition filter (where clause) in REPL dump command (Bootstrap Dump)
[ https://issues.apache.org/jira/browse/HIVE-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887784#comment-16887784 ] Hive QA commented on HIVE-21771: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 43s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 54s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 11s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 28s{color} | {color:blue} standalone-metastore/metastore-common in master has 31 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 3s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 36s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 34s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 22s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 44s{color} | {color:red} ql: The patch generated 10 new + 570 unchanged - 0 fixed = 580 total (was 570) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 12s{color} | {color:red} ql generated 9 new + 2241 unchanged - 9 fixed = 2250 total (was 2250) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 40m 10s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA239 be a _static_ inner class? At HiveParser.java:inner class? At HiveParser.java:[lines 48849-48862] | | | Dead store to LA31_128 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:[line 48589] | | | Dead store to LA31_130 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:[line 48602] | | | Dead store to LA31_132 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:[line 48615] | | | Dead store to LA31_134 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:[line 48628] | | | Dead store to LA31_136 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:[line 48641] | | | Dead store to LA31_138 in
[jira] [Commented] (HIVE-21771) Support partition filter (where clause) in REPL dump command (Bootstrap Dump)
[ https://issues.apache.org/jira/browse/HIVE-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886759#comment-16886759 ] Hive QA commented on HIVE-21771: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12974909/HIVE-21771.01.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 16680 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part] (batchId=52) org.apache.hive.minikdc.TestJdbcWithMiniKdc.testTokenAuth (batchId=296) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/18060/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18060/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18060/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 2 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12974909 - PreCommit-HIVE-Build > Support partition filter (where clause) in REPL dump command (Bootstrap Dump) > - > > Key: HIVE-21771 > URL: https://issues.apache.org/jira/browse/HIVE-21771 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 4.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-21771.01.patch > > > *Bootstrap for managed table* > User should be allowed to execute REPL DUMP with where clause. The where > clause should support filtering out partition from dump. Format of the where > clause should be similar to *"REPL DUMP dbname from 10 where 't0' where key < > 10,'t1'* where key = 3, '(t2*)|'t3' where key > 3".* For initial version, > very basic filter condition will be supported and later the complexity will > be increased as and when required. > * From the AST generated for the where clause, extract the table information. > * Generate AST for each table. > * List the partition for each table using the AST generated for each table > using the same metastore API used by select query. > * During bootstrap load use the partition list to dump the partitions. > * During incremental dump, use the list to filter out the event. > In case of bootstrap load, all the tables of the database will be scanned and > * If table is not partitioned, then it will be dumped. > * If key provided in the filter condition for the table is not a partition > column, then dump will fail. > * If table is not mentioned in the where clause, then all partitions of the > table will be dumped. > * All the partitioned of the table satisfying the where clause will be > dumped. > *Incremental for managed table (Not part of this patch)* > In case of Incremental Dump, the events from the notification log will be > scanned and once the partition spec is extracted from the event, the > partition spec will be filtered against the condition. > * If table is not partitioned then the event will be added to the dump. > * If key mentioned is not a partition column, then dump will fail. > * If the table is not mentioned in the filter then event will be added to > the dump. > * If the event is multi partitioned, then the event will be added to the > dump. (Filtering out redundant partitions from message will be done as part > of separate task). > * If the partition spec matches the filter, then the event will be added to > the dump*.* > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (HIVE-21771) Support partition filter (where clause) in REPL dump command (Bootstrap Dump)
[ https://issues.apache.org/jira/browse/HIVE-21771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886744#comment-16886744 ] Hive QA commented on HIVE-21771: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 39s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 6s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 22s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 9s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m 29s{color} | {color:blue} standalone-metastore/metastore-common in master has 31 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 3s{color} | {color:blue} ql in master has 2250 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 42s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 19s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 11s{color} | {color:red} standalone-metastore/metastore-common: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 46s{color} | {color:red} ql: The patch generated 16 new + 574 unchanged - 0 fixed = 590 total (was 574) {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s{color} | {color:red} itests/hive-unit: The patch generated 4 new + 4 unchanged - 0 fixed = 8 total (was 4) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 12s{color} | {color:red} ql generated 9 new + 2241 unchanged - 9 fixed = 2250 total (was 2250) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 42s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 40m 29s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:ql | | | Should org.apache.hadoop.hive.ql.parse.HiveParser$DFA239 be a _static_ inner class? At HiveParser.java:inner class? At HiveParser.java:[lines 48849-48862] | | | Dead store to LA31_128 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:[line 48589] | | | Dead store to LA31_130 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:[line 48602] | | | Dead store to LA31_132 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At HiveParser.java:[line 48615] | | | Dead store to LA31_134 in org.apache.hadoop.hive.ql.parse.HiveParser$DFA31.specialStateTransition(int, IntStream) At