[jira] [Commented] (HIVE-21693) Break up DDLTask - extract Process related operations
[ https://issues.apache.org/jira/browse/HIVE-21693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833951#comment-16833951 ] Zoltan Haindrich commented on HIVE-21693: - do we need to keep the useNewFormat in showLocks? Can't we use the "new" format only and forget the old? +1 > Break up DDLTask - extract Process related operations > - > > Key: HIVE-21693 > URL: https://issues.apache.org/jira/browse/HIVE-21693 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21693.01.patch, HIVE-21693.02.patch, > HIVE-21693.03.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #7: extract all the process related operations from the old DDLTask, and > move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21693) Break up DDLTask - extract Process related operations
[ https://issues.apache.org/jira/browse/HIVE-21693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833425#comment-16833425 ] Hive QA commented on HIVE-21693: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12967874/HIVE-21693.03.patch {color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified. {color:green}SUCCESS:{color} +1 due to 15972 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17123/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17123/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17123/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase {noformat} This message is automatically generated. ATTACHMENT ID: 12967874 - PreCommit-HIVE-Build > Break up DDLTask - extract Process related operations > - > > Key: HIVE-21693 > URL: https://issues.apache.org/jira/browse/HIVE-21693 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21693.01.patch, HIVE-21693.02.patch, > HIVE-21693.03.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #7: extract all the process related operations from the old DDLTask, and > move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21693) Break up DDLTask - extract Process related operations
[ https://issues.apache.org/jira/browse/HIVE-21693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833416#comment-16833416 ] Hive QA commented on HIVE-21693: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 31s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 23s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m 55s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} hcatalog/core in master has 28 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 47s{color} | {color:blue} itests/util in master has 46 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 27s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 50s{color} | {color:red} ql: The patch generated 2 new + 1346 unchanged - 22 fixed = 1348 total (was 1368) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 33m 41s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17123/dev-support/hive-personality.sh | | git revision | master / 341fc33 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17123/yetus/diff-checkstyle-ql.txt | | modules | C: ql hcatalog/core itests/util U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17123/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Break up DDLTask - extract Process related operations > - > > Key: HIVE-21693 > URL: https://issues.apache.org/jira/browse/HIVE-21693 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21693.01.patch, HIVE-21693.02.patch, > HIVE-21693.03.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > *
[jira] [Commented] (HIVE-21693) Break up DDLTask - extract Process related operations
[ https://issues.apache.org/jira/browse/HIVE-21693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833368#comment-16833368 ] Hive QA commented on HIVE-21693: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12967866/HIVE-21693.02.patch {color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 15972 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_abort] (batchId=46) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[lockneg4] (batchId=100) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[lockneg5] (batchId=101) org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testMergeOnTezEdges (batchId=320) org.apache.hadoop.hive.ql.parse.TestQBCompact.showCompactions (batchId=314) org.apache.hadoop.hive.ql.parse.TestQBCompact.showTxns (batchId=314) org.apache.hive.minikdc.TestSSLWithMiniKdc.testConnection (batchId=286) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17122/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17122/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17122/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.YetusPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12967866 - PreCommit-HIVE-Build > Break up DDLTask - extract Process related operations > - > > Key: HIVE-21693 > URL: https://issues.apache.org/jira/browse/HIVE-21693 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21693.01.patch, HIVE-21693.02.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each operation > * have a package for each operation group (database ddl, table ddl, etc), so > the amount of classes under a package is more manageable > * make all the requests (DDLDesc subclasses) immutable > * DDLTask should be agnostic to the actual operations > * right now let's ignore the issue of having some operations handled by > DDLTask which are not actual DDL operations (lock, unlock, desc...) > In the interim time when there are two DDLTask and DDLWork classes in the > code base the new ones in the new package are called DDLTask2 and DDLWork2 > thus avoiding the usage of fully qualified class names where both the old and > the new classes are in use. > Step #7: extract all the process related operations from the old DDLTask, and > move them under the new package. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-21693) Break up DDLTask - extract Process related operations
[ https://issues.apache.org/jira/browse/HIVE-21693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833363#comment-16833363 ] Hive QA commented on HIVE-21693: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} master Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 16s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s{color} | {color:green} master passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 21s{color} | {color:green} master passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m 3s{color} | {color:blue} ql in master has 2253 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 38s{color} | {color:blue} hcatalog/core in master has 28 extant Findbugs warnings. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 46s{color} | {color:blue} itests/util in master has 46 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s{color} | {color:green} master passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 54s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 51s{color} | {color:red} ql: The patch generated 6 new + 1343 unchanged - 22 fixed = 1349 total (was 1365) {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 33m 37s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Optional Tests | asflicense javac javadoc findbugs checkstyle compile | | uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux | | Build tool | maven | | Personality | /data/hiveptest/working/yetus_PreCommit-HIVE-Build-17122/dev-support/hive-personality.sh | | git revision | master / 341fc33 | | Default Java | 1.8.0_111 | | findbugs | v3.0.0 | | checkstyle | http://104.198.109.242/logs//PreCommit-HIVE-Build-17122/yetus/diff-checkstyle-ql.txt | | modules | C: ql hcatalog/core itests/util U: . | | Console output | http://104.198.109.242/logs//PreCommit-HIVE-Build-17122/yetus.txt | | Powered by | Apache Yetushttp://yetus.apache.org | This message was automatically generated. > Break up DDLTask - extract Process related operations > - > > Key: HIVE-21693 > URL: https://issues.apache.org/jira/browse/HIVE-21693 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.1.1 >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: refactor-ddl > Fix For: 4.0.0 > > Attachments: HIVE-21693.01.patch, HIVE-21693.02.patch > > > DDLTask is a huge class, more than 5000 lines long. The related DDLWork is > also a huge class, which has a field for each DDL operation it supports. The > goal is to refactor these in order to have everything cut into more > handleable classes under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for
[jira] [Commented] (HIVE-21693) Break up DDLTask - extract Process related operations
[ https://issues.apache.org/jira/browse/HIVE-21693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833291#comment-16833291 ] Hive QA commented on HIVE-21693: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12967862/HIVE-21693.01.patch {color:red}ERROR:{color} -1 due to build exiting with an error Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/17121/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/17121/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-17121/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests exited with: NonZeroExitCodeException Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ date '+%Y-%m-%d %T.%3N' 2019-05-05 09:31:53.401 + [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]] + export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64 + export PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m ' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m ' + export 'MAVEN_OPTS=-Xmx1g ' + MAVEN_OPTS='-Xmx1g ' + cd /data/hiveptest/working/ + tee /data/hiveptest/logs/PreCommit-HIVE-Build-17121/source-prep.txt + [[ false == \t\r\u\e ]] + mkdir -p maven ivy + [[ git = \s\v\n ]] + [[ git = \g\i\t ]] + [[ -z master ]] + [[ -d apache-github-source-source ]] + [[ ! -d apache-github-source-source/.git ]] + [[ ! -d apache-github-source-source ]] + date '+%Y-%m-%d %T.%3N' 2019-05-05 09:31:53.405 + cd apache-github-source-source + git fetch origin + git reset --hard HEAD HEAD is now at 341fc33 HIVE-14669: Have the actual error reported when a q test fails instead of having to go through the logs (Laszlo Bodor via Zoltan Haindrich) + git clean -f -d Removing ${project.basedir}/ Removing itests/${project.basedir}/ Removing standalone-metastore/metastore-server/src/gen/ + git checkout master Already on 'master' Your branch is up-to-date with 'origin/master'. + git reset --hard origin/master HEAD is now at 341fc33 HIVE-14669: Have the actual error reported when a q test fails instead of having to go through the logs (Laszlo Bodor via Zoltan Haindrich) + git merge --ff-only origin/master Already up-to-date. + date '+%Y-%m-%d %T.%3N' 2019-05-05 09:31:54.579 + rm -rf ../yetus_PreCommit-HIVE-Build-17121 + mkdir ../yetus_PreCommit-HIVE-Build-17121 + git gc + cp -R . ../yetus_PreCommit-HIVE-Build-17121 + mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-17121/yetus + patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hiveptest/working/scratch/build.patch + [[ -f /data/hiveptest/working/scratch/build.patch ]] + chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh + /data/hiveptest/working/scratch/smart-apply-patch.sh /data/hiveptest/working/scratch/build.patch Going to apply patch with: git apply -p0 + [[ maven == \m\a\v\e\n ]] + rm -rf /data/hiveptest/working/maven/org/apache/hive + mvn -B clean install -DskipTests -T 4 -q -Dmaven.repo.local=/data/hiveptest/working/maven protoc-jar: executing: [/tmp/protoc2749861970115698451.exe, --version] libprotoc 2.5.0 protoc-jar: executing: [/tmp/protoc2749861970115698451.exe, -I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore, --java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources, /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto] ANTLR Parser Generator Version 3.5.2 protoc-jar: executing: [/tmp/protoc2569106713924516175.exe, --version] libprotoc 2.5.0 ANTLR Parser Generator Version 3.5.2 Output file /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java does not exist: must build /data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g org/apache/hadoop/hive/metastore/parser/Filter.g log4j:WARN No appenders could be found for logger (DataNucleus.Persistence). log4j:WARN Please initialize the log4j system properly. DataNucleus Enhancer (version 4.1.17) for API "JDO" DataNucleus Enhancer completed with success for 41 classes. ANTLR Parser Generator Version 3.5.2 Output file