[jira] [Commented] (HIVE-12916) Cleanup metastore to put unnecessary parameters in EnvironmentContext
[
https://issues.apache.org/jira/browse/HIVE-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117167#comment-15117167
]
Hive QA commented on HIVE-12916:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784273/HIVE-12916.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10016 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestMiniTezCliDriver-update_orig_table.q-mapreduce2.q-load_dyn_part3.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.ql.TestTxnCommands2.testInitiatorWithMultipleFailedCompactions
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6744/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6744/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6744/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784273 - PreCommit-HIVE-TRUNK-Build
> Cleanup metastore to put unnecessary parameters in EnvironmentContext
> -
>
> Key: HIVE-12916
> URL: https://issues.apache.org/jira/browse/HIVE-12916
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>Priority: Blocker
> Attachments: HIVE-12916.01.patch, HIVE-12916.02.patch,
> HIVE-12916.03.patch
>
>
> for example, void alter_table(1:string dbname, 2:string tbl_name, 3:Table
> new_tbl) and void alter_table_with_environment_context(1:string dbname,
> 2:string tbl_name, 3:Table new_tbl, 4:EnvironmentContext
> environment_context), we plan to keep only the latter one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12916) Cleanup metastore to put unnecessary parameters in EnvironmentContext
[ https://issues.apache.org/jira/browse/HIVE-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116268#comment-15116268 ] Sergey Shelukhin commented on HIVE-12916: - Can we keep the old methods for backward compat though? Ideally, new method should be created with all args being in a request object (and response object) that are easy to modify in the future. See e.g. AddPartitionsRequest and add_partitions_req API - the new API was added to make it extensible, and all usages of old APIs were replaced inside Hive, while the old APIs are still preserved for backward compat for external users. Perhaps we can discuss separately the removal of all old APIs in Hive 2 line, but that should be done in a more systematic manner. > Cleanup metastore to put unnecessary parameters in EnvironmentContext > - > > Key: HIVE-12916 > URL: https://issues.apache.org/jira/browse/HIVE-12916 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Blocker > Attachments: HIVE-12916.01.patch, HIVE-12916.02.patch > > > for example, void alter_table(1:string dbname, 2:string tbl_name, 3:Table > new_tbl) and void alter_table_with_environment_context(1:string dbname, > 2:string tbl_name, 3:Table new_tbl, 4:EnvironmentContext > environment_context), we plan to keep only the latter one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12916) Cleanup metastore to put unnecessary parameters in EnvironmentContext
[ https://issues.apache.org/jira/browse/HIVE-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116231#comment-15116231 ] Pengcheng Xiong commented on HIVE-12916: [~sershe], right now some configuration parameters, such as deletedata and cascading have their own functions. It has the following disadvantages: (1) They should be kept in EnvironmentContext, not as a parameter of the function. This is the design purpose of EnvironmentContext. (2) Current solution is not good to extend in the future. For example, what if you want to add another configuration, then are you going to add another function in thrift? Putting all of them in an EnvironmentContext will be good to extend. The clean up that I am proposing is exactly redirect them to on internal call, with EnvironmentContext. > Cleanup metastore to put unnecessary parameters in EnvironmentContext > - > > Key: HIVE-12916 > URL: https://issues.apache.org/jira/browse/HIVE-12916 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Blocker > Attachments: HIVE-12916.01.patch, HIVE-12916.02.patch > > > for example, void alter_table(1:string dbname, 2:string tbl_name, 3:Table > new_tbl) and void alter_table_with_environment_context(1:string dbname, > 2:string tbl_name, 3:Table new_tbl, 4:EnvironmentContext > environment_context), we plan to keep only the latter one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12916) Cleanup metastore to put unnecessary parameters in EnvironmentContext
[ https://issues.apache.org/jira/browse/HIVE-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116154#comment-15116154 ] Sergey Shelukhin commented on HIVE-12916: - Actually, why do we even want to do it on master? It breaks compat. For new APIs, we should use Request/Response pattern (see new APIs at the bottom of the file for examples). For old APIs, we should preserve them for backward compat. The proper cleanup would be to redirect them all to one internal call in metastore imho. Preliminary -1 > Cleanup metastore to put unnecessary parameters in EnvironmentContext > - > > Key: HIVE-12916 > URL: https://issues.apache.org/jira/browse/HIVE-12916 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Blocker > Attachments: HIVE-12916.01.patch, HIVE-12916.02.patch > > > for example, void alter_table(1:string dbname, 2:string tbl_name, 3:Table > new_tbl) and void alter_table_with_environment_context(1:string dbname, > 2:string tbl_name, 3:Table new_tbl, 4:EnvironmentContext > environment_context), we plan to keep only the latter one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12916) Cleanup metastore to put unnecessary parameters in EnvironmentContext
[ https://issues.apache.org/jira/browse/HIVE-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15116144#comment-15116144 ] Pengcheng Xiong commented on HIVE-12916: [~sershe], we would like this to be in 2.0.0. cc'ing [~ashutoshc] > Cleanup metastore to put unnecessary parameters in EnvironmentContext > - > > Key: HIVE-12916 > URL: https://issues.apache.org/jira/browse/HIVE-12916 > Project: Hive > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Blocker > Attachments: HIVE-12916.01.patch, HIVE-12916.02.patch > > > for example, void alter_table(1:string dbname, 2:string tbl_name, 3:Table > new_tbl) and void alter_table_with_environment_context(1:string dbname, > 2:string tbl_name, 3:Table new_tbl, 4:EnvironmentContext > environment_context), we plan to keep only the latter one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12916) Cleanup metastore to put unnecessary parameters in EnvironmentContext
[
https://issues.apache.org/jira/browse/HIVE-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114890#comment-15114890
]
Hive QA commented on HIVE-12916:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784100/HIVE-12916.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10029 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_cascade
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_invalidate_column_stats
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_import
org.apache.hadoop.hive.metastore.TestHiveMetaStorePartitionSpecs.testGetPartitionSpecs_WithAndWithoutPartitionGrouping
org.apache.hive.jdbc.TestSSL.testSSLVersion
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6731/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6731/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6731/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12784100 - PreCommit-HIVE-TRUNK-Build
> Cleanup metastore to put unnecessary parameters in EnvironmentContext
> -
>
> Key: HIVE-12916
> URL: https://issues.apache.org/jira/browse/HIVE-12916
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-12916.01.patch, HIVE-12916.02.patch
>
>
> for example, void alter_table(1:string dbname, 2:string tbl_name, 3:Table
> new_tbl) and void alter_table_with_environment_context(1:string dbname,
> 2:string tbl_name, 3:Table new_tbl, 4:EnvironmentContext
> environment_context), we plan to keep only the latter one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12916) Cleanup metastore to put unnecessary parameters in EnvironmentContext
[
https://issues.apache.org/jira/browse/HIVE-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114775#comment-15114775
]
Hive QA commented on HIVE-12916:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784097/HIVE-12916.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6730/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6730/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6730/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hcatalog-core
---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/tmp/conf
[copy] Copying 16 files to
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-hcatalog-core ---
[INFO] Compiling 38 source files to
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/test-classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitioned.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitioned.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hcatalog-core
---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-core ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/hive-hcatalog-core-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-hcatalog-core ---
[INFO]
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-hcatalog-core ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/hive-hcatalog-core-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-hcatalog-core ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/hive-hcatalog-core-2.1.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hcatalog/hive-hcatalog-core/2.1.0-SNAPSHOT/hive-hcatalog-core-2.1.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hcatalog/hive-hcatalog-core/2.1.0-SNAPSHOT/hive-hcatalog-core-2.1.0-SNAPSHOT.pom
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/hive-hcatalog-core-2.1.0-SNAPSHOT-tests.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hcatalog/hive-hcatalog-core/2.1.0-SNAPSHOT/hive-hcatalog-core-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO]
[INFO] Building Hive HCatalog Pig Adapter 2.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
hive-hcatalog-pig-adapter ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/hcatalog/hcatalog-pig-adapter/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/hcatalog/hcatalog-pig-adapter
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-hcatalog-pig-adapter ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-hcatalog-pig-adapter ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-hcatalog-pig-adapter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/hcatalog/hcatalog-pig-adapter/src/main/
[jira] [Commented] (HIVE-12916) Cleanup metastore to put unnecessary parameters in EnvironmentContext
[
https://issues.apache.org/jira/browse/HIVE-12916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15114746#comment-15114746
]
Hive QA commented on HIVE-12916:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12784085/HIVE-12916.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6729/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/6729/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-6729/
Messages:
{noformat}
This message was trimmed, see log for full details
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-hcatalog-core
---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/tmp/conf
[copy] Copying 16 files to
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-hcatalog-core ---
[INFO] Compiling 38 source files to
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/test-classes
[WARNING]
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitioned.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatPartitioned.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @ hive-hcatalog-core
---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-hcatalog-core ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/hive-hcatalog-core-2.1.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-hcatalog-core ---
[INFO]
[INFO] --- maven-jar-plugin:2.2:test-jar (default) @ hive-hcatalog-core ---
[INFO] Building jar:
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/hive-hcatalog-core-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-hcatalog-core ---
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/hive-hcatalog-core-2.1.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hcatalog/hive-hcatalog-core/2.1.0-SNAPSHOT/hive-hcatalog-core-2.1.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hcatalog/hive-hcatalog-core/2.1.0-SNAPSHOT/hive-hcatalog-core-2.1.0-SNAPSHOT.pom
[INFO] Installing
/data/hive-ptest/working/apache-github-source-source/hcatalog/core/target/hive-hcatalog-core-2.1.0-SNAPSHOT-tests.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hcatalog/hive-hcatalog-core/2.1.0-SNAPSHOT/hive-hcatalog-core-2.1.0-SNAPSHOT-tests.jar
[INFO]
[INFO]
[INFO] Building Hive HCatalog Pig Adapter 2.1.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @
hive-hcatalog-pig-adapter ---
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/hcatalog/hcatalog-pig-adapter/target
[INFO] Deleting
/data/hive-ptest/working/apache-github-source-source/hcatalog/hcatalog-pig-adapter
(includes = [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-hcatalog-pig-adapter ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-hcatalog-pig-adapter ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-hcatalog-pig-adapter ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-github-source-source/hcatalog/hcatalog-pig-adapter/src/main/
