[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14967419#comment-14967419 ] Roshan Naik commented on HIVE-9582: --- In this patch, HCatUtil.getHiveMetastoreClient() uses double checked locking pattern to implement singleton, which is a broken pattern. Created HIVE-12221 > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch, > HIVE-9582.8.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530266#comment-14530266 ] Sushanth Sowmyan commented on HIVE-9582: Committed to master and branch-1.2. Thanks Thirvel & Thejas! > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch, > HIVE-9582.8.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530234#comment-14530234 ] Sushanth Sowmyan commented on HIVE-9582: The test failures don't seem related to this patch. Will go ahead and commit. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch, > HIVE-9582.8.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[
https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530169#comment-14530169
]
Hive QA commented on HIVE-9582:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12730401/HIVE-9582.8.patch
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 8900 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_parts
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_unencrypted_tbl
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_join_with_different_encryption_keys
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_load_data_to_encrypted_tables
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_select_read_only_encrypted_tbl
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_disallow_transform
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_droppartition
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_sba_drop_table
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_authorization_uri_alterpart_loc
org.apache.hadoop.hive.ql.security.TestStorageBasedClientSideAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropDatabase
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropPartition
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropTable
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationDrops.testDropView
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProvider.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationProviderWithACL.testSimplePrivileges
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadDbSuccess
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableFailure
org.apache.hadoop.hive.ql.security.TestStorageBasedMetastoreAuthorizationReads.testReadTableSuccess
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessing
org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.TestSQLStdHiveAccessControllerHS2.testConfigProcessingCustomSetWhitelistAppend
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3751/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3751/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3751/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12730401 - PreCommit-HIVE-TRUNK-Build
> HCatalog should use IMetaStoreClient interface
> --
>
> Key: HIVE-9582
> URL: https://issues.apache.org/jira/browse/HIVE-9582
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog, Metastore
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Labels: hcatalog, metastore, rolling_upgrade
> Fix For: 1.2.0
>
> Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch,
> HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch,
> HIVE-9582.8.patch, HIVE-9583.1.patch
>
>
> Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy.
> Hence during a failure, the client retries and possibly succeeds. But
> HCatalog has long been using HiveMetaStoreClient directly and hence failures
> are costly, especially if they are during the commit stage of a job. Its also
> not possible to do rolling upgrade of MetaStore Server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528182#comment-14528182 ] Thiruvel Thirumoolan commented on HIVE-9582: Yeah, same patch. Thanks! > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch, > HIVE-9582.8.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528162#comment-14528162 ] Sushanth Sowmyan commented on HIVE-9582: No worries - was it the same patch effectively? :) Or did you have further updates? > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch, > HIVE-9582.8.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528155#comment-14528155 ] Thiruvel Thirumoolan commented on HIVE-9582: Removed my patch. Sorry [~sushanth], was hoping to upload a patch to have precommit tests by tomm morning. Didn't realize I would clash. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch, > HIVE-9582.8.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528140#comment-14528140 ] Sushanth Sowmyan commented on HIVE-9582: Looks like the patch had a merge conflict as a result of HIVE-10564 being committed in. Uploading new patch. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch, > HIVE-9582.8.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[
https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14528094#comment-14528094
]
Hive QA commented on HIVE-9582:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12729624/HIVE-9582.7.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3728/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3728/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3728/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-3728/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9803344 HIVE-10485: Create md5 UDF (Alex Pivovarov via Jason
Dere)
+ git clean -f -d
+ git checkout master
Already on 'master'
+ git reset --hard origin/master
HEAD is now at 9803344 HIVE-10485: Create md5 UDF (Alex Pivovarov via Jason
Dere)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12729624 - PreCommit-HIVE-TRUNK-Build
> HCatalog should use IMetaStoreClient interface
> --
>
> Key: HIVE-9582
> URL: https://issues.apache.org/jira/browse/HIVE-9582
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog, Metastore
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Labels: hcatalog, metastore, rolling_upgrade
> Fix For: 1.2.0
>
> Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch,
> HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch,
> HIVE-9583.1.patch
>
>
> Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy.
> Hence during a failure, the client retries and possibly succeeds. But
> HCatalog has long been using HiveMetaStoreClient directly and hence failures
> are costly, especially if they are during the commit stage of a job. Its also
> not possible to do rolling upgrade of MetaStore Server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14527295#comment-14527295 ] Sushanth Sowmyan commented on HIVE-9582: Looks like we lost a bunch of queued tests in the precommit queue. Resubmitting. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 1.2.0 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9582.7.patch, > HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[
https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14522183#comment-14522183
]
Thejas M Nair commented on HIVE-9582:
-
Looks good . +1 pending tests.
Just a minor nit - Regarding the deprecation annotation, it would be useful to
add the alternate function as part of deprecation message (its much quicker to
parse than entire function description).
Example -
@deprecated use {@link #new()} instead.
> HCatalog should use IMetaStoreClient interface
> --
>
> Key: HIVE-9582
> URL: https://issues.apache.org/jira/browse/HIVE-9582
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog, Metastore
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Labels: hcatalog, metastore, rolling_upgrade
> Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch,
> HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9582.6.patch, HIVE-9583.1.patch
>
>
> Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy.
> Hence during a failure, the client retries and possibly succeeds. But
> HCatalog has long been using HiveMetaStoreClient directly and hence failures
> are costly, especially if they are during the commit stage of a job. Its also
> not possible to do rolling upgrade of MetaStore Server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[
https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381910#comment-14381910
]
Hive QA commented on HIVE-9582:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12707334/HIVE-9582.5.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 8347 tests executed
*Failed tests:*
{noformat}
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.testSparkQuery
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3166/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3166/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3166/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12707334 - PreCommit-HIVE-TRUNK-Build
> HCatalog should use IMetaStoreClient interface
> --
>
> Key: HIVE-9582
> URL: https://issues.apache.org/jira/browse/HIVE-9582
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog, Metastore
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Labels: hcatalog, metastore, rolling_upgrade
> Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch,
> HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch
>
>
> Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy.
> Hence during a failure, the client retries and possibly succeeds. But
> HCatalog has long been using HiveMetaStoreClient directly and hence failures
> are costly, especially if they are during the commit stage of a job. Its also
> not possible to do rolling upgrade of MetaStore Server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381040#comment-14381040 ] Sushanth Sowmyan commented on HIVE-9582: Admittedly, I also agree that we should slowly push people to use HCatClient, and away from HiveMetaStoreClient for our interface purposes, and so, eventually, we should internalize HCatUtil. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381036#comment-14381036 ] Sushanth Sowmyan commented on HIVE-9582: I think that was the goal behind InternalUtil, while HCatUtil was intended to be public-ish-facing. And even if there are functions in HCatUtil that might still be worth putting behind a InterfaceAudience.Private wall, getHiveClient, closeHiveClientQuietly and getHiveConf were intended to be public. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381023#comment-14381023 ] Thiruvel Thirumoolan commented on HIVE-9582: Thanks [~sushanth] for your time and inputs. Much appreciated. I was of the opinion getHiveClient() was internal and not supposed to be used outside (yeah, we have lot of public methods). I asked Oozie team at Yahoo! to use HCatClient going forward, I am not sure those changes are in trunk. I didn't know falcon/sqoop might be using it. I will change as you suggested and update another patch. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[
https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381017#comment-14381017
]
Mithun Radhakrishnan commented on HIVE-9582:
I wonder, should {{HCatUtils}} be package-protected?
> HCatalog should use IMetaStoreClient interface
> --
>
> Key: HIVE-9582
> URL: https://issues.apache.org/jira/browse/HIVE-9582
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog, Metastore
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Labels: hcatalog, metastore, rolling_upgrade
> Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch,
> HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch
>
>
> Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy.
> Hence during a failure, the client retries and possibly succeeds. But
> HCatalog has long been using HiveMetaStoreClient directly and hence failures
> are costly, especially if they are during the commit stage of a job. Its also
> not possible to do rolling upgrade of MetaStore Server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381007#comment-14381007 ] Sushanth Sowmyan commented on HIVE-9582: +cc [~venkatnrangan] > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9582.5.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380812#comment-14380812 ] Sushanth Sowmyan commented on HIVE-9582: I've gone through this patch, and am +1 on the intent of the .4.patch. It did not apply cleanly on trunk, however, so I've rebased the patch slightly to make it patch. I'm uploading the .5.patch to reflect this change so we can get one more unit test run complete with the latest patch. That said, I do have one point of concern. Changing HCatUtil.getHiveClient to return IMetaStoreClient instead of a HMSC can potentially break oozie, falcon and sqoop compilation, not to mention other external user code that uses that method to get a HiveMetaStoreClient. I agree that it should have been IMSC all along, rather than HMSC, but it is now published interface. We could solve this as follows: a) create a new function getMetaStoreClient() which returns a IMSC. Change all code in HCat to refer to this function instead of the current getHiveClient() b) retain the existing getHiveClient() and have it continue to return a HMSC, and mark it for deprecation in 2 releases - i.e. deprecated in 1.2, gone in 1.3. I worried if this was possible given that you change the Cache class, but luckily, your underlying implementation of ICacheableMetaStoreClient is a CacheableHiveMetaStoreClient which is a HMSC, so this is still possible with minimal changes. Once we do this, we should communicate to oozie/falcon/sqoop developers to change their usage of this function to the more generic one. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353949#comment-14353949 ] Thiruvel Thirumoolan commented on HIVE-9582: Thanks Sushanth. Thejas looked at the patch and gave a first level review. I have addressed his comments and uploaded the patch. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 0.14.1 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14349614#comment-14349614 ] Sushanth Sowmyan commented on HIVE-9582: Hi Thiruvel, sorry for taking so long to get to this, I'm looking through this now. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 0.14.1 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347785#comment-14347785 ] Thiruvel Thirumoolan commented on HIVE-9582: Updated review board with first set of comments from [~thejas] addressed. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 0.14.1 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9582.4.patch, HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[
https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347544#comment-14347544
]
Hive QA commented on HIVE-9582:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12702434/HIVE-9582.4.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7589 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2944/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2944/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2944/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12702434 - PreCommit-HIVE-TRUNK-Build
> HCatalog should use IMetaStoreClient interface
> --
>
> Key: HIVE-9582
> URL: https://issues.apache.org/jira/browse/HIVE-9582
> Project: Hive
> Issue Type: Sub-task
> Components: HCatalog, Metastore
>Affects Versions: 0.14.0, 0.13.1
>Reporter: Thiruvel Thirumoolan
>Assignee: Thiruvel Thirumoolan
> Labels: hcatalog, metastore, rolling_upgrade
> Fix For: 0.14.1
>
> Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch,
> HIVE-9582.4.patch, HIVE-9583.1.patch
>
>
> Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy.
> Hence during a failure, the client retries and possibly succeeds. But
> HCatalog has long been using HiveMetaStoreClient directly and hence failures
> are costly, especially if they are during the commit stage of a job. Its also
> not possible to do rolling upgrade of MetaStore Server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339480#comment-14339480 ] Thiruvel Thirumoolan commented on HIVE-9582: [~xiaobingo] I am eager to get this committed. I am waiting for review feedback. [~sushanth] Can you take a look? > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 0.14.1 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9582) HCatalog should use IMetaStoreClient interface
[ https://issues.apache.org/jira/browse/HIVE-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339177#comment-14339177 ] Xiaobing Zhou commented on HIVE-9582: - [~thiruvel] When will this patch be committed to trunk? HIVE-9642 is expecting this patch to work properly. Thanks. > HCatalog should use IMetaStoreClient interface > -- > > Key: HIVE-9582 > URL: https://issues.apache.org/jira/browse/HIVE-9582 > Project: Hive > Issue Type: Sub-task > Components: HCatalog, Metastore >Affects Versions: 0.14.0, 0.13.1 >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan > Labels: hcatalog, metastore, rolling_upgrade > Fix For: 0.14.1 > > Attachments: HIVE-9582.1.patch, HIVE-9582.2.patch, HIVE-9582.3.patch, > HIVE-9583.1.patch > > > Hive uses IMetaStoreClient and it makes using RetryingMetaStoreClient easy. > Hence during a failure, the client retries and possibly succeeds. But > HCatalog has long been using HiveMetaStoreClient directly and hence failures > are costly, especially if they are during the commit stage of a job. Its also > not possible to do rolling upgrade of MetaStore Server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
