[jira] [Updated] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2646: --- Attachment: HIVE-2646-fixtests.patch Patch file produced using arc export --unified --revision 2883 then replaced ivy/common-configurations section with output from svn diff ivy/common-configurations.xml > Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs > > > Key: HIVE-2646 > URL: https://issues.apache.org/jira/browse/HIVE-2646 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0 >Reporter: Andrew Bayer >Assignee: Andrew Bayer >Priority: Critical > Fix For: 0.9.0 > > Attachments: HIVE-2646-fixtests.patch, HIVE-2646.D2133.1.patch, > HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, > HIVE-2646.D2133.13.patch, HIVE-2646.D2133.14.patch, HIVE-2646.D2133.15.patch, > HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, > HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, > HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, HIVE-2646.D2883.1.patch, > HIVE-2646.diff.txt > > > The current Hive Ivy dependency logic for its Hadoop dependencies is > problematic - depending on the tarball and extracting the jars from there, > rather than depending on the jars directly. It'd be great if this was fixed > to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2646: --- Attachment: (was: HIVE-2646-fixtests.patch) > Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs > > > Key: HIVE-2646 > URL: https://issues.apache.org/jira/browse/HIVE-2646 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0 >Reporter: Andrew Bayer >Assignee: Andrew Bayer >Priority: Critical > Fix For: 0.9.0 > > Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, > HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, HIVE-2646.D2133.13.patch, > HIVE-2646.D2133.14.patch, HIVE-2646.D2133.15.patch, HIVE-2646.D2133.2.patch, > HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, > HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, > HIVE-2646.D2133.9.patch, HIVE-2646.D2883.1.patch, HIVE-2646.diff.txt > > > The current Hive Ivy dependency logic for its Hadoop dependencies is > problematic - depending on the tarball and extracting the jars from there, > rather than depending on the jars directly. It'd be great if this was fixed > to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2646: --- Attachment: HIVE-2646-fixtests.patch Patch to address unit test failures. This should apply cleanly to trunk (arc patch doesn't). > Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs > > > Key: HIVE-2646 > URL: https://issues.apache.org/jira/browse/HIVE-2646 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0 >Reporter: Andrew Bayer >Assignee: Andrew Bayer >Priority: Critical > Fix For: 0.9.0 > > Attachments: HIVE-2646-fixtests.patch, HIVE-2646.D2133.1.patch, > HIVE-2646.D2133.10.patch, HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, > HIVE-2646.D2133.13.patch, HIVE-2646.D2133.14.patch, HIVE-2646.D2133.15.patch, > HIVE-2646.D2133.2.patch, HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, > HIVE-2646.D2133.5.patch, HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, > HIVE-2646.D2133.8.patch, HIVE-2646.D2133.9.patch, HIVE-2646.D2883.1.patch, > HIVE-2646.diff.txt > > > The current Hive Ivy dependency logic for its Hadoop dependencies is > problematic - depending on the tarball and extracting the jars from there, > rather than depending on the jars directly. It'd be great if this was fixed > to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2702) listPartitionsByFilter only supports string partitions
[
https://issues.apache.org/jira/browse/HIVE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Weise updated HIVE-2702:
---
Summary: listPartitionsByFilter only supports string partitions (was:
listPartitionsByFilter only supports non-string partitions)
> listPartitionsByFilter only supports string partitions
> --
>
> Key: HIVE-2702
> URL: https://issues.apache.org/jira/browse/HIVE-2702
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.8.1
>Reporter: Aniket Mokashi
>Assignee: Aniket Mokashi
> Attachments: HIVE-2702.1.patch, HIVE-2702.D2043.1.patch
>
>
> listPartitionsByFilter supports only non-string partitions. This is because
> its explicitly specified in generateJDOFilterOverPartitions in
> ExpressionTree.java.
> //Can only support partitions whose types are string
> if( ! table.getPartitionKeys().get(partitionColumnIndex).
>
> getType().equals(org.apache.hadoop.hive.serde.Constants.STRING_TYPE_NAME) ) {
> throw new MetaException
> ("Filtering is supported only on partition keys of type string");
> }
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: HIVE-2923.patch Updated patch to increase timeout for slow test environments. > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: (was: HIVE-2923.patch) > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Attachment: HIVE-2923.patch The problem is that the ZooKeeper client is used before it is connected (may or may not occur depending on environment). I could reproduce this yesterday on my Mac but not this morning. Attached patch will add a wait for SyncConnected with timeout to address this. Can you please try the patch? > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2923) testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run on Mac
[ https://issues.apache.org/jira/browse/HIVE-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2923: --- Fix Version/s: 0.9.0 Affects Version/s: 0.9.0 Status: Patch Available (was: Open) > testAclPositive in TestZooKeeperTokenStore failing in clean checkout when run > on Mac > > > Key: HIVE-2923 > URL: https://issues.apache.org/jira/browse/HIVE-2923 > Project: Hive > Issue Type: Bug >Affects Versions: 0.9.0 > Environment: Mac OSX Lion >Reporter: Kevin Wilfong >Assignee: Thomas Weise >Priority: Blocker > Fix For: 0.9.0 > > Attachments: HIVE-2923.patch > > > When running testAclPositive in TestZooKeeperTokenStore in a clean checkout, > it fails with the error: > Failed to validate token path. > org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: > Failed to validate token path. > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:207) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.setConf(ZooKeeperTokenStore.java:225) > at > org.apache.hadoop.hive.thrift.TestZooKeeperTokenStore.testAclPositive(TestZooKeeperTokenStore.java:170) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) > at junit.framework.TestResult.runProtected(TestResult.java:128) > at junit.framework.TestResult.run(TestResult.java:113) > at junit.framework.TestCase.run(TestCase.java:124) > at junit.framework.TestSuite.runTest(TestSuite.java:232) > at junit.framework.TestSuite.run(TestSuite.java:227) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:518) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:1052) > at > org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:906) > Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /zktokenstore-testAcl > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:778) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.ensurePath(ZooKeeperTokenStore.java:119) > at > org.apache.hadoop.hive.thrift.ZooKeeperTokenStore.init(ZooKeeperTokenStore.java:204) > ... 17 more > This message is also printed to standard out: > Unable to load realm mapping info from SCDynamicStore > The test seems to run fine in Linux, but more than one developer has reported > this on a Mac. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2741) Single binary built against 0.20 and 0.23, does not work against 0.23 clusters.
[ https://issues.apache.org/jira/browse/HIVE-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2741: --- Fix Version/s: 0.9.0 Requires build against 0.23.1 > Single binary built against 0.20 and 0.23, does not work against 0.23 > clusters. > --- > > Key: HIVE-2741 > URL: https://issues.apache.org/jira/browse/HIVE-2741 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 0.8.1 >Reporter: Amareshwari Sriramadasu > Fix For: 0.9.0 > > > After HIVE-2629, if single binary is built for 0.20 and 0.23, it results in > following exception in 0.23 clusters: > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapred.Counters$Counter, but class was > expected > at > org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:341) > at > org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:685) > at > org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:458) > at > org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:200) > FAILED: Execution Error, return code -101 from > org.apache.hadoop.hive.ql.exec.MapRedTask > If we have to make single binary work against both 0.20 and 0.23, we need to > move all such in-compatibilities to Shims. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2646) Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs
[ https://issues.apache.org/jira/browse/HIVE-2646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2646: --- Fix Version/s: 0.9.0 > Hive Ivy dependencies on Hadoop should depend on jars directly, not tarballs > > > Key: HIVE-2646 > URL: https://issues.apache.org/jira/browse/HIVE-2646 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Affects Versions: 0.8.0 >Reporter: Andrew Bayer >Assignee: Andrew Bayer >Priority: Critical > Fix For: 0.9.0 > > Attachments: HIVE-2646.D2133.1.patch, HIVE-2646.D2133.10.patch, > HIVE-2646.D2133.11.patch, HIVE-2646.D2133.12.patch, HIVE-2646.D2133.2.patch, > HIVE-2646.D2133.3.patch, HIVE-2646.D2133.4.patch, HIVE-2646.D2133.5.patch, > HIVE-2646.D2133.6.patch, HIVE-2646.D2133.7.patch, HIVE-2646.D2133.8.patch, > HIVE-2646.D2133.9.patch, HIVE-2646.diff.txt > > > The current Hive Ivy dependency logic for its Hadoop dependencies is > problematic - depending on the tarball and extracting the jars from there, > rather than depending on the jars directly. It'd be great if this was fixed > to actually have the jar dependencies defined directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2712) Make ZooKeeper token store ACL configurable
[ https://issues.apache.org/jira/browse/HIVE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2712: --- Status: Patch Available (was: Open) > Make ZooKeeper token store ACL configurable > --- > > Key: HIVE-2712 > URL: https://issues.apache.org/jira/browse/HIVE-2712 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Server Infrastructure >Affects Versions: 0.8.1, 0.8.0, 0.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2712.3.patch, HIVE-2712.D1401.1.patch, > HIVE-2712.D1401.2.patch, HIVE-2712.D1401.3.patch > > > ACL needs to be set to secure the token store with ZK 3.4. > The patch will also include the review changes from HIVE-2467 that were not > committed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2712) Make ZooKeeper token store ACL configurable
[ https://issues.apache.org/jira/browse/HIVE-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2712: --- Attachment: HIVE-2712.3.patch Patch with license grant. Need this patch on 0.8 branch also (no 0.8.2 fix version exists currently). > Make ZooKeeper token store ACL configurable > --- > > Key: HIVE-2712 > URL: https://issues.apache.org/jira/browse/HIVE-2712 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Server Infrastructure >Affects Versions: 0.8.0, 0.8.1, 0.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2712.3.patch, HIVE-2712.D1401.1.patch, > HIVE-2712.D1401.2.patch, HIVE-2712.D1401.3.patch > > > ACL needs to be set to secure the token store with ZK 3.4. > The patch will also include the review changes from HIVE-2467 that were not > committed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2741) Single binary built against 0.20 and 0.23, does not work against 0.23 clusters.
[ https://issues.apache.org/jira/browse/HIVE-2741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2741: --- Component/s: Shims > Single binary built against 0.20 and 0.23, does not work against 0.23 > clusters. > --- > > Key: HIVE-2741 > URL: https://issues.apache.org/jira/browse/HIVE-2741 > Project: Hive > Issue Type: Bug > Components: Shims >Affects Versions: 0.8.1 >Reporter: Amareshwari Sriramadasu > > After HIVE-2629, if single binary is built for 0.20 and 0.23, it results in > following exception in 0.23 clusters: > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapred.Counters$Counter, but class was > expected > at > org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:341) > at > org.apache.hadoop.hive.ql.exec.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:685) > at > org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:458) > at > org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:133) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1332) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1123) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:931) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:255) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:212) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:671) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:200) > FAILED: Execution Error, return code -101 from > org.apache.hadoop.hive.ql.exec.MapRedTask > If we have to make single binary work against both 0.20 and 0.23, we need to > move all such in-compatibilities to Shims. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2665) Support for metastore service specific HADOOP_OPTS environment setting
[ https://issues.apache.org/jira/browse/HIVE-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2665: --- Fix Version/s: 0.9.0 Affects Version/s: 0.9.0 0.8.0 Status: Patch Available (was: Open) > Support for metastore service specific HADOOP_OPTS environment setting > -- > > Key: HIVE-2665 > URL: https://issues.apache.org/jira/browse/HIVE-2665 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.8.0, 0.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > Fix For: 0.9.0 > > Attachments: HIVE-2665.patch > > > For development/testing it would be helpful to have a way to define > HADOOP_OPTS that apply only to a specific launcher and don't affect > everything else launched through bin/hadoop. In this specific case I'm > looking for a way to set metastore JVM debug options w/o modifying the > HADOOP_OPTS environment setting or the hive scripts (which are replaced on > every build). > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2665) Support for metastore service specific HADOOP_OPTS environment setting
[ https://issues.apache.org/jira/browse/HIVE-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2665: --- Attachment: HIVE-2665.patch > Support for metastore service specific HADOOP_OPTS environment setting > -- > > Key: HIVE-2665 > URL: https://issues.apache.org/jira/browse/HIVE-2665 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.8.0, 0.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > Fix For: 0.9.0 > > Attachments: HIVE-2665.patch > > > For development/testing it would be helpful to have a way to define > HADOOP_OPTS that apply only to a specific launcher and don't affect > everything else launched through bin/hadoop. In this specific case I'm > looking for a way to set metastore JVM debug options w/o modifying the > HADOOP_OPTS environment setting or the hive scripts (which are replaced on > every build). > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2589) Newly created partition should inherit properties from table
[ https://issues.apache.org/jira/browse/HIVE-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2589: --- Fix Version/s: 0.8.1 Need this for HCat 0.3 > Newly created partition should inherit properties from table > > > Key: HIVE-2589 > URL: https://issues.apache.org/jira/browse/HIVE-2589 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 0.8.1, 0.9.0 > > Attachments: hive-2589.patch, hive-2589.patch, hive-2589_1.patch, > hive-2589_2.patch > > > This will make all the info contained in table properties available to > partitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2629: --- Attachment: HIVE-2629-InputSplitShim.patch Attached additional patch resolves the array cast issue Amareshwari identified. The test case did not pass on 0.23 before and after the patch, this is only to address the additional error. > Make a single Hive binary work with both 0.20.x and 0.23.0 > -- > > Key: HIVE-2629 > URL: https://issues.apache.org/jira/browse/HIVE-2629 > Project: Hive > Issue Type: Bug > Components: Shims >Reporter: Carl Steinbach >Assignee: Thomas Weise > Fix For: 0.8.1, 0.9.0 > > Attachments: HIVE-2629-InputSplitShim.patch, HIVE-2629.3.patch, > HIVE-2629.D711.1.patch, HIVE-2629.D711.2.patch, HIVE-2629.D711.3.patch, > HIVE-2629.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2629: --- Attachment: HIVE-2629.3.patch svn diff --show-copies-as-adds > HIVE-2629.3.patch (Apply using svn patch (svn 1.7) to preserve svn add/delete) > Make a single Hive binary work with both 0.20.x and 0.23.0 > -- > > Key: HIVE-2629 > URL: https://issues.apache.org/jira/browse/HIVE-2629 > Project: Hive > Issue Type: Bug > Components: Shims >Reporter: Carl Steinbach >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2629.3.patch, HIVE-2629.D711.1.patch, > HIVE-2629.D711.2.patch, HIVE-2629.D711.3.patch, HIVE-2629.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2688) arc patch does not handle file copy to newly added directories
[ https://issues.apache.org/jira/browse/HIVE-2688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2688: --- Assignee: John Sichi > arc patch does not handle file copy to newly added directories > --- > > Key: HIVE-2688 > URL: https://issues.apache.org/jira/browse/HIVE-2688 > Project: Hive > Issue Type: Bug > Components: Build Infrastructure >Reporter: Thomas Weise >Assignee: John Sichi > > https://reviews.facebook.net/D1077 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2629: --- Attachment: HIVE-2629.patch Attached patch was generated using svn client 1.7 to represent moved files as adds: svn diff --show-copies-as-adds > HIVE-2629.patch > Make a single Hive binary work with both 0.20.x and 0.23.0 > -- > > Key: HIVE-2629 > URL: https://issues.apache.org/jira/browse/HIVE-2629 > Project: Hive > Issue Type: Bug > Components: Shims >Reporter: Carl Steinbach >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2629.D711.1.patch, HIVE-2629.D711.2.patch, > HIVE-2629.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2629: --- Attachment: (was: HIVE-2629.patch) > Make a single Hive binary work with both 0.20.x and 0.23.0 > -- > > Key: HIVE-2629 > URL: https://issues.apache.org/jira/browse/HIVE-2629 > Project: Hive > Issue Type: Bug > Components: Shims >Reporter: Carl Steinbach >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2629.D711.1.patch, HIVE-2629.D711.2.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2629: --- Attachment: HIVE-2629.patch Uploading patch manually for Ashutosh to get past Phabricator issue. > Make a single Hive binary work with both 0.20.x and 0.23.0 > -- > > Key: HIVE-2629 > URL: https://issues.apache.org/jira/browse/HIVE-2629 > Project: Hive > Issue Type: Bug > Components: Shims >Reporter: Carl Steinbach >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2629.D711.1.patch, HIVE-2629.D711.2.patch, > HIVE-2629.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2665) Support for metastore service specific HADOOP_OPTS environment setting
[ https://issues.apache.org/jira/browse/HIVE-2665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2665: --- Summary: Support for metastore service specific HADOOP_OPTS environment setting (was: Support for metastore specific HADOOP_OPTS environment setting) > Support for metastore service specific HADOOP_OPTS environment setting > -- > > Key: HIVE-2665 > URL: https://issues.apache.org/jira/browse/HIVE-2665 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Thomas Weise >Assignee: Thomas Weise >Priority: Minor > > For development/testing it would be helpful to have a way to define > HADOOP_OPTS that apply only to a specific launcher and don't affect > everything else launched through bin/hadoop. In this specific case I'm > looking for a way to set metastore JVM debug options w/o modifying the > HADOOP_OPTS environment setting or the hive scripts (which are replaced on > every build). > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2629) Make a single Hive binary work with both 0.20.x and 0.23.0
[ https://issues.apache.org/jira/browse/HIVE-2629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2629: --- Fix Version/s: 0.9.0 Status: Patch Available (was: Open) https://reviews.facebook.net/D711 > Make a single Hive binary work with both 0.20.x and 0.23.0 > -- > > Key: HIVE-2629 > URL: https://issues.apache.org/jira/browse/HIVE-2629 > Project: Hive > Issue Type: Bug > Components: Shims >Reporter: Carl Steinbach >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2629.D711.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2467: --- Attachment: HIVE-2467.2.patch Updated patch: - No need to load master keys on retrievePassword - Make MemoryStore default when no token store is configured (instead of using old implementation) > HA Support for Metastore Server > > > Key: HIVE-2467 > URL: https://issues.apache.org/jira/browse/HIVE-2467 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Server Infrastructure >Affects Versions: 0.8.0, 0.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2467.2.patch, HIVE-2467.patch > > > We require HA deployment for metastore server for HCatalog: > * Multiple server instances run behind VIP > * Database provides HA > Metastore server instances will need to be able to share any state required > for VIP outside RDBMS. As of Hive 0.8 affected conversational state that > needs to support VIP/HA setup is limited to current delegation tokens. Is > this correct? > We are planning to use ZooKeeper to share current delegation tokens and > master keys between nodes of the VIP. ZK is already (optionally) used by Hive > for concurrency control. Access to ZK would be limited on the network level > or in the future, when ZooKeeper supports security, through Kerberos, similar > to NN access. > Currently Hive taps into Hadoop core security delegation token support > through extension of > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager > A solution could amend the Hive specific extension to support: > * Pluggable delegation token and master key store (ZooKeeper as alternative > for in-memory AbstractDelegationTokenSecretManager) > * Delegation token retrieval from token store when not found in memory > (wrap/extend retrievePassword(...)) > * Cancellation of token in token store > * Purging of expired tokens from token store > http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2467: --- Component/s: Metastore > HA Support for Metastore Server > > > Key: HIVE-2467 > URL: https://issues.apache.org/jira/browse/HIVE-2467 > Project: Hive > Issue Type: Improvement > Components: Metastore, Security, Server Infrastructure >Affects Versions: 0.8.0, 0.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2467.patch > > > We require HA deployment for metastore server for HCatalog: > * Multiple server instances run behind VIP > * Database provides HA > Metastore server instances will need to be able to share any state required > for VIP outside RDBMS. As of Hive 0.8 affected conversational state that > needs to support VIP/HA setup is limited to current delegation tokens. Is > this correct? > We are planning to use ZooKeeper to share current delegation tokens and > master keys between nodes of the VIP. ZK is already (optionally) used by Hive > for concurrency control. Access to ZK would be limited on the network level > or in the future, when ZooKeeper supports security, through Kerberos, similar > to NN access. > Currently Hive taps into Hadoop core security delegation token support > through extension of > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager > A solution could amend the Hive specific extension to support: > * Pluggable delegation token and master key store (ZooKeeper as alternative > for in-memory AbstractDelegationTokenSecretManager) > * Delegation token retrieval from token store when not found in memory > (wrap/extend retrievePassword(...)) > * Cancellation of token in token store > * Purging of expired tokens from token store > http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2467: --- Component/s: (was: Metastore) Server Infrastructure Security > HA Support for Metastore Server > > > Key: HIVE-2467 > URL: https://issues.apache.org/jira/browse/HIVE-2467 > Project: Hive > Issue Type: Improvement > Components: Security, Server Infrastructure >Affects Versions: 0.8.0, 0.9.0 >Reporter: Thomas Weise >Assignee: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2467.patch > > > We require HA deployment for metastore server for HCatalog: > * Multiple server instances run behind VIP > * Database provides HA > Metastore server instances will need to be able to share any state required > for VIP outside RDBMS. As of Hive 0.8 affected conversational state that > needs to support VIP/HA setup is limited to current delegation tokens. Is > this correct? > We are planning to use ZooKeeper to share current delegation tokens and > master keys between nodes of the VIP. ZK is already (optionally) used by Hive > for concurrency control. Access to ZK would be limited on the network level > or in the future, when ZooKeeper supports security, through Kerberos, similar > to NN access. > Currently Hive taps into Hadoop core security delegation token support > through extension of > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager > A solution could amend the Hive specific extension to support: > * Pluggable delegation token and master key store (ZooKeeper as alternative > for in-memory AbstractDelegationTokenSecretManager) > * Delegation token retrieval from token store when not found in memory > (wrap/extend retrievePassword(...)) > * Cancellation of token in token store > * Purging of expired tokens from token store > http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2467: --- Fix Version/s: 0.9.0 Affects Version/s: 0.9.0 Status: Patch Available (was: Open) > HA Support for Metastore Server > > > Key: HIVE-2467 > URL: https://issues.apache.org/jira/browse/HIVE-2467 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.8.0, 0.9.0 >Reporter: Thomas Weise > Fix For: 0.9.0 > > Attachments: HIVE-2467.patch > > > We require HA deployment for metastore server for HCatalog: > * Multiple server instances run behind VIP > * Database provides HA > Metastore server instances will need to be able to share any state required > for VIP outside RDBMS. As of Hive 0.8 affected conversational state that > needs to support VIP/HA setup is limited to current delegation tokens. Is > this correct? > We are planning to use ZooKeeper to share current delegation tokens and > master keys between nodes of the VIP. ZK is already (optionally) used by Hive > for concurrency control. Access to ZK would be limited on the network level > or in the future, when ZooKeeper supports security, through Kerberos, similar > to NN access. > Currently Hive taps into Hadoop core security delegation token support > through extension of > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager > A solution could amend the Hive specific extension to support: > * Pluggable delegation token and master key store (ZooKeeper as alternative > for in-memory AbstractDelegationTokenSecretManager) > * Delegation token retrieval from token store when not found in memory > (wrap/extend retrievePassword(...)) > * Cancellation of token in token store > * Purging of expired tokens from token store > http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2467) HA Support for Metastore Server
[ https://issues.apache.org/jira/browse/HIVE-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Weise updated HIVE-2467: --- Attachment: HIVE-2467.patch Attached is the initial patch (with unit test coverage). Limited end to end testing has been done to validate the token sharing across multiple servers. No scalability testing has been done yet. Changes all go into the Hadoop 20S shim module (depend on secure Hadoop). Everything except the ZooKeeper token store implementation logically extends Hadoop security. AbstractDelegationTokenSecretManager was not designed for extensibility and this reflects in the subclass introduced here, which nevertheless duplicates as little code as possible. Longer term Hadoop should support the token store abstraction or at least more appropriate hooks for extension and dependency injection, which will allow for significant enhanced simplicity and clarity (and of course sharing with other projects). The ZooKeeper backed token store: The delegation keys and tokens are stored in ZooKeeper. Multiple metastore servers access ZooKeeper and load keys and tokens from there. Keys are cached and each metastore server instance creates its dedicated key, shared through ZooKeeper with other server instances for token validation. Tokens are not cached. Each server will fetch the token from ZooKeeper when the client establishes a metastore connection (not per request, but per connection). By default, none of the extensions introduced with this patch will be used (previous behavior and dependencies remain intact). To enable ZooKeeper token store, additional configuration is needed (on top of secure metastore setup): hive.cluster.delegation.token.store.class org.apache.hadoop.hive.thrift.ZooKeeperTokenStore The delegation token store implementation class. hive.cluster.delegation.token.store.zookeeper.connectString localhost:2181 The ZooKeeper token store connect string. hive.cluster.delegation.token.store.zookeeper.rootNode /hcat/tokenstore The root path for token store data. There are two potential enhancements I would like to put up for discussion: 1) Partition token storage by master key id (right now all tokens are children of a single ZooKeeper node. This will make it easier to manually inspect the store and make token expiration more efficient (no need to read all children at once). 2) Introduce dependency injection point for HadoopThriftAuthBridge20S.createServer, which can be used to plug-in alternative implementations of TokenStore, DelegationTokenManager and potentially other objects for customization and unit testing. > HA Support for Metastore Server > > > Key: HIVE-2467 > URL: https://issues.apache.org/jira/browse/HIVE-2467 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.8.0 >Reporter: Thomas Weise > Attachments: HIVE-2467.patch > > > We require HA deployment for metastore server for HCatalog: > * Multiple server instances run behind VIP > * Database provides HA > Metastore server instances will need to be able to share any state required > for VIP outside RDBMS. As of Hive 0.8 affected conversational state that > needs to support VIP/HA setup is limited to current delegation tokens. Is > this correct? > We are planning to use ZooKeeper to share current delegation tokens and > master keys between nodes of the VIP. ZK is already (optionally) used by Hive > for concurrency control. Access to ZK would be limited on the network level > or in the future, when ZooKeeper supports security, through Kerberos, similar > to NN access. > Currently Hive taps into Hadoop core security delegation token support > through extension of > org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager > A solution could amend the Hive specific extension to support: > * Pluggable delegation token and master key store (ZooKeeper as alternative > for in-memory AbstractDelegationTokenSecretManager) > * Delegation token retrieval from token store when not found in memory > (wrap/extend retrievePassword(...)) > * Cancellation of token in token store > * Purging of expired tokens from token store > http://www.mail-archive.com/hcatalog-user@incubator.apache.org/msg00053.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
