[jira] [Commented] (HIVE-2903) Numeric binary type keys are not compared properly
[
https://issues.apache.org/jira/browse/HIVE-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239088#comment-13239088
]
Enis Soztutar commented on HIVE-2903:
-
Well, it is not a bug of hbase. HBase only provides int - byte[] conversion
as a convenience, and it seems that Bytes.toBytes(int) and others only
guarantees lexicographic ordering for unsigned numbers. We can definitely add
something like Bytes.toSignedBytes() in HBase so that you can ensure signed
numbers are sorted correctly in lexicographic order.
Coming to Hive, I think Ashutosh is right, that we have to keep supporting
already existing data in hbase serialized through Bytes.toBytes(). So, I would
suggest we add another storage type (hbase.table.default.storage.type), like
signedbinary, which should do the hive-specific signed byte conversion.
So, we would have:
- cf:col#string : serialize as string
- cf:col#binary : serialize as binary, compatible with Bytes.toBytes()
- cf:col#signedBinary : serialize as signed binary.
I would also suggest that, people might be interested in custom ser/de from
Hive types to byte[], but I am not sure how feasible that would be to
implement.
Numeric binary type keys are not compared properly
--
Key: HIVE-2903
URL: https://issues.apache.org/jira/browse/HIVE-2903
Project: Hive
Issue Type: Bug
Components: HBase Handler
Reporter: Navis
Assignee: Navis
Attachments: HIVE-2903.D2481.1.patch
In current binary format for numbers, minus values are always greater than
plus values, for example.
{code}
System.our.println(Bytes.compareTo(Bytes.toBytes(-100), Bytes.toBytes(100)));
// 255
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13222862#comment-13222862 ] Enis Soztutar commented on HIVE-2764: - We had some discussion with Ashutosh about how to best approach this. As a part of this issue, we have to obtain delegation tokens from the storage handler (hbase in this case), and then pass this to the submitted job. However, HiveStorageHandler.configureTableJobProperties() does not expose the JobConf object, or an interface to configure the job's credentials. There seems a couple of options we can go for: 1. At HiveStorageHandler.configureTableJobProperties(), create and save the job credentials somewhere, and at the time of job submit, merge all the credentials before submitting the job. 2. Change the HiveStorageHandler.configureTableJobProperties() to get the actual Job to be submitted, so that Storage handlers have a real shot at configuring the job. 3. Add a method like HiveStorageHandler.configureJobCredentials(), and call it from HIF.getSplits() and HiveHBaseTableOutputFormat.checkOutputSpecs(). Number 1 and 3 seems hackish, and we will want to give the storagehandlers ability to configure the actual job. So we are leaning towards that. But this will also be merged with HIVE-2773. I am still trying to make this work, so any comments / suggestions are more than welcome. Obtain delegation tokens for MR jobs in secure hbase setup Key: HIVE-2764 URL: https://issues.apache.org/jira/browse/HIVE-2764 Project: Hive Issue Type: Improvement Components: HBase Handler, Security Reporter: Enis Soztutar Assignee: Enis Soztutar As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to obtain delegation tokens for hbase and save it in jobconf, so that tasks can access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2833) Fix test failures caused by HIVE-2716
[ https://issues.apache.org/jira/browse/HIVE-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220597#comment-13220597 ] Enis Soztutar commented on HIVE-2833: - I can work on it, if we can get a reproducible test, or a list of tests which are failing. Is there an easy way to run the tests on trunk with mysql? Fix test failures caused by HIVE-2716 - Key: HIVE-2833 URL: https://issues.apache.org/jira/browse/HIVE-2833 Project: Hive Issue Type: Bug Components: Metastore Reporter: Carl Steinbach Assignee: Enis Soztutar -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2743) Enable SASL mode for HiveServer
[ https://issues.apache.org/jira/browse/HIVE-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13196621#comment-13196621 ] Enis Soztutar commented on HIVE-2743: - Carl, will you work on this? Enable SASL mode for HiveServer --- Key: HIVE-2743 URL: https://issues.apache.org/jira/browse/HIVE-2743 Project: Hive Issue Type: New Feature Components: Security, Server Infrastructure Reporter: Carl Steinbach Assignee: Carl Steinbach -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192549#comment-13192549 ] Enis Soztutar commented on HIVE-2698: - @Jack su, You are probably in the wrong directory. The patch applies cleanly with -p1 on the root hive directory. The a/b prefixes are due to the git-produced patch, which is why we are using -p1 instead of -p0. Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Assignee: Enis Soztutar Labels: hadoop, hadoop-1.0, jars Attachments: HIVE-2698.D1389.1.patch, HIVE-2698_v1.patch, HIVE-2698_v2.patch, HIVE-2698_v3.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13192584#comment-13192584 ] Enis Soztutar commented on HIVE-2698: - The patch from phabricator already applies with -p0. I was referring to the previous (v3) patch. Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Assignee: Enis Soztutar Labels: hadoop, hadoop-1.0, jars Attachments: HIVE-2698.D1389.1.patch, HIVE-2698_v1.patch, HIVE-2698_v2.patch, HIVE-2698_v3.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[
https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13190634#comment-13190634
]
Enis Soztutar commented on HIVE-2698:
-
You can apply the patch with the patch command.
{code}
patch -p1 ./HIVE-2698_v3.patch
{code}
Enable Hadoop-1.0.0 in Hive
---
Key: HIVE-2698
URL: https://issues.apache.org/jira/browse/HIVE-2698
Project: Hive
Issue Type: New Feature
Components: Security, Shims
Affects Versions: 0.9.0
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Labels: hadoop, hadoop-1.0, jars
Attachments: HIVE-2698_v1.patch, HIVE-2698_v2.patch,
HIVE-2698_v3.patch
Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the
0.20S release.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2720) Merge MetaStoreListener and HiveMetaHook interfaces
[ https://issues.apache.org/jira/browse/HIVE-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13187362#comment-13187362 ] Enis Soztutar commented on HIVE-2720: - MetaStoreListener.onXXX() methods take an Event object to wrap the function call context, however HiveMetaHook functions does not. We can do either way, but the first one will introduce a lot of classes (50+) in case we do add corresponding methods. Since we want both preXXX() and onXXX() methods, we can do smt like: preXXX() takes normal function args + context (containing back pointer to handler) onXXX() takes normal function args + returned object + context (containing back pointer to handler + method status) Anyone care to comment ? Merge MetaStoreListener and HiveMetaHook interfaces --- Key: HIVE-2720 URL: https://issues.apache.org/jira/browse/HIVE-2720 Project: Hive Issue Type: Sub-task Components: JDBC, Metastore, ODBC, Security Reporter: Enis Soztutar Assignee: Enis Soztutar MetaStoreListener and HiveMetaHook both serve as a notification mechanism for metastore-related events. The former is used by hcat and the latter is by the hbase-storage handler, and invoked by the client. I propose to merge these interfaces, and extend the MetaStoreListener, to add most of the on- and pre- methods at the Thrift interface. This way, extending metastore will be easier, and validation, storage-driver notification, and enforcement can be delegated to individual listeners. Besides, more functionality can be plugged-in by Hcat at this level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13185178#comment-13185178 ] Enis Soztutar commented on HIVE-2698: - This is the jar for the slf4j API only, not the binding for log4j. ql/ivy.xml and serde/ivy.xml already lists this as a dependency. Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Assignee: Enis Soztutar Attachments: HIVE-2698_v1.patch, HIVE-2698_v2.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[
https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183451#comment-13183451
]
Enis Soztutar commented on HIVE-2698:
-
You are right that we do not need a new shim, however, from my personal
preference, making an explicit shim for 1.0, but reusing {Hadoop|Jetty}20SShims
seems more clear. Moreover, the tarball directory structure has changed between
0.20.x and 1.0.0, so just changing the hadoop-0.20S.version does not work.
Enable Hadoop-1.0.0 in Hive
---
Key: HIVE-2698
URL: https://issues.apache.org/jira/browse/HIVE-2698
Project: Hive
Issue Type: New Feature
Components: Security, Shims
Affects Versions: 0.9.0
Reporter: Enis Soztutar
Attachments: HIVE-2698_v1.patch
Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the
0.20S release.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13183508#comment-13183508 ] Enis Soztutar commented on HIVE-2698: - Agreed for not needing a new shim. However, I'd rather compile and run secure tests with 1.0.0 rather than 0.20.3-CDH3. HIVE-2631 enables Hive binary to run with 1.0.0, but does not change the jar that Hive compiles against. We can change the patch for not introducing a new shim, but changing the 0.20S.version, and the classpath changes for the tarball. WDYT? Enable Hadoop-1.0.0 in Hive --- Key: HIVE-2698 URL: https://issues.apache.org/jira/browse/HIVE-2698 Project: Hive Issue Type: New Feature Components: Security, Shims Affects Versions: 0.9.0 Reporter: Enis Soztutar Attachments: HIVE-2698_v1.patch Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
