[jira] [Commented] (HIVE-2903) Numeric binary type keys are not compared properly
[
https://issues.apache.org/jira/browse/HIVE-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239088#comment-13239088
]
Enis Soztutar commented on HIVE-2903:
-
Well, it is not a "bug" of hbase. HBase only provides int -> byte[] conversion
as a convenience, and it seems that Bytes.toBytes(int) and others only
guarantees lexicographic ordering for unsigned numbers. We can definitely add
something like Bytes.toSignedBytes() in HBase so that you can ensure signed
numbers are sorted correctly in lexicographic order.
Coming to Hive, I think Ashutosh is right, that we have to keep supporting
already existing data in hbase serialized through Bytes.toBytes(). So, I would
suggest we add another storage type (hbase.table.default.storage.type), like
"signedbinary", which should do the hive-specific signed byte conversion.
So, we would have:
- cf:col#string : serialize as string
- cf:col#binary : serialize as binary, compatible with Bytes.toBytes()
- cf:col#signedBinary : serialize as signed binary.
I would also suggest that, people might be interested in custom ser/de from
Hive types to byte[], but I am not sure how feasible that would be to
implement.
> Numeric binary type keys are not compared properly
> --
>
> Key: HIVE-2903
> URL: https://issues.apache.org/jira/browse/HIVE-2903
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Reporter: Navis
>Assignee: Navis
> Attachments: HIVE-2903.D2481.1.patch
>
>
> In current binary format for numbers, minus values are always greater than
> plus values, for example.
> {code}
> System.our.println(Bytes.compareTo(Bytes.toBytes(-100), Bytes.toBytes(100)));
> // 255
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2764) Obtain delegation tokens for MR jobs in secure hbase setup
[ https://issues.apache.org/jira/browse/HIVE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222862#comment-13222862 ] Enis Soztutar commented on HIVE-2764: - We had some discussion with Ashutosh about how to best approach this. As a part of this issue, we have to obtain delegation tokens from the storage handler (hbase in this case), and then pass this to the submitted job. However, HiveStorageHandler.configureTableJobProperties() does not expose the JobConf object, or an interface to configure the job's credentials. There seems a couple of options we can go for: 1. At HiveStorageHandler.configureTableJobProperties(), create and save the job credentials somewhere, and at the time of job submit, merge all the credentials before submitting the job. 2. Change the HiveStorageHandler.configureTableJobProperties() to get the actual Job to be submitted, so that Storage handlers have a real shot at configuring the job. 3. Add a method like HiveStorageHandler.configureJobCredentials(), and call it from HIF.getSplits() and HiveHBaseTableOutputFormat.checkOutputSpecs(). Number 1 and 3 seems hackish, and we will want to give the storagehandlers ability to configure the actual job. So we are leaning towards that. But this will also be merged with HIVE-2773. I am still trying to make this work, so any comments / suggestions are more than welcome. > Obtain delegation tokens for MR jobs in secure hbase setup > > > Key: HIVE-2764 > URL: https://issues.apache.org/jira/browse/HIVE-2764 > Project: Hive > Issue Type: Improvement > Components: HBase Handler, Security >Reporter: Enis Soztutar >Assignee: Enis Soztutar > > As discussed in HCATALOG-244, in a secure hbase setup with 0.92, we need to > obtain delegation tokens for hbase and save it in jobconf, so that tasks can > access region servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2833) Fix test failures caused by HIVE-2716
[ https://issues.apache.org/jira/browse/HIVE-2833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220597#comment-13220597 ] Enis Soztutar commented on HIVE-2833: - I can work on it, if we can get a reproducible test, or a list of tests which are failing. Is there an easy way to run the tests on trunk with mysql? > Fix test failures caused by HIVE-2716 > - > > Key: HIVE-2833 > URL: https://issues.apache.org/jira/browse/HIVE-2833 > Project: Hive > Issue Type: Bug > Components: Metastore >Reporter: Carl Steinbach >Assignee: Enis Soztutar > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2743) Enable SASL mode for HiveServer
[ https://issues.apache.org/jira/browse/HIVE-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196621#comment-13196621 ] Enis Soztutar commented on HIVE-2743: - Carl, will you work on this? > Enable SASL mode for HiveServer > --- > > Key: HIVE-2743 > URL: https://issues.apache.org/jira/browse/HIVE-2743 > Project: Hive > Issue Type: New Feature > Components: Security, Server Infrastructure >Reporter: Carl Steinbach >Assignee: Carl Steinbach > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192584#comment-13192584 ] Enis Soztutar commented on HIVE-2698: - The patch from phabricator already applies with -p0. I was referring to the previous (v3) patch. > Enable Hadoop-1.0.0 in Hive > --- > > Key: HIVE-2698 > URL: https://issues.apache.org/jira/browse/HIVE-2698 > Project: Hive > Issue Type: New Feature > Components: Security, Shims >Affects Versions: 0.9.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: hadoop, hadoop-1.0, jars > Attachments: HIVE-2698.D1389.1.patch, HIVE-2698_v1.patch, > HIVE-2698_v2.patch, HIVE-2698_v3.patch > > > Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the > 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192549#comment-13192549 ] Enis Soztutar commented on HIVE-2698: - @Jack su, You are probably in the wrong directory. The patch applies cleanly with -p1 on the root hive directory. The a/b prefixes are due to the git-produced patch, which is why we are using -p1 instead of -p0. > Enable Hadoop-1.0.0 in Hive > --- > > Key: HIVE-2698 > URL: https://issues.apache.org/jira/browse/HIVE-2698 > Project: Hive > Issue Type: New Feature > Components: Security, Shims >Affects Versions: 0.9.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Labels: hadoop, hadoop-1.0, jars > Attachments: HIVE-2698.D1389.1.patch, HIVE-2698_v1.patch, > HIVE-2698_v2.patch, HIVE-2698_v3.patch > > > Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the > 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[
https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190634#comment-13190634
]
Enis Soztutar commented on HIVE-2698:
-
You can apply the patch with the patch command.
{code}
patch -p1 < ./HIVE-2698_v3.patch
{code}
> Enable Hadoop-1.0.0 in Hive
> ---
>
> Key: HIVE-2698
> URL: https://issues.apache.org/jira/browse/HIVE-2698
> Project: Hive
> Issue Type: New Feature
> Components: Security, Shims
>Affects Versions: 0.9.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Labels: hadoop, hadoop-1.0, jars
> Attachments: HIVE-2698_v1.patch, HIVE-2698_v2.patch,
> HIVE-2698_v3.patch
>
>
> Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the
> 0.20S release.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2720) Merge MetaStoreListener and HiveMetaHook interfaces
[ https://issues.apache.org/jira/browse/HIVE-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187362#comment-13187362 ] Enis Soztutar commented on HIVE-2720: - MetaStoreListener.onXXX() methods take an Event object to wrap the function call context, however HiveMetaHook functions does not. We can do either way, but the first one will introduce a lot of classes (50+) in case we do add corresponding methods. Since we want both preXXX() and onXXX() methods, we can do smt like: preXXX() takes normal function args + context (containing back pointer to handler) onXXX() takes normal function args + returned object + context (containing back pointer to handler + method status) Anyone care to comment ? > Merge MetaStoreListener and HiveMetaHook interfaces > --- > > Key: HIVE-2720 > URL: https://issues.apache.org/jira/browse/HIVE-2720 > Project: Hive > Issue Type: Sub-task > Components: JDBC, Metastore, ODBC, Security >Reporter: Enis Soztutar >Assignee: Enis Soztutar > > MetaStoreListener and HiveMetaHook both serve as a notification mechanism for > metastore-related events. The former is used by hcat and the latter is by the > hbase-storage handler, and invoked by the client. > I propose to merge these interfaces, and extend the MetaStoreListener, to add > most of the on- and pre- methods at the Thrift interface. This way, extending > metastore will be easier, and validation, storage-driver notification, and > enforcement can be delegated to individual listeners. Besides, more > functionality can be plugged-in by Hcat at this level. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185178#comment-13185178 ] Enis Soztutar commented on HIVE-2698: - This is the jar for the slf4j API only, not the binding for log4j. ql/ivy.xml and serde/ivy.xml already lists this as a dependency. > Enable Hadoop-1.0.0 in Hive > --- > > Key: HIVE-2698 > URL: https://issues.apache.org/jira/browse/HIVE-2698 > Project: Hive > Issue Type: New Feature > Components: Security, Shims >Affects Versions: 0.9.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Attachments: HIVE-2698_v1.patch, HIVE-2698_v2.patch > > > Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the > 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[ https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183508#comment-13183508 ] Enis Soztutar commented on HIVE-2698: - Agreed for not needing a new shim. However, I'd rather compile and run secure tests with 1.0.0 rather than 0.20.3-CDH3. HIVE-2631 enables Hive binary to run with 1.0.0, but does not change the jar that Hive compiles against. We can change the patch for not introducing a new shim, but changing the 0.20S.version, and the classpath changes for the tarball. WDYT? > Enable Hadoop-1.0.0 in Hive > --- > > Key: HIVE-2698 > URL: https://issues.apache.org/jira/browse/HIVE-2698 > Project: Hive > Issue Type: New Feature > Components: Security, Shims >Affects Versions: 0.9.0 >Reporter: Enis Soztutar > Attachments: HIVE-2698_v1.patch > > > Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the > 0.20S release. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2698) Enable Hadoop-1.0.0 in Hive
[
https://issues.apache.org/jira/browse/HIVE-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183451#comment-13183451
]
Enis Soztutar commented on HIVE-2698:
-
You are right that we do not need a new shim, however, from my personal
preference, making an explicit shim for 1.0, but reusing {Hadoop|Jetty}20SShims
seems more clear. Moreover, the tarball directory structure has changed between
0.20.x and 1.0.0, so just changing the hadoop-0.20S.version does not work.
> Enable Hadoop-1.0.0 in Hive
> ---
>
> Key: HIVE-2698
> URL: https://issues.apache.org/jira/browse/HIVE-2698
> Project: Hive
> Issue Type: New Feature
> Components: Security, Shims
>Affects Versions: 0.9.0
>Reporter: Enis Soztutar
> Attachments: HIVE-2698_v1.patch
>
>
> Hadoop-1.0.0 is recently released, which is AFAIK, API compatible to the
> 0.20S release.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
