[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693208#comment-14693208 ] Wojciech Indyk commented on HIVE-11329: --- [~swarnim] , I answered on the review comment. Could you advise me where to define enabling a flag for the prefix cutting (e.g. as a parameter of a table or hive variable or something else)? > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.1.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693725#comment-14693725 ] Swarnim Kulkarni commented on HIVE-11329: - [~woj_in] I would follow the example configuration in the HBaseSerDe class[1] to see how a configuration parameter can be passed in and then used within the code. Also let me know if you have any questions. [1] https://github.com/apache/hive/blob/master/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseSerDe.java#L72-L85 > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.1.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695075#comment-14695075 ] Wojciech Indyk commented on HIVE-11329: --- [~swarnim] thank you for your advice. I have prepared the final patch (attachment) and also updated a review board with the new patch https://reviews.apache.org/r/36974/ > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.1.patch, HIVE-11329.2.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[
https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695878#comment-14695878
]
Hive QA commented on HIVE-11329:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12750272/HIVE-11329.2.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 9357 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_partitioned
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4953/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4953/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-4953/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12750272 - PreCommit-HIVE-TRUNK-Build
> Column prefix in key of hbase column prefix map
> ---
>
> Key: HIVE-11329
> URL: https://issues.apache.org/jira/browse/HIVE-11329
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 0.14.0
>Reporter: Wojciech Indyk
>Assignee: Wojciech Indyk
>Priority: Minor
> Attachments: HIVE-11329.1.patch, HIVE-11329.2.patch
>
>
> When I create a table with hbase column prefix
> https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result
> map in hive.
> E.g. record in HBase
> rowkey: 123
> column: tag_one, value: 0.5
> column: tag_two, value 0.5
> representation in Hive via column prefix mapping "tag_.*":
> column: tag map
> key: tag_one, value: 0.5
> key: tag_two, value: 0.5
> should be:
> key: one, value: 0.5
> key: two: value: 0.5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699970#comment-14699970 ] Swarnim Kulkarni commented on HIVE-11329: - [~woj_in] I left comments on your RB review. However, I am not a committer. So you would also need blessing of a committer to get this patch merged in. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.1.patch, HIVE-11329.2.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14701709#comment-14701709 ] Brock Noland commented on HIVE-11329: - [~ctang.ma] or [~spena] can you take a look? > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[
https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702051#comment-14702051
]
Hive QA commented on HIVE-11329:
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12751041/HIVE-11329.3.patch
{color:green}SUCCESS:{color} +1 9370 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5000/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5000/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-5000/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12751041 - PreCommit-HIVE-TRUNK-Build
> Column prefix in key of hbase column prefix map
> ---
>
> Key: HIVE-11329
> URL: https://issues.apache.org/jira/browse/HIVE-11329
> Project: Hive
> Issue Type: Improvement
> Components: HBase Handler
>Affects Versions: 0.14.0
>Reporter: Wojciech Indyk
>Assignee: Wojciech Indyk
>Priority: Minor
> Attachments: HIVE-11329.3.patch
>
>
> When I create a table with hbase column prefix
> https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result
> map in hive.
> E.g. record in HBase
> rowkey: 123
> column: tag_one, value: 0.5
> column: tag_two, value 0.5
> representation in Hive via column prefix mapping "tag_.*":
> column: tag map
> key: tag_one, value: 0.5
> key: tag_two, value: 0.5
> should be:
> key: one, value: 0.5
> key: two: value: 0.5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723598#comment-14723598 ] Wojciech Indyk commented on HIVE-11329: --- [~spena], [~ctang.ma] could you review this patch, please? > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723883#comment-14723883 ] Sergio Peña commented on HIVE-11329: The patch looks good [~woj_in]. I don't know the hbase support, but the changes look simple. I run the tests as well, and they were executed correctly. +1 > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14733708#comment-14733708 ] Wojciech Indyk commented on HIVE-11329: --- [~spena], what we need to do to commit this patch? > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736258#comment-14736258 ] Lefty Leverenz commented on HIVE-11329: --- (1) Version query: Since this was committed to master, shouldn't Fix Version/s be 2.0.0? Or was it previously committed to branch-1 (1.3.0)? The commit to master (Tue Sept 8) is d51c62a455eb08ee49f10ea2e117ca90de0bf47b although patch 3 has a header dated Fri Jul 24 which gives commit ID a7a15acb58742bca61824d6221446ad1446d5ab7, so I'm confused. (2) Doc query: Does this need any documentation? In particular, should *hbase.columns.mapping.prefix.hide* be documented in the Hive HBase Integration wikidoc? * [Hive HBase Integration | https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration] > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Fix For: 1.3.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736957#comment-14736957 ] Sergio Peña commented on HIVE-11329: Thanks [~leftylev]. Isn't branch-1 for code that should work with hadoop-1 versions? I set this patch to 1.3 version because I did not see a branch-1.3 created yet. Regarding the commit Id, it might be because the patch was generated from a local commit [~woj_in] did. Once the patch is applied, such commit ID is not used, and a new one is generated. For documentation, I think [~woj_in] can respond to that. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Fix For: 1.3.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737951#comment-14737951 ] Lefty Leverenz commented on HIVE-11329: --- [~spena], my understanding is that commits to master will be in the 2.0.0 release and commits to branch-1 will be in the 1.3.0 release. And yes, branch-1 must work with Hadoop 1 as well as Hadoop 2 but master (2.0.0) only has to work with Hadoop 2. The wiki has more details: * [How To Contribute -- Understanding Hive Branches | https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-UnderstandingHiveBranches] > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Fix For: 1.3.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738319#comment-14738319 ] Wojciech Indyk commented on HIVE-11329: --- [~leftylev], [~spena] ok, I'll prepare a documentation. The patch works with both Hadoop 2 as well as Hadoop 1. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Fix For: 1.3.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738353#comment-14738353 ] Lefty Leverenz commented on HIVE-11329: --- Sorry to be a pest, but I still don't understand the fix version number. Was the patch committed to branch-1 (for upcoming release 1.3.0) in addition to the commit to master (for upcoming 2.0.0) that I saw in the dev@hive mailing list? > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Fix For: 1.3.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738870#comment-14738870 ] Sergio Peña commented on HIVE-11329: Thanks [~leftylev] for your help on this. Sorry for the late update on this, but I now committed the patch to branch-1 as well. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740243#comment-14740243 ] Lefty Leverenz commented on HIVE-11329: --- Okay, now it makes sense. Thanks [~spena]. I'm adding a TODOC1.3 label which we can remove after Wojciech provides documentation. [~woj_in], do you want to put the documentation in the wiki yourself or will you add it to this issue? Here's how you can get wiki edit permission: * [About This Wiki -- How to get permission to edit | https://cwiki.apache.org/confluence/display/Hive/AboutThisWiki#AboutThisWiki-Howtogetpermissiontoedit] > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Labels: TODOC1.3 > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740982#comment-14740982 ] Wojciech Indyk commented on HIVE-11329: --- I'll put the documentation directly in the wiki. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Labels: TODOC1.3 > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743393#comment-14743393 ] Wojciech Indyk commented on HIVE-11329: --- [~leftylev] thank you for the access to the wiki. I put a documentation to this page https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration - paragraph Hiding column prefixes. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Labels: TODOC1.3 > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743507#comment-14743507 ] Swarnim Kulkarni commented on HIVE-11329: - +1 on the doc. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Labels: TODOC1.3 > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744311#comment-14744311 ] Lefty Leverenz commented on HIVE-11329: --- +1 on the doc. (Just a few trivial edits.) I removed the TODOC1.3 label. Thanks [~woj_in]. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11329.3.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14640064#comment-14640064 ] Wojciech Indyk commented on HIVE-11329: --- I have prepared ability to flag a map to hide hbase column prefix. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.1.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14641832#comment-14641832 ] Swarnim Kulkarni commented on HIVE-11329: - [~woj_in] Thanks for the patch! I am pretty sure I am missing something here but would you mind explaining with an example to me as to what problems is having a prefix in the column name causing? The reason I ask is this needs to be consistently applied in all the other cases as well if needed, for example when reading all columns of hbase etc. > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.1.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[
https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642511#comment-14642511
]
Wojciech Indyk commented on HIVE-11329:
---
[~swarnim] Example: I define a map prefix "loc_" for geographical locations of
a row in HBase.
I create table in Hive
CREATE EXTERNAL TABLE xyz(id int, locations map)
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key, source:loc_.*")
Then I want to query:
select id where locations['California']!=null
instead of
select id where locations['loc_California']!=null
Moreover if I query:
select id, locations where locations['California']!=null
I would like to have a result like:
1, {California:state, New York:state}
instead of
1, {loc_California:state, loc_New York:state}
In general: I don't want to receive the prefix for each element of the map in
hive. I know what the prefix for the map is (it is defined in SERDEPROPERTIES).
It is hard to use prefixed data with another data sources, e.g. a
IP->geolocation libraries.
All in all it's easier to integrate data without prefixes. IMO Prefixes are
artificial structure (like 'super-column') to optimize queries and be able to
store a map in hbase. That's why i want to cut prefixes. I know a solution with
column family -> hive map, but HBase doesn't support more than 2 CF well. I
need ~10 maps in row.
I think the idea with a flag is very well. IMO it could be flag defined on
creating table.
> Column prefix in key of hbase column prefix map
> ---
>
> Key: HIVE-11329
> URL: https://issues.apache.org/jira/browse/HIVE-11329
> Project: Hive
> Issue Type: Bug
> Components: HBase Handler
>Affects Versions: 0.14.0
>Reporter: Wojciech Indyk
>Assignee: Wojciech Indyk
>Priority: Minor
> Attachments: HIVE-11329.1.patch
>
>
> When I create a table with hbase column prefix
> https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result
> map in hive.
> E.g. record in HBase
> rowkey: 123
> column: tag_one, value: 0.5
> column: tag_two, value 0.5
> representation in Hive via column prefix mapping "tag_.*":
> column: tag map
> key: tag_one, value: 0.5
> key: tag_two, value: 0.5
> should be:
> key: one, value: 0.5
> key: two: value: 0.5
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14646264#comment-14646264 ] Swarnim Kulkarni commented on HIVE-11329: - Thanks [~woj_in]. Would you mind opening a review board request for me? > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.1.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11329) Column prefix in key of hbase column prefix map
[ https://issues.apache.org/jira/browse/HIVE-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14648781#comment-14648781 ] Wojciech Indyk commented on HIVE-11329: --- sure, here is the request: https://reviews.apache.org/r/36974/ > Column prefix in key of hbase column prefix map > --- > > Key: HIVE-11329 > URL: https://issues.apache.org/jira/browse/HIVE-11329 > Project: Hive > Issue Type: Bug > Components: HBase Handler >Affects Versions: 0.14.0 >Reporter: Wojciech Indyk >Assignee: Wojciech Indyk >Priority: Minor > Attachments: HIVE-11329.1.patch > > > When I create a table with hbase column prefix > https://issues.apache.org/jira/browse/HIVE-3725 I have the prefix in result > map in hive. > E.g. record in HBase > rowkey: 123 > column: tag_one, value: 0.5 > column: tag_two, value 0.5 > representation in Hive via column prefix mapping "tag_.*": > column: tag map > key: tag_one, value: 0.5 > key: tag_two, value: 0.5 > should be: > key: one, value: 0.5 > key: two: value: 0.5 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
