[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13893465#comment-13893465 ] Brock Noland commented on HIVE-2599: bq. But there are increasing number of cases where custom and more complicated serializers are written to serialize the keys in a more efficient way (salting is such an example) I've seen this as well. That is where a simple struct cannot deserialize a key. Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Fix For: 0.13.0 Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13892839#comment-13892839 ] Navis commented on HIVE-2599: - [~swarnim] [~brocknoland] I'm really late on this issue. Could I ask why LazySimpleStructObjectInspector started to make new List for getStructFieldsDataAsList()? Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Fix For: 0.13.0 Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13892894#comment-13892894
]
Navis commented on HIVE-2599:
-
I cannot understand the intention of this issue. Without this patch,
{noformat}
hive CREATE TABLE hbase_struct(key structcol1:int,col2:int, value string)
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string);
...
hive insert into table hbase_struct select struct(1000,2000),'value' from src
tablesample (1 rows);
...
hive select * from hbase_struct;
{col1:1000,col2:2000} value
hive select key.col1,key.col2 from hbase_struct;
10002000
{noformat}
I can do anything with default LazyStruct and now it's making new Struct and
using it. Is it for controlling wire-format on hbase rathe than default json
serialization?
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
Fix For: 0.13.0
Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt,
HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key and make data available. I noticed John Sichi mentioned in
HIVE-1228 that this would be a separate issue but I cant find any follow up.
How feasible is this in the HBaseStorageHandler?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13892996#comment-13892996
]
Swarnim Kulkarni commented on HIVE-2599:
{quote}
Could I ask why LazySimpleStructObjectInspector started to make new List for
getStructFieldsDataAsList()?
{quote}
Basically a bit of testing convenience on my side. Also tried to make the
implementation here similar to some of the existing ObjectInspectors[1]. It
tried my best to ensure that this passed all existing tests and also wasn't
violating any obvious contracts on the class. Did I miss something?
{quote}
I cannot understand the intention of this issue.
{quote}
You are correct in the sense that a basic struct doesn't need a custom
HBaseCompositeKey implementation(also refer to my first comment on this JIRA
for another example). But there are increasing number of cases where custom and
more complicated serializers are written to serialize the keys in a more
efficient way (salting is such an example). For such serializers, a custom
HBaseCompositeKey implementation would help hive understand such complicated
keys.
[1]
https://github.com/apache/hive/blob/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java#L174
[2]
https://github.com/apache/hive/blob/trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/UnionStructObjectInspector.java#L153
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
Fix For: 0.13.0
Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt,
HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key and make data available. I noticed John Sichi mentioned in
HIVE-1228 that this would be a separate issue but I cant find any follow up.
How feasible is this in the HBaseStorageHandler?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863009#comment-13863009 ] Brock Noland commented on HIVE-2599: Great to hear, it sounds like the composite key is working as desired -- both inserts and selects work as expected! The patch looks good to me as well! The only change required I noticed is that the two new classes related to the composite keys require apache license headers. I can commit this after that change! Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863122#comment-13863122 ] Nick Dimiduk commented on HIVE-2599: After a cursory review, patch v3 looks good to me also. It should be trivial to extend these concepts to provide a LazyStructObjectInspector over the Strict/StructIterator types in HBase. Let me see about finding time to do so this week. Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863380#comment-13863380 ] Brock Noland commented on HIVE-2599: +1 Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863391#comment-13863391 ] Brock Noland commented on HIVE-2599: I created HIVE-6150 - Take advantage of Native HBase Compound keys for the changes to take advantage of HBASE-8693. Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt, HIVE-2599.4.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862249#comment-13862249
]
Hive QA commented on HIVE-2599:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12621458/HIVE-2599.3.patch.txt
{color:green}SUCCESS:{color} +1 4877 tests passed
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/806/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/806/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12621458
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt,
HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key and make data available. I noticed John Sichi mentioned in
HIVE-1228 that this would be a separate issue but I cant find any follow up.
How feasible is this in the HBaseStorageHandler?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861705#comment-13861705 ] Brock Noland commented on HIVE-2599: Hi Swarnim, This looks pretty good! Am I correct that the patch takes care of both selects and inserts? Hi Nick, Do you have a simple example? Brock Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861727#comment-13861727
]
Swarnim Kulkarni commented on HIVE-2599:
{quote}
Am I correct that the patch takes care of both selects and inserts?
{quote}
Unfortunately no. This one would allow to querying of custom composite keys but
currently doesn't support writing them back to HBase. Do you want me to include
that support as a part of this patch itself or open up a separate issue for
that?
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt,
HIVE-2599.2.patch.txt
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key and make data available. I noticed John Sichi mentioned in
HIVE-1228 that this would be a separate issue but I cant find any follow up.
How feasible is this in the HBaseStorageHandler?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861736#comment-13861736 ] Brock Noland commented on HIVE-2599: What happens if an insert is tried? We can address that in a follow on JIRA as long as the results of an insert aren't data corruption or a terrible error message. Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861802#comment-13861802 ] Nick Dimiduk commented on HIVE-2599: bq. Do you have a simple example? The best documentation available is still in the [unit tests|https://github.com/apache/hbase/blob/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/types/TestStruct.java]. I will do a proper writeup of using this feature, it's just not a priority for me as of late. Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861808#comment-13861808 ] Brock Noland commented on HIVE-2599: Cool, sounds good. I think we can address this in a follow on JIRA since Swarnim has a working patch here for a common use case. Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13861998#comment-13861998
]
Hive QA commented on HIVE-2599:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597571/HIVE-2599.2.patch.txt
Test results:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/800/testReport
Console output:
http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/800/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-800/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted
'ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists2.q.out'
Reverted
'ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists.q.out'
Reverted
'ql/src/test/results/clientnegative/exchange_partition_neg_partition_exists3.q.out'
Reverted
'ql/src/test/results/clientnegative/exchange_partition_neg_incomplete_partition.q.out'
Reverted 'ql/src/test/results/clientpositive/exchange_partition3.q.out'
Reverted 'ql/src/test/results/clientpositive/exchange_partition.q.out'
Reverted 'ql/src/test/results/clientpositive/exchange_partition2.q.out'
Reverted
'ql/src/test/queries/clientnegative/exchange_partition_neg_incomplete_partition.q'
Reverted
'ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists2.q'
Reverted
'ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists3.q'
Reverted
'ql/src/test/queries/clientnegative/exchange_partition_neg_partition_missing.q'
Reverted
'ql/src/test/queries/clientnegative/exchange_partition_neg_partition_exists.q'
Reverted 'ql/src/test/queries/clientpositive/exchange_partition.q'
Reverted 'ql/src/test/queries/clientpositive/exchange_partition2.q'
Reverted 'ql/src/test/queries/clientpositive/exchange_partition3.q'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target
shims/0.20S/target shims/0.23/target shims/aggregator/target
shims/common/target shims/common-secure/target packaging/target
hbase-handler/target testutils/target jdbc/target metastore/target
itests/target itests/hcatalog-unit/target itests/test-serde/target
itests/qtest/target itests/hive-unit/target itests/custom-serde/target
itests/util/target hcatalog/target hcatalog/storage-handlers/hbase/target
hcatalog/server-extensions/target hcatalog/core/target
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target
hcatalog/hcatalog-pig-adapter/target hwi/target common/target common/src/gen
contrib/target service/target serde/target beeline/target odbc/target
cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1555274.
At revision 1555274.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12597571
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt,
HIVE-2599.2.patch.txt
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862231#comment-13862231
]
Swarnim Kulkarni commented on HIVE-2599:
Attached is the latest patch rebased with the master. The patch should apply
cleanly now.
[~brocknoland] On your question about inserts, I think I might have
misunderstood you a little bit. I ran the following queries to test inserts on
composite keys and was able to do it successfully.
{noformat}
CREATE EXTERNAL TABLE test_table_1(key structpersonId:string,value:string,
data string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:value)
TBLPROPERTIES (hbase.table.name =
hbase_test_table_1,hbase.composite.key.class=com.test.hive.TestHBaseCompositeKey)
select * from test_table_1;
{personid:person1,value:value1} 1385435417948
{personid:person2,value:value2} 1386691798261
{personid:person3,value:value3} 1387481795304
{personid:person4,value:value4} 1386705359123
{personid:person5,value:value5} 1386972894836
..
CREATE TABLE test_table_2(key structpersonId:string,value:string, value
string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.hbase.HBaseSerDe'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf1:val)
TBLPROPERTIES (hbase.table.name = hbase_test_table_2);
INSERT OVERWRITE TABLE test_table_2 select key,data from test_table_1;
14/01/04 00:32:33 INFO ql.Driver: Launching Job 1 out of 1
14/01/04 00:32:58 INFO exec.Task: 2014-01-04 00:32:58,720 Stage-0 map = 0%,
reduce = 0%
2014-01-04 00:33:29,930 Stage-0 map = 100%, reduce = 100%, Cumulative CPU 5.48
sec
select * from test_table_2;
{personid:person1,value:value1} 1385435417948
{personid:person2,value:value2} 1386691798261
{personid:person3,value:value3} 1387481795304
{personid:person4,value:value4} 1386705359123
{personid:person5,value:value5} 1386972894836
..
{noformat}
If this is what you meant, then yes the patch will handle both select and
inserts. If not, then please let me know so that I will log a new bug and
tackle it accordingly.
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt,
HIVE-2599.2.patch.txt
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key and make data available. I noticed John Sichi mentioned in
HIVE-1228 that this would be a separate issue but I cant find any follow up.
How feasible is this in the HBaseStorageHandler?
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13862233#comment-13862233 ] Swarnim Kulkarni commented on HIVE-2599: As a side note, I also ran into HIVE-4515 and HIVE-5680 while testing the master. Tackling those next... Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.3.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13797506#comment-13797506 ] Nick Dimiduk commented on HIVE-2599: Hi Swarnim, Would you mind taking a look at HBASE-8693? Specifically, HBase now provides a Struct for defining compound keys, with support for composition using both field-width and delimiter-based separation. It would be great to get these type encoders supported as first-class citizens for bridging the gap between Hive and HBase. Thanks, Nick Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13746743#comment-13746743 ] Swarnim Kulkarni commented on HIVE-2599: This patch has been available for quite sometime and also passes all tests on Hive QA. If someone gets a chance to review this, I will really appreciate that. Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737401#comment-13737401 ] Swarnim Kulkarni commented on HIVE-2599: This should be ready for review. If someone has a chance to take a look, that will be great! Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig Assignee: Swarnim Kulkarni Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt, HIVE-2599.2.patch.txt It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13737688#comment-13737688
]
Hive QA commented on HIVE-2599:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12597571/HIVE-2599.2.patch.txt
{color:green}SUCCESS:{color} +1 2850 tests passed
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/408/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/408/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt,
HIVE-2599.2.patch.txt
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key and make data available. I noticed John Sichi mentioned in
HIVE-1228 that this would be a separate issue but I cant find any follow up.
How feasible is this in the HBaseStorageHandler?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13722112#comment-13722112
]
Hive QA commented on HIVE-2599:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594616/HIVE-2599.2.patch.txt
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 2737 tests executed
*Failed tests:*
{noformat}
org.apache.hcatalog.pig.TestHCatStorer.testMultiPartColsInData
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/219/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/219/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
Assignee: Swarnim Kulkarni
Attachments: HIVE-2599.1.patch.txt, HIVE-2599.2.patch.txt
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key and make data available. I noticed John Sichi mentioned in
HIVE-1228 that this would be a separate issue but I cant find any follow up.
How feasible is this in the HBaseStorageHandler?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[
https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422237#comment-13422237
]
Swarnim Kulkarni commented on HIVE-2599:
If your composite keys are delimited by a separator, here is a possible way to
query them in hive:
{noformat}
CREATE EXTERNAL TABLE hbase_table_1(key structa:string,b:string,c:string,
value string)
ROW FORMAT DELIMITED
COLLECTION ITEMS TERMINATED BY '~'
STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES (hbase.columns.mapping = :key,test-family:test-qual)
TBLPROPERTIES (hbase.table.name = SIMPLE_TABLE);
{noformat}
Basically what this means is that the composite key here should be mapped to a
struct and we specifying that the keys in the composite key are separated by a
~. After doing this, querying for the individual keys in the composite keys
should be as simple as:
{noformat}
select key.a,key.b,key.c from hbase_table_1;
{noformat}
Support Composit/Compound Keys with HBaseStorageHandler
---
Key: HIVE-2599
URL: https://issues.apache.org/jira/browse/HIVE-2599
Project: Hive
Issue Type: Improvement
Components: HBase Handler
Affects Versions: 0.8.0
Reporter: Hans Uhlig
It would be really nice for hive to be able to understand composite keys from
an underlying HBase schema. Currently we have to store key fields twice to be
able to both key and make data available. I noticed John Sichi mentioned in
HIVE-1228 that this would be a separate issue but I cant find any follow up.
How feasible is this in the HBaseStorageHandler?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422361#comment-13422361 ] Hans Uhlig commented on HIVE-2599: -- Do I need to do this for all fields then? Any suggestions for binary? Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2599) Support Composit/Compound Keys with HBaseStorageHandler
[ https://issues.apache.org/jira/browse/HIVE-2599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422396#comment-13422396 ] Swarnim Kulkarni commented on HIVE-2599: For all parts of your key? yes. For binary, using :key#b in the columns mapping should work. Support Composit/Compound Keys with HBaseStorageHandler --- Key: HIVE-2599 URL: https://issues.apache.org/jira/browse/HIVE-2599 Project: Hive Issue Type: Improvement Components: HBase Handler Affects Versions: 0.8.0 Reporter: Hans Uhlig It would be really nice for hive to be able to understand composite keys from an underlying HBase schema. Currently we have to store key fields twice to be able to both key and make data available. I noticed John Sichi mentioned in HIVE-1228 that this would be a separate issue but I cant find any follow up. How feasible is this in the HBaseStorageHandler? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
