[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134621#comment-14134621 ] Swarnim Kulkarni commented on HIVE-6148: RB: https://reviews.apache.org/r/25669 > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135336#comment-14135336 ] Hive QA commented on HIVE-6148: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12668872/HIVE-6148.1.patch.txt {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6277 tests executed *Failed tests:* {noformat} org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/818/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/818/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-818/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12668872 > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141281#comment-14141281 ] Swarnim Kulkarni commented on HIVE-6148: [~brocknoland][~xuefuz] If you guys get a chance to review this, I would highly appreciate that. The above one test failure is unrelated to my patch. > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141309#comment-14141309 ] Brock Noland commented on HIVE-6148: Thank you! Comments on RB. > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14146978#comment-14146978 ] Hive QA commented on HIVE-6148: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12670972/HIVE-6148.2.patch.txt {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6347 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.ql.parse.TestParse.testParse_union {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/967/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/967/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-967/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12670972 > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt, HIVE-6148.2.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147754#comment-14147754 ] Swarnim Kulkarni commented on HIVE-6148: The above test failure is unrelated to the change. > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt, HIVE-6148.2.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14148427#comment-14148427 ] Brock Noland commented on HIVE-6148: Patch looks good and I am +1, but I still see: {noformat} @@ -1403,6 +1484,8 @@ private void deserializeAndSerializeHiveAvro(HBaseSerDe serDe, Result r, Put p, assertNotNull(fieldData); assertEquals(expectedFieldsData[j], fieldData.toString().trim()); } + +SerDeUtils.getJSONString(row, soi); // Now serialize Put put = ((PutWritable) serDe.serialize(row, soi)).getPut(); {noformat} which doesn't seem to make sense? > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt, HIVE-6148.2.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14150139#comment-14150139 ] Hive QA commented on HIVE-6148: --- {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12671376/HIVE-6148.3.patch.txt {color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6357 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority2 {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1001/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1001/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1001/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12671376 > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt, HIVE-6148.2.patch.txt, > HIVE-6148.3.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14151846#comment-14151846 ] Swarnim Kulkarni commented on HIVE-6148: The failed test seems flaky and unrelated to my changes here. > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt, HIVE-6148.2.patch.txt, > HIVE-6148.3.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14152287#comment-14152287 ] Brock Noland commented on HIVE-6148: +1 > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Attachments: HIVE-6148.1.patch.txt, HIVE-6148.2.patch.txt, > HIVE-6148.3.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14158954#comment-14158954 ] Lefty Leverenz commented on HIVE-6148: -- By the time Hive 0.15 is released, this should be documented in the wiki: * [Hive HBase Integration | https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration] > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni >Assignee: Swarnim Kulkarni > Labels: TODOC15 > Fix For: 0.15.0 > > Attachments: HIVE-6148.1.patch.txt, HIVE-6148.2.patch.txt, > HIVE-6148.3.patch.txt > > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863520#comment-13863520 ] Sergey Shelukhin commented on HIVE-6148: Can you elaborate? HBase provides an option for structured keys w/type support and correct multi-field ordering (as of 0.96 IIRC, [~ndimiduk] might know more). Using these, if they fit the use case, might be nice for both sides (they are already there so no new code, and this will provide the real use case for them). We (ok, at least myself) can volunteer to fix things quickly to make it work :) > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863648#comment-13863648 ] Swarnim Kulkarni commented on HIVE-6148: Hi Sergey, This bug is similar to other two bugs[1][2] that I logged to get an out-of-the-box support for querying structs stored in HBase. The other two issues mainly deal with supporting protobufs, thrift and avros which can be deeply nested while this one was for simple structs that don't fall in either category (fix being pretty similar to one suggested in HIVE-2599). I kind of already have something which is working but I can completely throw it away if we can come up with a better strategy. :) [1] https://issues.apache.org/jira/browse/HIVE-3555 [2] https://issues.apache.org/jira/browse/HIVE-6147 > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HIVE-6148) Support arbitrary structs stored in HBase
[ https://issues.apache.org/jira/browse/HIVE-6148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13863738#comment-13863738 ] Nick Dimiduk commented on HIVE-6148: How is this not a duplicate of HIVE-3211? I think it would be trivial to create Avro and Protobuf implementations of [{{org.apache.hadoop.hbase.types.DataType}}|https://github.com/apache/hbase/blob/hbase-0.96.0/hbase-common/src/main/java/org/apache/hadoop/hbase/types/DataType.java]. The details will be similar to what's already provided in [{{Struct}}|https://github.com/apache/hbase/blob/hbase-0.96.0/hbase-common/src/main/java/org/apache/hadoop/hbase/types/Struct.java] (which I suggested you examine over in HIVE-2599). If you build on top of that interface, all HBase users will benefit from from you code, not just Hive + HBase users. > Support arbitrary structs stored in HBase > - > > Key: HIVE-6148 > URL: https://issues.apache.org/jira/browse/HIVE-6148 > Project: Hive > Issue Type: Improvement > Components: HBase Handler >Affects Versions: 0.12.0 >Reporter: Swarnim Kulkarni > > We should add support to be able to query arbitrary structs stored in HBase. -- This message was sent by Atlassian JIRA (v6.1.5#6160)