[jira] [Commented] (HIVE-17002) decimal (binary) is not working when creating external table for hbase
[ https://issues.apache.org/jira/browse/HIVE-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288046#comment-16288046 ] Artur Tamazian commented on HIVE-17002: --- I don't have time right now, but I looked at your patch and it's almost exactly the same I ended up doing for our installation. Only difference is in LazyDioHiveDecimal::init I did data field initialization like this: {code} data = new HiveDecimalWritable(HiveDecimal.create(Bytes.toBigDecimal(bytes.getData(), start, length))); {code} > decimal (binary) is not working when creating external table for hbase > -- > > Key: HIVE-17002 > URL: https://issues.apache.org/jira/browse/HIVE-17002 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 > Environment: HBase 1.2.0, Hive 2.1.1 >Reporter: Artur Tamazian >Assignee: Naveen Gangam > > I have a table in Hbase which has a column stored using > Bytes.toBytes((BigDecimal) value). Hbase version is 1.2.0 > I'm creating an external table in hive to access it like this: > {noformat} > create external table `Users`(key int, ..., `example_column` decimal) > stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > with serdeproperties ("hbase.columns.mapping" = ":key, > db:example_column") > tblproperties("hbase.table.name" = > "Users","hbase.table.default.storage.type" = "binary"); > {noformat} > Table is created without errors. After that I try running "select * from > users;" and see this error: > {noformat} > org.apache.hive.service.cli.HiveSQLException:java.io.IOException: > java.lang.RuntimeException: java.lang.RuntimeException: Hive Internal Error: > no LazyObject for > org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyHiveDecimalObjectInspector@1f18cebb:25:24 > > > org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:484 > > > org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:308 > > > org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:847 > >sun.reflect.GeneratedMethodAccessor11:invoke::-1 > > sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43 > >java.lang.reflect.Method:invoke:Method.java:498 > > org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78 > > > org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36 > > > org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63 > >java.security.AccessController:doPrivileged:AccessController.java:-2 >javax.security.auth.Subject:doAs:Subject.java:422 > > org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1698 > > > org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59 > >com.sun.proxy.$Proxy33:fetchResults::-1 >org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:504 > > org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:698 > > > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1717 > > > org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1702 > >org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39 >org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39 > > org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56 > > > org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286 > > > java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142 > > > java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:617 > >java.lang.Thread:run:Thread.java:748 >*java.io.IOException:java.lang.RuntimeException: > java.lang.RuntimeException: Hive Internal Error: no LazyObject for > org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyHiveDecimalObjectInspector@1f18cebb:27:2 > >org.apache.hadoop.hive.ql.exec.FetchTask:fetch:FetchTask.java:164 >org.apache.hadoop.hive.ql.Driver:getResults:Driver.java:2098 > > org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:479 > >*java.lang.RuntimeException:java.lang.RuntimeException: Hive Internal > Error: no LazyObject for > org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyHiveDecimalObjectInspector@1f18cebb:43:16 > > > org.apache.hadoop.hive.serde2.lazy.LazyStruct:initLazyFields:LazyStruct.java:172 > >
[jira] [Commented] (HIVE-15883) HBase mapped table in Hive insert fail for decimal
[ https://issues.apache.org/jira/browse/HIVE-15883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072145#comment-16072145 ] Artur Tamazian commented on HIVE-15883: --- Possibly related issue: https://issues.apache.org/jira/browse/HIVE-17002 > HBase mapped table in Hive insert fail for decimal > -- > > Key: HIVE-15883 > URL: https://issues.apache.org/jira/browse/HIVE-15883 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 1.1.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam > Attachments: HIVE-15883.patch > > > CREATE TABLE hbase_table ( > id int, > balance decimal(15,2)) > ROW FORMAT DELIMITED > COLLECTION ITEMS TERMINATED BY '~' > STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' > WITH SERDEPROPERTIES ( > "hbase.columns.mapping"=":key,cf:balance#b"); > insert into hbase_table values (1,1); > > Diagnostic Messages for this Task: > Error: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while > processing row {"tmp_values_col1":"1","tmp_values_col2":"1"} > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1783) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row {"tmp_values_col1":"1","tmp_values_col2":"1"} > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507) > at > org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170) > ... 8 more > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > org.apache.hadoop.hive.serde2.SerDeException: java.lang.RuntimeException: > Hive internal error. > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:733) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) > at > org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) > at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) > at > org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:97) > at > org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497) > ... 9 more > Caused by: org.apache.hadoop.hive.serde2.SerDeException: > java.lang.RuntimeException: Hive internal error. > at > org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:286) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:668) > ... 15 more > Caused by: java.lang.RuntimeException: Hive internal error. > at > org.apache.hadoop.hive.serde2.lazy.LazyUtils.writePrimitive(LazyUtils.java:328) > at > org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:220) > at > org.apache.hadoop.hive.hbase.HBaseRowSerializer.serializeField(HBaseRowSerializer.java:194) > at > org.apache.hadoop.hive.hbase.HBaseRowSerializer.serialize(HBaseRowSerializer.java:118) > at > org.apache.hadoop.hive.hbase.HBaseSerDe.serialize(HBaseSerDe.java:282) > ... 16 more -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17002) decimal (binary) is not working when creating external table for hbase
[ https://issues.apache.org/jira/browse/HIVE-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Artur Tamazian updated HIVE-17002: -- Description: I have a table in Hbase which has a column stored using Bytes.toBytes((BigDecimal) value). Hbase version is 1.2.0 I'm creating an external table in hive to access it like this: {noformat} create external table `Users`(key int, ..., `example_column` decimal) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' with serdeproperties ("hbase.columns.mapping" = ":key, db:example_column") tblproperties("hbase.table.name" = "Users","hbase.table.default.storage.type" = "binary"); {noformat} Table is created without errors. After that I try running "select * from users;" and see this error: {noformat} org.apache.hive.service.cli.HiveSQLException:java.io.IOException: java.lang.RuntimeException: java.lang.RuntimeException: Hive Internal Error: no LazyObject for org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyHiveDecimalObjectInspector@1f18cebb:25:24 org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:484 org.apache.hive.service.cli.operation.OperationManager:getOperationNextRowSet:OperationManager.java:308 org.apache.hive.service.cli.session.HiveSessionImpl:fetchResults:HiveSessionImpl.java:847 sun.reflect.GeneratedMethodAccessor11:invoke::-1 sun.reflect.DelegatingMethodAccessorImpl:invoke:DelegatingMethodAccessorImpl.java:43 java.lang.reflect.Method:invoke:Method.java:498 org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:78 org.apache.hive.service.cli.session.HiveSessionProxy:access$000:HiveSessionProxy.java:36 org.apache.hive.service.cli.session.HiveSessionProxy$1:run:HiveSessionProxy.java:63 java.security.AccessController:doPrivileged:AccessController.java:-2 javax.security.auth.Subject:doAs:Subject.java:422 org.apache.hadoop.security.UserGroupInformation:doAs:UserGroupInformation.java:1698 org.apache.hive.service.cli.session.HiveSessionProxy:invoke:HiveSessionProxy.java:59 com.sun.proxy.$Proxy33:fetchResults::-1 org.apache.hive.service.cli.CLIService:fetchResults:CLIService.java:504 org.apache.hive.service.cli.thrift.ThriftCLIService:FetchResults:ThriftCLIService.java:698 org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1717 org.apache.hive.service.rpc.thrift.TCLIService$Processor$FetchResults:getResult:TCLIService.java:1702 org.apache.thrift.ProcessFunction:process:ProcessFunction.java:39 org.apache.thrift.TBaseProcessor:process:TBaseProcessor.java:39 org.apache.hive.service.auth.TSetIpAddressProcessor:process:TSetIpAddressProcessor.java:56 org.apache.thrift.server.TThreadPoolServer$WorkerProcess:run:TThreadPoolServer.java:286 java.util.concurrent.ThreadPoolExecutor:runWorker:ThreadPoolExecutor.java:1142 java.util.concurrent.ThreadPoolExecutor$Worker:run:ThreadPoolExecutor.java:617 java.lang.Thread:run:Thread.java:748 *java.io.IOException:java.lang.RuntimeException: java.lang.RuntimeException: Hive Internal Error: no LazyObject for org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyHiveDecimalObjectInspector@1f18cebb:27:2 org.apache.hadoop.hive.ql.exec.FetchTask:fetch:FetchTask.java:164 org.apache.hadoop.hive.ql.Driver:getResults:Driver.java:2098 org.apache.hive.service.cli.operation.SQLOperation:getNextRowSet:SQLOperation.java:479 *java.lang.RuntimeException:java.lang.RuntimeException: Hive Internal Error: no LazyObject for org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyHiveDecimalObjectInspector@1f18cebb:43:16 org.apache.hadoop.hive.serde2.lazy.LazyStruct:initLazyFields:LazyStruct.java:172 org.apache.hadoop.hive.hbase.LazyHBaseRow:initFields:LazyHBaseRow.java:122 org.apache.hadoop.hive.hbase.LazyHBaseRow:getField:LazyHBaseRow.java:116 org.apache.hadoop.hive.serde2.lazy.objectinspector.LazySimpleStructObjectInspector:getStructFieldData:LazySimpleStructObjectInspector.java:128 org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator:_evaluate:ExprNodeColumnEvaluator.java:94 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator:evaluate:ExprNodeEvaluator.java:77 org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject:get:ExprNodeGenericFuncEvaluator.java:87 org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPEqual:evaluate:GenericUDFOPEqual.java:103 org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator:_evaluate:ExprNodeGenericFuncEvaluator.java:186 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator:evaluate:ExprNodeEvaluator.java:77 org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator:evaluate:ExprNodeEvaluator.java:65