Based on the stack trace (an NPE in HRegion$RegionScannerImpl.nextRow), it looks like an HBase issue. Here are some questions: 1) What version of Phoenix and HBase are you using? 2) Can you reproduce the problem by issuing a scan in the hbase shell? 3) If yes, then would it be possible for you to try this on the latest version of HBase on the branch you're using? 4) If yes again, then I'd recommend filing a bug with HBase, mentioning (2) as the way you can reproduce the issue. Thanks, James
On Wed, Apr 9, 2014 at 12:56 AM, liujs3 <[email protected]> wrote: > Dear all: > > > > Im a programmer in asainfo-linkage.com of china. > > > > This is a problem for this .my engilish is very poor. > > > > : > > > > I bulkload lots of data into a hbase table. The table have 2000 columns。 > 215G size。 > > when I query by phoenix,the sql is ‘select ID from DW_COC_LABEL_USER_002 > limit 1’ ,wait a long time after,I get a result : > > > > > > 0: jdbc:phoenix:ocdata02> select ID from DW_COC_LABEL_USER_002 LIMIT 1; > +------------+ > | ID | > +------------+ > java.lang.RuntimeException: > com.salesforce.phoenix.exception.PhoenixIOException > at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2440) > at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2074) > at sqlline.SqlLine.print(SqlLine.java:1735) > at sqlline.SqlLine$Commands.execute(SqlLine.java:3683) > at sqlline.SqlLine$Commands.sql(SqlLine.java:3584) > at sqlline.SqlLine.dispatch(SqlLine.java:821) > at sqlline.SqlLine.begin(SqlLine.java:699) > at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:441) > at sqlline.SqlLine.main(SqlLine.java:424) > > > > > > > > > > AND the hbase have the problem : > > > > 2014-04-09 14:56:23 [IPC Server handler 11 on > 60327][org.apache.hadoop.hbase.regionserver.HRegionServer][1228]-[ERROR] > Failed openScanner > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call > openScanner([B@5fa3b19e, > > {"timeRange":[0,1397026662530],"batch":-1,"startRow":"000","stopRow":"02b"," > > totalColumns":1,"cacheBlocks":true,"families":{"F":["_0"]},"maxVersions":1," > caching":100}), rpc version=1, client version=29, > methodsFingerPrint=-1368823753 from 10.1.253.90:44730 after 110349 ms, > since > caller disconnected > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnec > ted(HBaseServer.java:436) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal( > HRegion.java:3980) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3890) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3880) > at > > com.salesforce.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostSc > annerOpen(UngroupedAggregateRegionObserver.java:217) > at > > com.salesforce.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen > (BaseScannerRegionObserver.java:59) > at > > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(R > egionCoprocessorHost.java:1316) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer > .java:2553) > at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > > 2014-04-09 14:56:23 [IPC Server handler 53 on > 60327][org.apache.hadoop.hbase.regionserver.HRegionServer][1228]-[ERROR] > Failed openScanner > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call > openScanner([B@58784167, > > {"timeRange":[0,1397026662530],"batch":-1,"startRow":"f74","stopRow":"f9f"," > > totalColumns":1,"cacheBlocks":true,"families":{"F":["_0"]},"maxVersions":1," > caching":100}), rpc version=1, client version=29, > methodsFingerPrint=-1368823753 from 10.1.253.90:44730 after 110537 ms, > since > caller disconnected > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnec > ted(HBaseServer.java:436) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal( > HRegion.java:3980) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3890) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3880) > at > > com.salesforce.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostSc > annerOpen(UngroupedAggregateRegionObserver.java:217) > at > > com.salesforce.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen > (BaseScannerRegionObserver.java:59) > at > > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(R > egionCoprocessorHost.java:1316) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer > .java:2553) > at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > 2014-04-09 14:56:23 [IPC Server handler 12 on > 60327][org.apache.hadoop.hbase.regionserver.HRegionServer][1228]-[ERROR] > Failed openScanner > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call > openScanner([B@d16e445, > > {"timeRange":[0,1397026662530],"batch":-1,"startRow":"0d7","stopRow":"102"," > > totalColumns":1,"cacheBlocks":true,"families":{"F":["_0"]},"maxVersions":1," > caching":100}), rpc version=1, client version=29, > methodsFingerPrint=-1368823753 from 10.1.253.90:44730 after 110371 ms, > since > caller disconnected > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnec > ted(HBaseServer.java:436) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal( > HRegion.java:3980) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3890) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3880) > at > > com.salesforce.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostSc > annerOpen(UngroupedAggregateRegionObserver.java:217) > at > > com.salesforce.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen > (BaseScannerRegionObserver.java:59) > at > > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(R > egionCoprocessorHost.java:1316) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer > .java:2553) > at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > 2014-04-09 14:56:23 [IPC Server handler 56 on > 60327][org.apache.hadoop.hbase.regionserver.HRegionServer][1228]-[ERROR] > Failed openScanner > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call > openScanner([B@10fcce18, > > {"timeRange":[0,1397026662530],"batch":-1,"startRow":"9be","stopRow":"9e9"," > > totalColumns":1,"cacheBlocks":true,"families":{"F":["_0"]},"maxVersions":1," > caching":100}), rpc version=1, client version=29, > methodsFingerPrint=-1368823753 from 10.1.253.90:44730 after 110327 ms, > since > caller disconnected > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnec > ted(HBaseServer.java:436) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal( > HRegion.java:3980) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3890) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3880) > at > > com.salesforce.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostSc > annerOpen(UngroupedAggregateRegionObserver.java:217) > at > > com.salesforce.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen > (BaseScannerRegionObserver.java:59) > at > > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(R > egionCoprocessorHost.java:1316) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer > .java:2553) > at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > 2014-04-09 14:56:23 [IPC Server handler 8 on > 60327][org.apache.hadoop.hbase. > regionserver.HRegionServer][1228]-[ERROR] Failed openScanner > org.apache.hadoop.hbase.ipc.CallerDisconnectedException: Aborting call > openScanner([B@25ab4bca, > > {"timeRange":[0,1397026662530],"batch":-1,"startRow":"50a","stopRow":"535"," > > totalColumns":1,"cacheBlocks":true,"families":{"F":["_0"]},"maxVersions":1," > caching":100}), rpc version=1, client version=29, > methodsFingerPrint=-1368823753 from 10.1.253.90:44730 after 110330 ms, > since > caller disconnected > at > > org.apache.hadoop.hbase.ipc.HBaseServer$Call.throwExceptionIfCallerDisconnec > ted(HBaseServer.java:436) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal( > HRegion.java:3980) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3890) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3880) > at > > com.salesforce.phoenix.coprocessor.UngroupedAggregateRegionObserver.doPostSc > annerOpen(UngroupedAggregateRegionObserver.java:217) > at > > com.salesforce.phoenix.coprocessor.BaseScannerRegionObserver.postScannerOpen > (BaseScannerRegionObserver.java:59) > at > > org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.postScannerOpen(R > egionCoprocessorHost.java:1316) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer > .java:2553) > at sun.reflect.GeneratedMethodAccessor32.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > > > > org.apache.hadoop.hbase.UnknownScannerException: No lease for > -27544761971936617: lease '-27544761971936617' does not exist > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2 > 614) > at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > 2014-04-09 15:09:23 [IPC Server handler 23 on > 60327][org.apache.hadoop.hbase.regionserver.HRegionServer][2613]-[INFO] > Client tried to access missing scanner 5955469386530140236 (no lease) > 2014-04-09 15:09:23 [IPC Server handler 23 on > 60327][org.apache.hadoop.hbase.regionserver.HRegionServer][1226]-[ERROR] > org.apache.hadoop.hbase.UnknownScannerException: No lease for > 5955469386530140236: lease '5955469386530140236' does not exist > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2 > 614) > at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > 2014-04-09 15:09:23 [IPC Server handler 1 on > 60327][org.apache.hadoop.hbase. > regionserver.HRegionServer][2613]-[INFO] Client tried to access missing > scanner 8159705548427277534 (no lease) > 2014-04-09 15:09:23 [IPC Server handler 1 on > 60327][org.apache.hadoop.hbase. > regionserver.HRegionServer][1226]-[ERROR] > org.apache.hadoop.hbase.UnknownScannerException: No lease for > 8159705548427277534: lease '8159705548427277534' does not exist > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2 > 614) > at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > > > > 2014-04-09 15:09:23 [IPC Server handler 13 on > 60327][org.apache.hadoop.hbase.regionserver.HRegionServer][1226]-[ERROR] > org.apache.hadoop.hbase.DoNotRetryIOException: > DW_COC_LABEL_USER_002,0d7,1397012934302.d93e39821bee9e5ccbd73c301e136955.: > null > at > com.salesforce.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:83) > at > > com.salesforce.phoenix.coprocessor.ScanRegionObserver$2.nextRaw(ScanRegionOb > server.java:303) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2 > 648) > at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > Caused by: java.lang.NullPointerException > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRow(HRegi > on.java:4104) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal( > HRegion.java:4041) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3890) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3880) > at > > com.salesforce.phoenix.coprocessor.ScanRegionObserver$2.nextRaw(ScanRegionOb > server.java:301) > ... 6 more > 2014-04-09 15:09:23 [IPC Server handler 49 on > 60327][org.apache.hadoop.hbase.regionserver.HRegionServer][1226]-[ERROR] > org.apache.hadoop.hbase.DoNotRetryIOException: > DW_COC_LABEL_USER_002,f74,1397012934309.5430b0b2136cddecd67dd6aca389ce91.: > null > at > com.salesforce.phoenix.util.ServerUtil.throwIOException(ServerUtil.java:83) > at > > com.salesforce.phoenix.coprocessor.ScanRegionObserver$2.nextRaw(ScanRegionOb > server.java:303) > at > > org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2 > 648) > at sun.reflect.GeneratedMethodAccessor33.invoke(Unknown Source) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl > .java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine. > java:320) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > Caused by: java.lang.NullPointerException > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRow(HRegi > on.java:4104) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal( > HRegion.java:4041) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3890) > at > > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextRaw(HRegi > on.java:3880) > > > >
