I'm reducing keys by regions and then building a Scan with
a. start/stop = minkey/max+1
b. 100 cache rows
c. cache blocks false
d. configuring scan like this
public static void setRowFilters(Scan scan, Iterator<byte[]> keys) {
FilterList filterlist = new FilterList(Operator.MUST_PASS_ONE);
for (; keys.hasNext(); ) {
filterlist.addFilter(new RowFilter(CompareOp.EQUAL, new
BinaryComparator(keys.next())));
}
scan.setFilter(filterlist);
}
where the iterator contains the keys I want
Are you saying I should also re-instantiate the htable?
Also, looking at Regionserver stack trace shows the following blocked threads
"IPC Server handler 19 on 60020" daemon prio=10 tid=0x0000000053838000
nid=0x3224 waiting for monitor entry [0x0000000045799000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2322)
- waiting to lock <0x0000000782a96ad8> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1823)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
"IPC Server handler 18 on 60020" daemon prio=10 tid=0x0000000053835800
nid=0x3223 waiting for monitor entry [0x0000000045698000]
java.lang.Thread.State: BLOCKED (on object monitor)
at
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.next(HRegion.java:2322)
- waiting to lock <0x000000071d024b60> (a
org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1823)
at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
"IPC Server handler 17 on 60020" daemon prio=10 tid=0x0000000053833800
nid=0x3222 waiting for monitor entry [0x0000000045597000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.hadoop.io.compress.CodecPool.payback(CodecPool.java:75)
- locked <0x00000006fbe527f8> (a java.util.HashMap)
at
org.apache.hadoop.io.compress.CodecPool.returnDecompressor(CodecPool.java:152)
at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.returnDecompressor(Compression.java:245)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:1098)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:1036)
- locked <0x000000074aa2a950> (a [B)
at
org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1276)
at
org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:115)
at
org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:255)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:394)
________________________________
From: Jean-Daniel Cryans <[email protected]>
To: [email protected]
Sent: Friday, September 16, 2011 10:47 AM
Subject: Re: REcovering from SocketTimeout during scan in 90.3
Yeah this should be at the HTable level...
Your solution sounds right.
How did you get yourself in this situation btw? Using a large scanner
caching value plus filters?
J-D
On Fri, Sep 16, 2011 at 10:18 AM, Douglas Campbell <[email protected]> wrote:
> What's the best way to recover from this?
>
> Set the start row to last sucessfully processed row + 1 and re-init scanner?
>
> java.io.IOException: java.net.SocketTimeoutException: Call to $host failed on
> socket timeout exception:
> java.net.SocketTimeoutException: 60000 millis timeout while waiting for
> channel to be ready for read. ch
> : java.nio.channels.SocketChannel[connected local=/216.109.127.129:53241
> remote=hostname/ip:60020]
>
> at
> org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1232)
