Hi, I missed that comment about not being able to see my test code. Sorry about that, and here it is: https://dl.dropbox.com/u/608155/HBase/Test.java
It's just a quick hack and nothing clever, but has 10000 threads (500 running concurrently) running the scan and I have repeated it many times with no errors (it is only 10 rows, but easy to modify). Cheers, Tim On Tue, Apr 27, 2010 at 3:31 PM, Stack <st...@duboce.net> wrote: > You may be running into HBASE-2481? > St.Ack > > On Tue, Apr 27, 2010 at 1:30 AM, steven zhuang > <steven.zhuang.1...@gmail.com> wrote: > > the first thread can be found at: > > http://permalink.gmane.org/gmane.comp.java.hadoop.hbase.user/10074 > > > > After some dig, it seems that the problem is caused by long > pause > > between two "scanner.next()" call. > > > > In my case the program has to spent a relatively long while to > > process one row, when it calls "scanner.next()" again, seems that the > > returned Result will be null even if there should be more rows in the > > tables. The rowcaching is set to 1. > > I have checked some of the source code, seems there is some > > mechanism which will call the close() method of the ClientScanner, but I > am > > still checking. > > I don't know if there is a certain timeout on > > ClientScanner/ScannerCallable after a row has been successfully returned, > > seems that timeout cause my problem here. > > > > Any reply is appreciated. > > > > > > On Fri, Apr 23, 2010 at 11:10 AM, steven zhuang < > > steven.zhuang.1...@gmail.com> wrote: > > > >> hi, > >> sorry I start another thread here. This mail is actually > answer > >> to a previous thread "multiple scanners on same table will cause > problem? > >> Scan results change among different tries.". > >> the mail system kept saying that I am spamming, now it seems > that > >> it's right! :) > >> > >> here is my reply to people in that thread: > >> > >> I don't know if there is a limit on reads to a single row/region > in > >> HBase, but if there is, I might have exceeded that limit. :( > >> in my case, there are hundreds of rows, with dozens of kilos of > cells > >> in a row(a 256 MB region may contain 10- rows). for each row, I started > a > >> thread on each CF, there are 8 of them, so there might be dozens of > scanners > >> on the same region. > >> and, to Tim, I could not see your attached mail, my test code is > >> pasted below, it just iterate on the rows and column families, output > all > >> the cells. > >> > >> private void doScan() throws Exception { > >> if (null == CopyOfTestTTT234.table) { > >> return; > >> } > >> Scan s = new Scan(); > >> s.setStartRow("aaa".getBytes()); > >> s.setStopRow("ccc".getBytes()); > >> s.setCaching(CopyOfTestTTT234.ROWCACHING); //it's 1 here. > >> ResultScanner scanner = CopyOfTestTTT234.table.getScanner(s); > >> while (true) { > >> Result row = scanner.next(); > >> if(null==row) break; > >> String rowKey = new String(row.getRow()); > >> NavigableMap<byte[], NavigableMap<byte[], byte[]>> fm = row > >> .getNoVersionMap(); > >> while (fm.size() > 0) { > >> Entry<byte[], NavigableMap<byte[], byte[]>> ee = fm > >> .pollFirstEntry(); > >> String fName = new String(ee.getKey()); > >> NavigableMap<byte[], byte[]> ff = ee.getValue(); > >> while (ff.size() > 0) { > >> Entry<byte[], byte[]> cell = ff.pollFirstEntry(); > >> String key = new String(cell.getKey()); > >> String val = new String(cell.getValue()); > >> System.out.println(Thread.currentThread().hashCode() + "\t" > >> + rowKey + "\t" + fName + "\t" + key + "\t" + val); > >> } > >> } > >> } > >> } > >> > > >
