Thanks JM. Hmm... TestRegionObserverScannerOpenHook has not failed on jenkins (both Apache or Andy EC2 setup).
The exception just means that a region either moved or was split and has not been reassigned, yet. Unless this *only* happens in 0.94.9 and did not happen before I do not think this is a reason for concern. -- Lars ________________________________ From: Jean-Marc Spaggiari <[email protected]> To: [email protected]; lars hofhansl <[email protected]> Sent: Tuesday, June 25, 2013 3:51 PM Subject: Re: [ANNOUNCE] The 1st hbase 0.94.9 release candidate is available for download Here is my feedback. - Downloaded the files, checked the signature, uncompressed, checked changes.txt => Fine - Checked the documentation => One page not found under "Ref Guide (multi-page)" link on the index page. Should it be there? - Ran the test suite on 2 different servers, TestRegionObserverScannerOpenHook failed 10 times in a row with 8 thread, 4, 2 and 1. - IntegrationTestLoadAndVerify => REFERENCES_CHECKED=985565 with 0.94.9 and REFERENCES_CHECKED=9855611 with 0.94.8. Don't know what to deduct. - integrationTestBigLinkedList => Fine - HBCK still reporting fine after all those tests. However, I have this exception many times on the logs, every 3 seconds: 2013-06-24 22:24:01,159 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: Closing scanner for IntegrationTestLoadAndVerify,\xA8k\xCA\x1A\xF2\x86\xBC\x8A,1372126832536.31c325cb8ae32b1ed0f4c4667d787165. org.apache.hadoop.hbase.NotServingRegionException: Region is not online: IntegrationTestLoadAndVerify,\xA8k\xCA\x1A\xF2\x86\xBC\x8A,1372126832536.31c325cb8ae32b1ed0f4c4667d787165. at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:3509) at org.apache.hadoop.hbase.regionserver.HRegionServer$ScannerListener.leaseExpired(HRegionServer.java:2740) at org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:100) at java.lang.Thread.run(Thread.java:662) - Performed a rolling restart for the RS. 2500 regions to move around... => Fine. Took 74 minutes. Changed to regoin_mover to multhread, took 16 minutes. - Looked at the master and region logs, everything seems to be ok. - Merged all the regions from a table to a single region. - Checked HBCK, one issue on one table which is not the one merged. Issue was not there before the restarts, etc. Few hbck repairs repaired everything. - Logs still good, table merged correctly. - Triggered a major compaction, got the table splitted back to 36 regions. - Ran the balalancer, region got spreaded over the nodes. => Fine, but had to run it twice. First time 50% of the regions stayed assigned to one RS. - Stopped, restored my customed balancer, restart, rebalanced => Fine. - Create table, put, scan, get, disable, drop, balance, list, truncate_preserve, truncate from shell => Fine - Ran one MR job (416 mappers), went well, nothing in the logs. So to summarize: +1 ;) Even is there is 2 things to verify (TestRegionObserverScannerOpenHook and NotServingRegionException). I don't think they related to this release. I will post the performances comparison later tonight or tomorrow. (Need to add 0.94.8 also since I'm late), JM 2013/6/24 Jean-Marc Spaggiari <[email protected]>: > Oh! Cool! A new toy for the week! ;) > > Thanks Lars! I will start right now to test it! (And I will do the > remaining perf testing just after or in parallel...) > > 2013/6/24 lars hofhansl <[email protected]>: >> The 1st 0.94.9 RC is available for download at >> http://people.apache.org/~larsh/hbase-0.94.9-rc0/ >> Signed with my code signing key: C7CFE328 >> >> Like the previous point releases, 0.94.9 is a primarily a bug fix release. >> This RC is comparatively small with 26 issues resolved against it: >> [HBASE-8453] - TestImportExport failing again due to configuration issues >> [HBASE-8494] - TestRemoteAdmin#testClusterStatus should not assume >>'requests' does not change >> [HBASE-8522] - Archived hfiles and old hlogs may be deleted immediately >>by HFileCleaner, LogCleaner in HMaster >> [HBASE-8555] - FilterList correctness may be affected by random ordering >>of sub-filter(list) >> [HBASE-8590] - [0.94] BlockingMetaScannerVisitor should check for parent >>meta entry while waiting for split daughter >> [HBASE-8639] - Poor performance of htable#getscanner in multithreaded >>environment due to DNS.getDefaultHost() being called in >>ScannerCallable#prepare() >> [HBASE-8640] - ServerName in master may not initialize with the >>configured ipc address of hbase.master.ipc.address >> [HBASE-8655] - Backport to 94 - HBASE-8346(Prefetching .META. rows in >>case only when useCache is set to true) >> [HBASE-8656] - Rpc call may not be notified in SecureClient >> [HBASE-8671] - Per-region WAL breaks CP backwards compatibility in 0.94 >>for non-enabled case >> [HBASE-8684] - Table Coprocessor can't access external HTable by default >> [HBASE-8700] - IntegrationTestBigLinkedList can fail due to random >>number collision >> [HBASE-8724] - [0.94] ExportSnapshot should not use hbase.tmp.dir as a >>staging dir on hdfs >> [HBASE-8742] - HTableDescriptor Properties not preserved when cloning >> [HBASE-8743] - upgrade hadoop-23 version to 0.23.7 >> [HBASE-8749] - Potential race condition between >>FSUtils.renameAndSetModifyTime() and HFile/LogCleaner >> [HBASE-8762] - Performance/operational penalty when calling HTable.get >>with a list of one Get >> [HBASE-8783] - RSSnapshotManager.ZKProcedureMemberRpcs may be >>initialized with the wrong server name >> [HBASE-5083] - Backup HMaster should have http infoport open with link >>to the active master >> [HBASE-8609] - Make the CopyTable support startRow, stopRow options >> [HBASE-8636] - Backport KeyValue Codec to 0.94 (HBASE-7413) >> [HBASE-8683] - Add major compaction support in CompactionTool >> [HBASE-8692] - [AccessController] Restrict HTableDescriptor enumeration >> [HBASE-8702] - Make WALEditCodec pluggable >> [HBASE-8504] - HTable.getRegionsInRange() should provide a non-cached API >> [HBASE-8603] - Backport HBASE-6921 to 0.94 >> >> The full list of changes is also available here: >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12324431 >> >> Please try out the RC, check out the doc, take it for a spin, etc, and vote >> +1/-1 by July 2nd on whether we should release this as 0.94.9. >> >> The release testing spreadsheet is available here: >> https://docs.google.com/spreadsheet/ccc?key=0AvLqcVIqNtlTdGFHaktDR0FqLWdtQWZwdncyVzE3Z2c#gid=0 >> (if you test this release, please add your test to this spreadsheet, so that >> we can gauge the coverage) >> >> Thanks. >> >> -- Lars >> >> ps. I will likely be without access to the Internet for the next five days
