ZK-less assignment - meta is not being updated

Hi all, Running HBase 1.1.2: I'm running into an issue where during forced region failover (kill -9 on the regionserver), the new region location is not being updated in meta (and never is updated, even after waiting for some time). ERROR: Region { meta =>

current leaseholder is trying to recreate file error with ProcedureV2

Hi all, I'm getting a strange error during table creation / disable in HBase 1.1.2: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): failed to create file /hbase/MasterProcWALs/state-0073.log for

Re: hbase.bucketcache.percentage.in.combinedcache with ioengine= on HBase 0.98

Thanks Nick! Unfortunately, I'm stuck with 0.98. Perhaps it's something I should look into backporting if all else fails... Would also bigger great if anyone had any insight in the differences between offheap and file:/ in terms of configuration. When RAM to tmpfs was used as the workaround for

Re: Using HBase with a shared filesystem (gluster, nfs, s3, etc)

movement? On Tue, Aug 25, 2015 at 10:40 AM, Ted Yu yuzhih...@gmail.com wrote: Please see http://hbase.apache.org/book.html#regions.arch.assignment On Tue, Aug 25, 2015 at 10:37 AM, donmai dood...@gmail.com wrote: NFS 0.98.10 Will get to you as soon as I am able, on travel Is my

Re: Using HBase with a shared filesystem (gluster, nfs, s3, etc)

balanced are the table regions in your cluster ? Cheers On Thu, Aug 27, 2015 at 6:15 AM, donmai dood...@gmail.com wrote: I figured out the issue - the reason wasn't actually region movement taking a while, the balancer is actually the thing taking forever: 2015-08-27 12:50:13,582 DEBUG

Re: Using HBase with a shared filesystem (gluster, nfs, s3, etc)

thread since recent discussion is no longer about shared filesystem. If you can take a look at TestStochasticLoadBalancer and add a test which reproduces what you saw, that would help us troubleshoot. Cheers On Thu, Aug 27, 2015 at 6:56 AM, donmai dood...@gmail.com wrote: Very unbalanced

Re: Sporadic incorrect table directory structure when running restore_snapshot

Can't seem to find anything in the logs that even matches /foo/foo being created. Any ideas as to where I can start looking in the codebase to figure out exactly what's going on? On Wed, Aug 26, 2015 at 1:20 PM, donmai dood...@gmail.com wrote: DEBUG is on, trying to look through the logs again

Sporadic incorrect table directory structure when running restore_snapshot

Hi, Occasionally when I run restore_snapshot on HBase 0.98.10, it appears that the table directory structure created by the restore_snapshot command is not correct: rootdir/data/default/foo Is what it should be, but I end up with rootdir/data/default/foo/foo The extra foo should not be

Re: Sporadic incorrect table directory structure when running restore_snapshot

DEBUG is on, trying to look through the logs again. Thanks! On Wed, Aug 26, 2015 at 12:59 PM, Ted Yu yuzhih...@gmail.com wrote: Did you enable DEBUG logging ? Can you find the logs for one such occurrence and pastebin relevant portion ? Thanks On Wed, Aug 26, 2015 at 9:58 AM, donmai dood

Re: Using HBase with a shared filesystem (gluster, nfs, s3, etc)

filesystem you use which hbase release you use master log snippet for the long region assignment Thanks On Tue, Aug 25, 2015 at 9:30 AM, donmai dood...@gmail.com wrote: Hi, I'm curious about how exactly region movement works with regard to data transfer. To my understanding from the docs

Using HBase with a shared filesystem (gluster, nfs, s3, etc)

Hi, I'm curious about how exactly region movement works with regard to data transfer. To my understanding from the docs given an HDFS-backed cluster, a region movement / transition involves changing things in meta only, all data movement for locality is handled by HDFS. In the case where rootdir

Re: readAtOffset error when reading from HFiles

Hi Ted - would you happen to have any insight as to what issues might cause a positional read failure for an HFile? Running the HFile tool seems to have normal output. On Wed, May 13, 2015 at 12:40 PM, donmai dood...@gmail.com wrote: Ahhh, thanks for that. Yep, all flushes will (or should

Re: readAtOffset error when reading from HFiles

(so it is not offset): -w,--seekToRow arg Seek to this row and print all the kvs for this row only Do you store all your data on s3 ? Cheers On Tue, May 12, 2015 at 8:50 AM, donmai dood...@gmail.com wrote: Actually, looking deeper into it, things don't seem

readAtOffset error when reading from HFiles

Hi, I'm getting this error when trying to read from HFiles: http://pastebin.com/SJci7uQM Any idea what's going on here? Thanks!

Re: readAtOffset error when reading from HFiles

Thanks for the help! I am getting 'okay' for everything. I am using S3 for the store, do you think it could be an issue with how S3 is reading the files? On Tue, May 12, 2015 at 9:45 AM, Talat Uyarer ta...@uyarer.com wrote: Hi Donmai, Seems to be something wrong with this file. Can you try

Re: readAtOffset error when reading from HFiles

Yu yuzhih...@gmail.com wrote: What release of hbase are you using ? Please read http://hbase.apache.org/book.html#hfile where you can find description about HFile tool. This tool would allow you to investigate given HFile. Cheers On Tue, May 12, 2015 at 6:02 AM, donmai dood...@gmail.com

Re: readAtOffset error when reading from HFiles

/5cce51bd0bcc8f7507c7e594b73d2d15/family/ec3d1516dba0447e875d489f3ad8bdc0' seeking to position '1693329' Am I using the HFIle command correctly? On Tue, May 12, 2015 at 11:09 AM, donmai dood...@gmail.com wrote: Thanks Ted, this actually helped me out alot! I'm running 0.98.6 and was able to determine