Region Server Thread with a Single High Idle CPU

2014-12-15 Thread uamadman
I've done a bit of digging and hope someone can shed some light on my particular issue. One and Only One of my region servers after each restart is randomly "Plagued" with a single maxed out CPU-Core and a Read Request chart registering around 40k read requests per second. The remaining 13 dance ar

Re: Region Server Thread with a Single High Idle CPU

2014-12-15 Thread uamadman
hbase:meta,,1.1588230740620282632 < This is the Offending Table. I currently do not know what this means yet. But my basic understanding it is probably an Index of sorts. Jon -- View this message in context: http://apache-hbase.679495.n3.nabble.com/Region-Server-Thread-with-a-Singl

Re: Region Server Thread with a Single High Idle CPU

2014-12-16 Thread uamadman
Currently, hbase-master.com:60010/master-status reports 1500 reads/s when no clients are connected. and 1750-2000 to the .meta when I'm running a load operation. Total R/W spikes to 20k/s when adding all the remaining tables together. I run a multi-core load process that pools 20 cores using happ

Re: Region Server Thread with a Single High Idle CPU

2014-12-16 Thread uamadman
I posted these stats in the first post/ I updated to include the Thrift Server: Cluster Stats: 1 Master/14 Nodes/1 Thrift Server Quad\Dual-Core 2.8-3.0Ghz 6-8gb ram each with a single 500Gb Drive Table Stats: ~150 Tables Total Split into 590 Regions 1 table w/ 2.5Billion Rows 500 Columns 4 tab

Re: Region Server Thread with a Single High Idle CPU

2014-12-19 Thread uamadman
Yes, I tested the following by restarting the cluster and waiting approximately 5-10 minutes for its initial ramp up. There are no clients asking for data. In the following example KVM15 was randomly assigned to serve the META Table. root@KVM15:~# lsof -n | grep :60020- | sed 's/.*->//;s/:.*//' |