On Sat, Jan 24, 2015 at 5:15 PM, Pun Intended <punintende...@gmail.com> wrote:
> Hello, > > I have noticed lately that my apps started running longer. You are processing more data now? > The longest > running tasks seem all to be requesting data from a single region server. > Has the rate at which you access hbase:mta gone up since when the job ran faster? Anything changed in your processing? Is the trip to hbase:meta what is slowing your jobs (you could add some printout in your maptask but meta looks should be fast usually out of cache). How long do the tasks last? Are they short or long? If long running, then they'll build up a local cache of locations and won't have to go to hbase:meta. St.Ack > That region server read rate is very high in comparison to the read rate of > all the other region servers (1000reqs/sec vs 4-5 reqs/sec elsewhere). That > region server has about the same number of regions as all the rest: 26-27 > regions. Number of store files, total region size, everything else on the > region server seems ok and in the same ranges as the rest of the region > servers. The keys should be evenly distributed - randomly generated > 38-digit numbers. I am doing a simple Hbase scan from all my MR jobs. > > I'd appreciate any suggestions on what to look into it or if you have any > ideas how I can solve this issue. > > Thanks! >
