Hi Tariq Thanks, this blog is very informational!
I roughly figure out the usage of xceiver. While the weird thing is even according to this blog, I should have far enough xceiever in my full table scan job. Especially if it count the number of storefile instead of real blk_xxxx tfile on nativeFS. Then I only have around 24 storefiles for 24 region since I do major compact them. And then it exceeding 1024 limit during full table scan.... Might need more investigation into my case to check out what wrong with it. > > Hello Raymond, > > You might find this > link<http://blog.cloudera.com/blog/2012/03/hbase-hadoop-xceivers/>helpful. > It explains the problem > and solution in great detail. > > Warm Regards, > Tariq > https://mtariq.jux.com/ > cloudfront.blogspot.com > > > On Thu, Jan 17, 2013 at 12:14 PM, Liu, Raymond > <raymond....@intel.com>wrote: > > > Hi > > > > I have a table with about 24 region on the one regionserver, > > and each region have about 20 block files on hdfs. > > The xceiverCount is set to 1024, I have thought that this is > > quite enough since at most 480 blocks will be opened. > > While when I do a MR job to scan the table, with 24 map task > > each open and scan a different region at the same time, it turn out > > that the DataXceiver is run out... > > > > I am a little bit puzzled, those blocks will only be read by > > one task, then shouldn't region server scan blocks one by one? And > > since there are 480 blocks at most, how can it use up dataXceiver? > > > > Best Regards, > > Raymond Liu > > > >
