On Tue, Jun 9, 2009 at 10:13 AM, Bradford Stephens < [email protected]> wrote:
> Hey rock stars, > Flattery makes us perk up for sure. > > I'm having problems loading large amounts of data into a table (about > 120 GB, 250million rows). My Map task runs fine, but when it comes to > reducing, things start burning. 'top' inidcates that I only have ~ > 100M of RAM free on my datanodes, and every process starts thrashing > ... even ssh and ping. Then I start to get errors like: > > "org.apache.hadoop.hbase.client.RegionOfflineException: region > offline: joinedcontent,,1244513452487" > See if said region is actually offline? Try getting a row from it in shell. > > and: > > "Task attempt_200906082135_0001_r_000002_0 failed to report status for > 603 seconds. Killing!" Sounds like nodes are heavily loaded.. so loaded either the task can't report in... or its stuck on an hbase update so long, its taking ten minutes or more to return. One thing to look at is disabling batching or making batches smaller. When batch is big, can take a while under high-load for all row edits to go in. HBase client will not return till all row commits have succeeded. Smaller batches will mean more likely to return and not have the task killed because takes longer than the report period to checkin. Whats your MR job like? Your updating hbase in the reduce phase i presume (TableOutputFormat?). Do you need the reduce? Can you update hbase in the map step? Saves on the sort the MR framework is doing -- a sort that is unnecessary given as hbase orders on insertion. Can you try with a lighter load? Maybe a couple of smaller MR jobs rather than one big one? St.Ack > > > I'm running Hadoop .19.1 and HBase .19.3, with 1 master/name node and > 8 regionservers. 2 x Dual Core Intel 3.2 GHz procs, 4 GB of RAM. 16 > map tasks, 8 reducers. I've set the MAX_HEAP in hadoop-env to 768, and > the one in hbase-env is at its default with 1000. I've also done all > the performance enchancements in the Wiki with the file handlers, the > garbage collection, and the epoll limits. > > What am I missing? :) > > Cheers, > Bradford >
