Did you observe anything interesting with such a large Java heap? You said you have 3G for the memstore, most of the rest is for the block cache I assume. Any long GC pauses, or other strange behavior?
Thanks. -- Lars ________________________________ From: Timo Schaepe <t...@timoschaepe.de> To: user@hbase.apache.org Sent: Saturday, December 14, 2013 5:27 AM Subject: Re: Problems with hbase.hregion.max.filesize Sorry, forgot our Hardwareconfiguration… 1 NameNode/SecondaryNameNode/HBase master 31 Datanodes/Regionserver All of them with 2x XEON E5-2640 2.5 GHz 128 GB RAM /dev/sda 90 GB /dev/sdb 1.1 TB /dev/sdc 1.1 TB where sda are SSDs disks for system and sdb and sdc are disks for HDFS/HBase Heapsize for Regionserver: 80 GB bye, Timo Am 14.12.2013 um 14:21 schrieb Timo Schaepe <t...@timoschaepe.de>: > Hey, > > @JM: Thanks for the hint with hbase.regionserver.fileSplitTimeout. At the > moment (the import is actually working) and after I splittet the specific > regions manually, we do not have growing regions anymore. > > hbase hbck says, all things are going fine. > 0 inconsistencies detected. > Status: OK > > @Ted Yu: Sure, have a look here: http://pastebin.com/2ANFVZEU > The relevant tablename ist data_1091. > > Thanks for your time. > > Timo > > Am 13.12.2013 um 20:18 schrieb Ted Yu <yuzhih...@gmail.com>: > >> Timo: >> Can you pastebin regionserver log around 2013-12-12 13:54:20 so that we can >> see what happened ? >> >> Thanks >> >> >> On Fri, Dec 13, 2013 at 11:02 AM, Jean-Marc Spaggiari < >> jean-m...@spaggiari.org> wrote: >> >>> Try to increase hbase.regionserver.fileSplitTimeout but put it back to its >>> default value after. >>> >>> Default value is 30 seconds. I think it's not normal for a split to take >>> more than that. >>> >>> What is your hardware configuration? >>> >>> Have you run hbck to see if everything is correct? >>> >>> JM >>> >>> >>> 2013/12/13 Timo Schaepe <t...@timoschaepe.de> >>> >>>> Hello again, >>>> >>>> digging in the logs of the specific regionserver shows me that: >>>> >>>> 2013-12-12 13:54:20,194 INFO >>>> org.apache.hadoop.hbase.regionserver.SplitRequest: Running >>> rollback/cleanup >>>> of failed split of >>>> >>> data,OR\x83\xCF\x02\x82\xAE\xF3U,1386851456415.d04cadb1b2252dafc476c138e9651ca7.; >>>> Took too long to split the files and create the references, aborting >>> split >>>> >>>> This message appears two time, so it seems, that HBase tried to split the >>>> region but it failed. I don't know why. How is the behaviour of HBase, >>> if a >>>> region split fails? Are there more tries to split this region again? I >>>> didn't find any new tries in the log. Now I split the big regions >>> manually >>>> and this works. And also it seems, that HBase split the new regions again >>>> to crunch they down to the given limit. >>>> >>>> But also it is a mystery for me, why the split size in Hannibal shows me >>>> 10 GB and in base-site.xml I put 2 GB… >>>> >>>> Thanks, >>>> >>>> Timo >>>> >>>> >>>> Am 13.12.2013 um 10:22 schrieb Timo Schaepe <t...@timoschaepe.de>: >>>> >>>>> Hello, >>>>> >>>>> during the loading of data in our cluster I noticed some strange >>>> behavior of some regions, that I don't understand. >>>>> >>>>> Scenario: >>>>> We convert data from a mysql database to HBase. The data is inserted >>>> with a put to the specific HBase table. The row key is a timestamp. I >>> know >>>> the problem with timestamp keys, but in our requirement it works quiet >>>> well. The problem is now, that there are some regions, which are growing >>>> and growing. >>>>> >>>>> For example the table on the picture [1]. First, all data was >>>> distributed over regions and node. And now, the data is written into only >>>> one region, which is growing and I can see no splitting at all. Actually >>>> the size of the big region is nearly 60 GB. >>>>> >>>>> HBase version is 0.94.11. I cannot understand, why the splitting is not >>>> happening. In hbase-site.xml I limit the hbase.hregion.max.filesize to 2 >>> GB >>>> and HBase accepted this value. >>>>> >>>>> <property> >>>>> <!--Loaded from hbase-site.xml--> >>>>> <name>hbase.hregion.max.filesize</name> >>>>> <value>2147483648</value> >>>>> </property> >>>>> >>>>> First mystery: Hannibal shows me the split size is 10 GB (see >>>> screenshot). >>>>> Second mystery: HBase is not splitting some regions neither at 2 GB nor >>>> 10 GB. >>>>> >>>>> Any ideas? Could be the timestamp rowkey cause this problem? >>>>> >>>>> Thanks, >>>>> >>>>> Timo >>>>> >>>>> [1] https://www.dropbox.com/s/lm286xkcpglnj1t/big_region.png >>>> >>>> >>> >
