Hey, sorry for the answer delay, I had a flight to San Francisco and fighting with the jetleg. I am here on vacation, maybe I can visit some interesting talks about HBase/Hadoop :).
Am 14.12.2013 um 13:14 schrieb lars hofhansl <la...@apache.org>:
> Did you observe anything interesting with such a large Java heap?
Not really. From the beginning we worked with such a big heap so I haven't any
big experience with less heap. But so far, this large heap space works well.
> You said you have 3G for the memstore, most of the rest is for the block
> cache I assume.
Not exactely. Memstore is 2.4 GB and the rest is Blockcache. The relevant
values in hbase-site.xml values are:
hfile.block.cache.size = 0.77
hbase.regionserver.global.memstore.upperLimit = 0.03
> Any long GC pauses, or other strange behavior?
Sometimes we have long GC pauses. For example yesterday we had a 213 Seconds GC
pause on one region server and he passed out. But those long pauses are very,
very rare. For GCing we use the default value in hbase-env.xml, I think:
export HBASE_OPTS="-XX:+UseConcMarkSweepGC"
I cannot see other strange behavior, except the problem with
hbase.hregion.max.filesize and why HBase is not splitting automatically at 2 GB.
Thanks,
Timo
>
> Thanks.
>
> -- Lars
>
>
>
> ________________________________
> From: Timo Schaepe <t...@timoschaepe.de>
> To: user@hbase.apache.org
> Sent: Saturday, December 14, 2013 5:27 AM
> Subject: Re: Problems with hbase.hregion.max.filesize
>
>
> Sorry, forgot our Hardwareconfiguration…
>
> 1 NameNode/SecondaryNameNode/HBase master
> 31 Datanodes/Regionserver
>
> All of them with
> 2x XEON E5-2640 2.5 GHz
> 128 GB RAM
> /dev/sda 90 GB
> /dev/sdb 1.1 TB
> /dev/sdc 1.1 TB
>
> where sda are SSDs disks for system and sdb and sdc are disks for HDFS/HBase
>
> Heapsize for Regionserver: 80 GB
>
> bye,
>
> Timo
>
>
>
> Am 14.12.2013 um 14:21 schrieb Timo Schaepe <t...@timoschaepe.de>:
>
>> Hey,
>>
>> @JM: Thanks for the hint with hbase.regionserver.fileSplitTimeout. At the
>> moment (the import is actually working) and after I splittet the specific
>> regions manually, we do not have growing regions anymore.
>>
>> hbase hbck says, all things are going fine.
>> 0 inconsistencies detected.
>> Status: OK
>>
>> @Ted Yu: Sure, have a look here: http://pastebin.com/2ANFVZEU
>> The relevant tablename ist data_1091.
>>
>> Thanks for your time.
>>
>> Timo
>>
>> Am 13.12.2013 um 20:18 schrieb Ted Yu <yuzhih...@gmail.com>:
>>
>>> Timo:
>>> Can you pastebin regionserver log around 2013-12-12 13:54:20 so that we can
>>> see what happened ?
>>>
>>> Thanks
>>>
>>>
>>> On Fri, Dec 13, 2013 at 11:02 AM, Jean-Marc Spaggiari <
>>> jean-m...@spaggiari.org> wrote:
>>>
>>>> Try to increase hbase.regionserver.fileSplitTimeout but put it back to its
>>>> default value after.
>>>>
>>>> Default value is 30 seconds. I think it's not normal for a split to take
>>>> more than that.
>>>>
>>>> What is your hardware configuration?
>>>>
>>>> Have you run hbck to see if everything is correct?
>>>>
>>>> JM
>>>>
>>>>
>>>> 2013/12/13 Timo Schaepe <t...@timoschaepe.de>
>>>>
>>>>> Hello again,
>>>>>
>>>>> digging in the logs of the specific regionserver shows me that:
>>>>>
>>>>> 2013-12-12 13:54:20,194 INFO
>>>>> org.apache.hadoop.hbase.regionserver.SplitRequest: Running
>>>> rollback/cleanup
>>>>> of failed split of
>>>>>
>>>> data,OR\x83\xCF\x02\x82\xAE\xF3U,1386851456415.d04cadb1b2252dafc476c138e9651ca7.;
>>>>> Took too long to split the files and create the references, aborting
>>>> split
>>>>>
>>>>> This message appears two time, so it seems, that HBase tried to split the
>>>>> region but it failed. I don't know why. How is the behaviour of HBase,
>>>> if a
>>>>> region split fails? Are there more tries to split this region again? I
>>>>> didn't find any new tries in the log. Now I split the big regions
>>>> manually
>>>>> and this works. And also it seems, that HBase split the new regions again
>>>>> to crunch they down to the given limit.
>>>>>
>>>>> But also it is a mystery for me, why the split size in Hannibal shows me
>>>>> 10 GB and in base-site.xml I put 2 GB…
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 13.12.2013 um 10:22 schrieb Timo Schaepe <t...@timoschaepe.de>:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> during the loading of data in our cluster I noticed some strange
>>>>> behavior of some regions, that I don't understand.
>>>>>>
>>>>>> Scenario:
>>>>>> We convert data from a mysql database to HBase. The data is inserted
>>>>> with a put to the specific HBase table. The row key is a timestamp. I
>>>> know
>>>>> the problem with timestamp keys, but in our requirement it works quiet
>>>>> well. The problem is now, that there are some regions, which are growing
>>>>> and growing.
>>>>>>
>>>>>> For example the table on the picture [1]. First, all data was
>>>>> distributed over regions and node. And now, the data is written into only
>>>>> one region, which is growing and I can see no splitting at all. Actually
>>>>> the size of the big region is nearly 60 GB.
>>>>>>
>>>>>> HBase version is 0.94.11. I cannot understand, why the splitting is not
>>>>> happening. In hbase-site.xml I limit the hbase.hregion.max.filesize to 2
>>>> GB
>>>>> and HBase accepted this value.
>>>>>>
>>>>>> <property>
>>>>>> <!--Loaded from hbase-site.xml-->
>>>>>> <name>hbase.hregion.max.filesize</name>
>>>>>> <value>2147483648</value>
>>>>>> </property>
>>>>>>
>>>>>> First mystery: Hannibal shows me the split size is 10 GB (see
>>>>> screenshot).
>>>>>> Second mystery: HBase is not splitting some regions neither at 2 GB nor
>>>>> 10 GB.
>>>>>>
>>>>>> Any ideas? Could be the timestamp rowkey cause this problem?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Timo
>>>>>>
>>>>>> [1] https://www.dropbox.com/s/lm286xkcpglnj1t/big_region.png
>>>>>
>>>>>
>>>>
smime.p7s
Description: S/MIME cryptographic signature
