Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread Josh Elser
FWIW, last I looked into this, https://issues.apache.org/jira/browse/HBASE-15154 would be the long-term solution to the Master also requiring the MaxDirectMemorySize configuration (even when it is not acting as a RegionServer). Obviously, it's a lower priority fix as there is a simple

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread Ted Yu
bq. Use different hbase config xml for HM process other than what used for RS In Ambari, there is concept of host group which should allow the above. Again, further discussion should be carried out in Ambari mailing list. On Tue, Sep 12, 2017 at 10:17 AM, Anoop John

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread ramkrishna vasudevan
Oh I did not know that the version was HDP. I just thought the set up process was through some HDP links. Thanks. On Tue, Sep 12, 2017 at 10:44 PM, Ted Yu wrote: > Ram: > As Arul indicated, HDP is being used where there're a lot of backports to > 1.1.2 baseline. > > Arul: >

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread Anoop John
+1 on what Ram said abt why HM also need this setting in ur version.. HM starts a RS internally..You can do one thing.. Use different hbase config xml for HM process other than what used for RS. May be we need some fix also as Ram said.. Specially when no regions are allowed in HM side. In

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread Ted Yu
Ram: As Arul indicated, HDP is being used where there're a lot of backports to 1.1.2 baseline. Arul: I suggest going thru vendor's channel first. Cheers On Tue, Sep 12, 2017 at 10:11 AM, ramkrishna vasudevan < ramkrishna.s.vasude...@gmail.com> wrote: > Why Master also accepts MaxDirectMemory

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread ramkrishna vasudevan
Why Master also accepts MaxDirectMemory config is probably because in 1.1 onwards we treat HMaster as HregionServer but it does the region managment also. Am not very sure in 1.1.2 is HMaster allowed to host regions ? If so you need to configure MaxDirectMemory, if not probably we can see how we

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread Arul Ramachandran
Thank you, Ram. >> So are you trying to use bucket cache feature in offheap mode with 1.1.2? Yes. >> So even in 0.98 you were using bucket cache in offheap mode? Yes, but it is a different hbase cluster and it run 0.98. The one I am trying to setup offheap cache is hbase 1.1.2 -arul On

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread Arul Ramachandran
Ted, For #1, I was following: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.1/bk_data-access/content/ch_managing-hbase.html#ref-db219cd6-c586-49c1-bc56-c9c1c5475276.1 ++ For #2, I got the error "Caused by: java.lang.OutOfMemoryError: Direct buffer memory" when starting hbase master.

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread ramkrishna vasudevan
Hi So are you trying to use bucket cache feature in offheap mode with 1.1.2? If so then it is needed that the MaxDirectMemory is greater than the offheap bucket cache size. If you are not using in offheap mode then probably there is no need for MaxDirectMemory to be greater than bucket cache

Re: Offheap config question for Hbase 1.1.2

2017-09-12 Thread Ted Yu
Looks like the config you meant should be hbase.bucketcache.size As the refguide says: A float that EITHER represents a percentage of total heap memory size to give to the cache (if < 1.0) OR, it is the total capacity in megabytes of BucketCache. Default: 0.0 If you specify the size as

Offheap config question for Hbase 1.1.2

2017-09-12 Thread Arul Ramachandran
In HBase 1.1.2, I am setting up bucket cache. I set MaxDirectMemory size greater than hbase.bucket.cache.size - only then it would work. 1) Does HBASE_REGIONSERVER_OPTS -XX:MaxDirectMemorySize needs to be greater than hbase.bucket.cache.size? 2) It seems with hbase 1.1.2, HBASE_MASTER_OPTS also