bq. Do I need to restart master for the memstore to take effect?
No. memstore is used by region server.
Looks like debug logging was not turned on (judging from your previous
pastebin).
Some of flush related logs are at INFO level. e.g. Do you see any of the
following log ?
LOG.info("Flush of region " + regionToFlush + " due to global heap
pressure");
Take a look
at ./src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreFlusher.java
and you will find all the logs.
Cheers
On Thu, Nov 6, 2014 at 10:05 AM, Pere Kyle <p...@whisper.sh> wrote:
> So I have set the heap to 12Gb and the memstore limit to upperLimit .5
> lowerLimit .45. I am not seeing any changes in behavior from the cluster so
> far, i have restarted 4/17 region servers. Do I need to restart master for
> the memstore to take effect? Also how do I enable logging to show why a
> region is being flushed? I don’t ever seen the region flushes in my logs.
>
> Thanks,
> Pere
> On Nov 6, 2014, at 7:12 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > bq. to increase heap and increase the memstore limit?
> >
> > Yes. That would be an action that bears fruit.
> > Long term, you should merge the small regions.
> >
> > Cheers
> >
> > On Wed, Nov 5, 2014 at 11:20 PM, Pere Kyle <p...@whisper.sh> wrote:
> >
> >> Watching closely a region server in action. It seems that the memstores
> >> are being flushed at around 2MB on the regions. This would seem to
> >> indicate that there is not enough heap for the memstore and I am hitting
> >> the upper bound of limit (default). Would this be a fair assumption?
> Should
> >> I look to increase heap and increase the memstore limit?
> >>
> >> Thanks!
> >> -Pere
> >>
> >> On Nov 5, 2014, at 10:26 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>
> >>> You can use ConstantSizeRegionSplitPolicy.
> >>> Split policy can be specified per table. See the following example
> >>> in create.rb :
> >>>
> >>> hbase> create 't1', 'f1', {NUMREGIONS => 15, SPLITALGO =>
> >>> 'HexStringSplit'}
> >>>
> >>> In 0.94.18, there isn't online merge. So you have to use other method
> to
> >>> merge the small regions.
> >>>
> >>> Cheers
> >>>
> >>> On Wed, Nov 5, 2014 at 10:14 PM, Pere Kyle <p...@whisper.sh> wrote:
> >>>
> >>>> Ted,
> >>>>
> >>>> Thanks so much for that information. I now see why this split too
> often,
> >>>> but what I am not sure of is how to fix this without blowing away the
> >>>> cluster. Add more heap?
> >>>>
> >>>> Another symptom I have noticed is that load on the Master instance
> hbase
> >>>> daemon has been pretty high (load average 4.0, whereas it used to be
> >> 1.0)
> >>>>
> >>>> Thanks,
> >>>> Pere
> >>>>
> >>>> On Nov 5, 2014, at 9:56 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>>>
> >>>>> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
> >>>>>
> >>>>> You can read the javadoc of this class to see how it works.
> >>>>>
> >>>>> Cheers
> >>>>>
> >>>>> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> >>>>>
> >>>>>> Can you provide a bit more information (such as HBase release) ?
> >>>>>>
> >>>>>> If you pastebin one of the region servers' log, that would help us
> >>>>>> determine the cause.
> >>>>>>
> >>>>>> Cheers
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <p...@whisper.sh> wrote:
> >>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> Recently our cluster which has been running fine for 2 weeks split
> to
> >>>>>>> 1024 regions at 1GB per region, after this split the cluster is
> >>>> unusable.
> >>>>>>> Using the performance benchmark I was getting a little better than
> >> 100
> >>>> w/s,
> >>>>>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge
> with
> >>>> 8GB
> >>>>>>> heap reserved for Hbase
> >>>>>>>
> >>>>>>> Any Ideas? I am stumped:
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Pere
> >>>>>>>
> >>>>>>> Here is the current
> >>>>>>> hbase-site.xml
> >>>>>>> <?xml version="1.0"?>
> >>>>>>> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
> >>>>>>> <configuration>
> >>>>>>> <property>
> >>>>>>> <name>hbase.snapshot.enabled</name>
> >>>>>>> <value>true</value>
> >>>>>>> </property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.regionserver.handler.count</name><value>50</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.cluster.distributed</name><value>true</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.master.wait.for.log.splitting</name><value>true</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.flush.size</name><value>134217728</value></property>
> >>>>>>> <property><name>hbase.hregion.max.filesize</name><value>5073741824
> >>>>>>> </value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>zookeeper.session.timeout</name><value>60000</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.client.scanner.caching</name><value>1000</value></property>
> >>>>>>>
> >>>>>>>
> >>>>
> >>
> <property><name>hbase.hregion.memstore.block.multiplier</name><value>4</value></property>
> >>>>>>> </configuration>
> >>>>>>>
> >>>>>>> hbase-env.sh
> >>>>>>> # The maximum amount of heap to use, in MB. Default is 1000.
> >>>>>>> export HBASE_HEAPSIZE=8000
> >>>>>>>
> >>>>>>> # Extra Java runtime options.
> >>>>>>> # Below are what we set by default. May only work with SUN JVM.
> >>>>>>> # For more on why as well as other possible settings,
> >>>>>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
> >>>>>>> export HBASE_OPTS="-XX:+UseConcMarkSweepGC”
> >>>>>>>
> >>>>>>> hbase-env.sh
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>
> >>
>
>
