See the following:
http://hbase.apache.org/book.html#build
If you continue to have problem building hbase, I suggest starting another
thread on user@ with errors encountered.
Cheers
On Thu, Oct 8, 2015 at 7:49 AM, Randy Fox <r...@connexity.com> wrote:
> Thanks Ted, looks the right change. We are using CDH and it does not have
> that fix. Easy enough to put in, though I tried to build hbase and ran
> into many issues.
> Is there a doc somewhere on the prereqs for building hbase (1.0.0) in
> particular and the mvn flags?
>
> Thanks,
>
> Randy
>
>
>
> On 10/7/15, 1:34 PM, "Ted Yu" <yuzhih...@gmail.com> wrote:
>
> >Have you seen the following code ?
> >
> > ThreadPoolExecutor pool = (selectNow &&
> >s.throttleCompaction(compaction.getRequest().getSize()))
> > ? longCompactions : shortCompactions;
> >
> >Looks like throttleCompaction() returned false.
> >Please see the following method in RatioBasedCompactionPolicy :
> >
> > public boolean throttleCompaction(long compactionSize) {
> >
> > return compactionSize > comConf.getThrottlePoint();
> >
> >FYI
> >
> >On Wed, Oct 7, 2015 at 11:07 AM, Randy Fox <r...@connexity.com> wrote:
> >
> >> Looking at the source I don’t see how a manual major compact could get
> in
> >> the large compaction pool:
> >> If selectNow Is True the pool selector has to be shortCompactions. From
> >> the logs it sure seems that selectNow has to be true, else it would have
> >> logged “system” instead of
> >>
> "org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
> >> ;”
> >>
> >>
> >> Sure seems like an hbase bug.
> >>
> >> Am I missing something?
> >>
> >> Thanks,
> >>
> >> Randy
> >>
> >> /**
> >> * @param r HRegion store belongs to
> >> * @param s Store to request compaction on
> >> * @param why Why compaction requested -- used in debug messages
> >> * @param priority override the default priority (NO_PRIORITY ==
> decide)
> >> * @param request custom compaction request. Can be <tt>null</tt> in
> >> which case a simple
> >> * compaction will be used.
> >> */
> >> private synchronized CompactionRequest requestCompactionInternal(final
> >> HRegion r, final Store s,
> >> final String why, int priority, CompactionRequest request, boolean
> >> selectNow)
> >> throws IOException {
> >> if (this.server.isStopped()
> >> || (r.getTableDesc() != null &&
> >> !r.getTableDesc().isCompactionEnabled())) {
> >> return null;
> >> }
> >>
> >> CompactionContext compaction = null;
> >> if (selectNow) {
> >> compaction = selectCompaction(r, s, priority, request);
> >> if (compaction == null) return null; // message logged inside
> >> }
> >>
> >> // We assume that most compactions are small. So, put system
> >> compactions into small
> >> // pool; we will do selection there, and move to large pool if
> >> necessary.
> >> long size = selectNow ? compaction.getRequest().getSize() : 0;
> >> ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
> >> ? longCompactions : shortCompactions;
> >> pool.execute(new CompactionRunner(s, r, compaction, pool));
> >> if (LOG.isDebugEnabled()) {
> >> String type = (pool == shortCompactions) ? "Small " : "Large ";
> >> LOG.debug(type + "Compaction requested: " + (selectNow ?
> >> compaction.toString() : "system")
> >> + (why != null && !why.isEmpty() ? "; Because: " + why : "") +
> >> "; " + this);
> >> }
> >> return selectNow ? compaction.getRequest() : null;
> >> }
> >>
> >>
> >>
> >>
> >>
> >> On 10/6/15, 9:58 PM, "Randy Fox" <r...@connexity.com> wrote:
> >>
> >> >Sorry, I do not follow. I thought there was one thread per pool, one
> for
> >> large and one for small. Thus my major compact should have been in the
> >> large pool and not blocking the small pool. That is how it worked in
> 0.94.
> >> >From the logs it demoted my compact to the small pool.
> >> >
> >> >-randy
> >> >
> >> >
> >> >
> >> >On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vladrodio...@gmail.com>
> wrote:
> >> >
> >> >>>> If I read this correct the size is 4.8G and the throttle is 2.5G
> so it
> >> >>should have been put into the Large compaction pool.
> >> >>
> >> >>You answered your question yourself. Minor compaction in the same
> pool (1
> >> >>thread default) will be waiting until major is finished.
> >> >>
> >> >>-Vlad
> >> >>
> >> >>
> >> >>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <r...@connexity.com> wrote:
> >> >>
> >> >>> 2015-10-06 14:50:35,644 INFO
> >> org.apache.hadoop.hbase.regionserver.HStore:
> >> >>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
> >> >>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49.
> into
> >> >>>
> >>
> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
> >> >>> totalSize=4.8 G
> >> >>> 2015-10-06 14:50:35,646 DEBUG
> >> >>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small
> >> Compaction
> >> >>> requested:
> >> >>>
> >>
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8
> >> ;
> >> >>> Because: User-triggered major compaction; compaction_queue=(0:0),
> >> >>> split_queue=0, merge_queue=0
> >> >>>
> >> >>>
> >> >>> If I read this correct the size is 4.8G and the throttle is 2.5G so
> it
> >> >>> should have been put into the Large compaction pool.
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On 10/6/15, 3:07 PM, "Randy Fox" <r...@connexity.com> wrote:
> >> >>>
> >> >>> >We just did a big leap forward from 0.94 to 1.0. We do our own
> major
> >> >>> compacts manually every night. One of the first things we noticed is
> >> that
> >> >>> when a major compact runs, no minors run. The compaction queue
> grows
> >> and
> >> >>> when the major compact finishes the minors then run. I have not
> >> found any
> >> >>> new knobs we should be setting. Any ideas?
> >> >>> >Our config is:
> >> >>> >
> >> >>> ><property>
> >> >>> > <name>hbase.hregion.majorcompaction</name>
> >> >>> > <value>0</value>
> >> >>> > <final>true</final>
> >> >>> ></property>
> >> >>> ><property>
> >> >>> > <name>hbase.hstore.compactionThreshold</name>
> >> >>> > <value>3</value>
> >> >>> ></property>
> >> >>> >
> >> >>> ><property>
> >> >>> > <name>hbase.hstore.compaction.max</name>
> >> >>> > <value>9</value>
> >> >>> ></property>
> >> >>> ><property>
> >> >>> > <name>hbase.regionserver.thread.compaction.throttle</name>
> >> >>> > <value>2684354560</value>
> >> >>> ></property>
> >> >>> >
> >> >>> >
> >> >>> >Thanks in advance,
> >> >>> >
> >> >>> >Randy
> >> >>>
> >>
>
