Looking at the source I don’t see how a manual major compact could get in the
large compaction pool:
If selectNow Is True the pool selector has to be shortCompactions. From the
logs it sure seems that selectNow has to be true, else it would have logged
“system” instead of
"org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8;”
Sure seems like an hbase bug.
Am I missing something?
Thanks,
Randy
/**
* @param r HRegion store belongs to
* @param s Store to request compaction on
* @param why Why compaction requested -- used in debug messages
* @param priority override the default priority (NO_PRIORITY == decide)
* @param request custom compaction request. Can be <tt>null</tt> in which
case a simple
* compaction will be used.
*/
private synchronized CompactionRequest requestCompactionInternal(final
HRegion r, final Store s,
final String why, int priority, CompactionRequest request, boolean
selectNow)
throws IOException {
if (this.server.isStopped()
|| (r.getTableDesc() != null &&
!r.getTableDesc().isCompactionEnabled())) {
return null;
}
CompactionContext compaction = null;
if (selectNow) {
compaction = selectCompaction(r, s, priority, request);
if (compaction == null) return null; // message logged inside
}
// We assume that most compactions are small. So, put system compactions
into small
// pool; we will do selection there, and move to large pool if necessary.
long size = selectNow ? compaction.getRequest().getSize() : 0;
ThreadPoolExecutor pool = (!selectNow && s.throttleCompaction(size))
? longCompactions : shortCompactions;
pool.execute(new CompactionRunner(s, r, compaction, pool));
if (LOG.isDebugEnabled()) {
String type = (pool == shortCompactions) ? "Small " : "Large ";
LOG.debug(type + "Compaction requested: " + (selectNow ?
compaction.toString() : "system")
+ (why != null && !why.isEmpty() ? "; Because: " + why : "") + "; " +
this);
}
return selectNow ? compaction.getRequest() : null;
}
On 10/6/15, 9:58 PM, "Randy Fox" <r...@connexity.com> wrote:
>Sorry, I do not follow. I thought there was one thread per pool, one for large
>and one for small. Thus my major compact should have been in the large pool
>and not blocking the small pool. That is how it worked in 0.94.
>From the logs it demoted my compact to the small pool.
>
>-randy
>
>
>
>On 10/6/15, 4:32 PM, "Vladimir Rodionov" <vladrodio...@gmail.com> wrote:
>
>>>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
>>should have been put into the Large compaction pool.
>>
>>You answered your question yourself. Minor compaction in the same pool (1
>>thread default) will be waiting until major is finished.
>>
>>-Vlad
>>
>>
>>On Tue, Oct 6, 2015 at 3:59 PM, Randy Fox <r...@connexity.com> wrote:
>>
>>> 2015-10-06 14:50:35,644 INFO org.apache.hadoop.hbase.regionserver.HStore:
>>> Starting compaction of 4 file(s) in L of PROD_audience4,\x00
>>> \xB6\x0B\xA7,\x186,1443751119137.26f321d7b240c85a9350a95f6c288e49. into
>>> tmpdir=hdfs://woz/hbase/data/default/PROD_audience4/26f321d7b240c85a9350a95f6c288e49/.tmp,
>>> totalSize=4.8 G
>>> 2015-10-06 14:50:35,646 DEBUG
>>> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Small Compaction
>>> requested:
>>> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@7f236eb8;
>>> Because: User-triggered major compaction; compaction_queue=(0:0),
>>> split_queue=0, merge_queue=0
>>>
>>>
>>> If I read this correct the size is 4.8G and the throttle is 2.5G so it
>>> should have been put into the Large compaction pool.
>>>
>>>
>>>
>>>
>>>
>>> On 10/6/15, 3:07 PM, "Randy Fox" <r...@connexity.com> wrote:
>>>
>>> >We just did a big leap forward from 0.94 to 1.0. We do our own major
>>> compacts manually every night. One of the first things we noticed is that
>>> when a major compact runs, no minors run. The compaction queue grows and
>>> when the major compact finishes the minors then run. I have not found any
>>> new knobs we should be setting. Any ideas?
>>> >Our config is:
>>> >
>>> ><property>
>>> > <name>hbase.hregion.majorcompaction</name>
>>> > <value>0</value>
>>> > <final>true</final>
>>> ></property>
>>> ><property>
>>> > <name>hbase.hstore.compactionThreshold</name>
>>> > <value>3</value>
>>> ></property>
>>> >
>>> ><property>
>>> > <name>hbase.hstore.compaction.max</name>
>>> > <value>9</value>
>>> ></property>
>>> ><property>
>>> > <name>hbase.regionserver.thread.compaction.throttle</name>
>>> > <value>2684354560</value>
>>> ></property>
>>> >
>>> >
>>> >Thanks in advance,
>>> >
>>> >Randy
>>>
