[
https://issues.apache.org/jira/browse/ACCUMULO-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15238551#comment-15238551
]
ASF GitHub Bot commented on ACCUMULO-4187:
------------------------------------------
Github user joshelser commented on a diff in the pull request:
https://github.com/apache/accumulo/pull/90#discussion_r59489500
--- Diff:
core/src/main/java/org/apache/accumulo/core/file/rfile/RFileOperations.java ---
@@ -132,7 +136,8 @@ FileSKVWriter openWriter(String file, FileSystem fs,
Configuration conf, Accumul
sampler = SamplerFactory.newSampler(samplerConfig, acuconf);
}
- CachableBlockFile.Writer _cbw = new
CachableBlockFile.Writer(fs.create(new Path(file), false, bufferSize, (short)
rep, block), compression, conf, acuconf);
+ CachableBlockFile.Writer _cbw = new CachableBlockFile.Writer(new
RateLimitedOutputStream(fs.create(new Path(file), false, bufferSize, (short)
rep, block),
--- End diff --
nit: I would be ok with splitting up this line. It's getting a little busy
:)
> Rate limiting of major compactions
> ----------------------------------
>
> Key: ACCUMULO-4187
> URL: https://issues.apache.org/jira/browse/ACCUMULO-4187
> Project: Accumulo
> Issue Type: Improvement
> Components: core, tserver
> Affects Versions: 1.8.0
> Reporter: Shawn Walker
> Assignee: Shawn Walker
> Priority: Minor
> Fix For: 1.8.0
>
>
> In discussing [ACCUMULO-4166] with Keith Turner, we decided that the
> underlying issue is that major compactions can overwhelm a tablet server,
> rendering it nearly unresponsive.
> To address this, we should take a cue from Apache Cassandra and restrict how
> quickly we perform major compactions. Rate limiting reads and writes
> involved in major compactions will directly affect the IO load caused by
> major compactions, and should also indirectly affect the CPU load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
