[
https://issues.apache.org/jira/browse/ACCUMULO-201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13161806#comment-13161806
]
Keith Turner edited comment on ACCUMULO-201 at 12/2/11 7:37 PM:
----------------------------------------------------------------
Another option is to spawn a process to do major compaction. It can die w/o
bringing down the tserver. Can still get locality by spawning it on the local
machine.
was (Author: kturner):
Another option is to spawn a process to do major compaction. It can die
w/o bringing down the tserver.
> tablet server runs out of memory performing a major compaction
> --------------------------------------------------------------
>
> Key: ACCUMULO-201
> URL: https://issues.apache.org/jira/browse/ACCUMULO-201
> Project: Accumulo
> Issue Type: Bug
> Components: tserver
> Reporter: Eric Newton
> Assignee: Eric Newton
>
> An accumulo user watched their cluster slowly shrink: one tablet server would
> fail every 8-10 minutes.
> We determined that a major compaction of a single tablet would cause the
> tablet server to run out of memory. That tablet would then be sent to a new
> server, which would schedule a major compaction, and it would die as well.
> # it was harder than it should have been to identify the tablet causing the
> problem
> # the tablet had a combination of several large existing files and a few
> bulk loaded files with a few very large key/values
> # large key/values were between *10 and 100 megabytes each*, the tablet
> server had a 1G memory limit
> # the next key for each file will sit in memory while performing the
> merge-sort
> There exists a Constraint which can limit the size of mutations during normal
> ingest. However, there is no constraint or check on the size of mutations
> that may be bulk loaded.
> The tablet server should log the key extent (range) of a tablet prior to
> attempting a major compaction.
> Large key values (those that approach a significant portion of the working
> memory of the JVM) might need to go into a separate merge file, or might
> result in multi-stage merges just to defend against an out-of-memory failure.
> Tablet servers could mark tablets during a major compaction attempt. Tablets
> with multiple markers could use a multi-pass merge to attempt to survive the
> merge. Alternatively, the master could refuse to assign tablets with too
> many markers.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
