[
https://issues.apache.org/jira/browse/ACCUMULO-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Park updated ACCUMULO-2889:
------------------------------------
Attachment: ACCUMULO-2889.2.patch
I'll gather a new set of #s when I get access to a cluster of machines.
> Batch metadata table updates for new walogs
> -------------------------------------------
>
> Key: ACCUMULO-2889
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2889
> Project: Accumulo
> Issue Type: Improvement
> Affects Versions: 1.5.1, 1.6.0
> Reporter: Jonathan Park
> Assignee: Jonathan Park
> Attachments: ACCUMULO-2889.0.patch.txt, ACCUMULO-2889.1.patch,
> ACCUMULO-2889.2.patch, accumulo-2889-withpatch.png,
> accumulo-2889_withoutpatch.png, batch_perf_test.sh, run_all.sh,
> start-ingest.sh
>
>
> Currently, when we update the Metadata table with new loggers, we will update
> the metadata for each tablet serially. We could optimize this to instead use
> a batchwriter to send all metadata updates for all tablets in a batch.
> A few special cases include:
> - What if the !METADATA tablet was included in the batch?
> - What about the root tablet?
> Benefit:
> In one of our clusters, we're experiencing particularly slow HDFS operations
> leading to large oscillations in ingest performance. We haven't isolated the
> cause in HDFS but when we profile the tservers, we noticed that they were
> waiting for metadata table operations to complete. This would target the
> waiting.
> Potential downsides:
> Given the existing locking scheme, it looks like we may have to lock a tablet
> for slightly longer (we'll lock for the duration of the batch).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
