[
https://issues.apache.org/jira/browse/ACCUMULO-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047833#comment-14047833
]
Josh Elser commented on ACCUMULO-2889:
--------------------------------------
bq. We could just add all the hosted tablets to the log when it is new
(common), and when new tablets are hosted (rare).
Could also try to be a little smarter and keep some basic measure of "recently
written to" tablets and update the log entry there instead of all tablets. That
would help the write-once, read-many case that's common.
bq. This would cause unnecessary recoveries in the case where a tablet is
flushed or isn't otherwise using the log.
I'm not entirely clear on what performing log recovery for a tablet using a WAL
that doesn't have any updates for that tablet entails. Would we just be sorting
and re-reading that WAL to not find any updates that need to be replayed? Do
you think we would incur a noticeable penalty for doing this?
> Batch metadata table updates for new walogs
> -------------------------------------------
>
> Key: ACCUMULO-2889
> URL: https://issues.apache.org/jira/browse/ACCUMULO-2889
> Project: Accumulo
> Issue Type: Improvement
> Affects Versions: 1.5.1, 1.6.0
> Reporter: Jonathan Park
> Assignee: Jonathan Park
> Attachments: ACCUMULO-2889.0.patch.txt, ACCUMULO-2889.1.patch,
> accumulo-2889-withpatch.png, accumulo-2889_withoutpatch.png,
> batch_perf_test.sh, run_all.sh, start-ingest.sh
>
>
> Currently, when we update the Metadata table with new loggers, we will update
> the metadata for each tablet serially. We could optimize this to instead use
> a batchwriter to send all metadata updates for all tablets in a batch.
> A few special cases include:
> - What if the !METADATA tablet was included in the batch?
> - What about the root tablet?
> Benefit:
> In one of our clusters, we're experiencing particularly slow HDFS operations
> leading to large oscillations in ingest performance. We haven't isolated the
> cause in HDFS but when we profile the tservers, we noticed that they were
> waiting for metadata table operations to complete. This would target the
> waiting.
> Potential downsides:
> Given the existing locking scheme, it looks like we may have to lock a tablet
> for slightly longer (we'll lock for the duration of the batch).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
