[ https://issues.apache.org/jira/browse/ACCUMULO-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14047833#comment-14047833 ]
Josh Elser commented on ACCUMULO-2889: -------------------------------------- bq. We could just add all the hosted tablets to the log when it is new (common), and when new tablets are hosted (rare). Could also try to be a little smarter and keep some basic measure of "recently written to" tablets and update the log entry there instead of all tablets. That would help the write-once, read-many case that's common. bq. This would cause unnecessary recoveries in the case where a tablet is flushed or isn't otherwise using the log. I'm not entirely clear on what performing log recovery for a tablet using a WAL that doesn't have any updates for that tablet entails. Would we just be sorting and re-reading that WAL to not find any updates that need to be replayed? Do you think we would incur a noticeable penalty for doing this? > Batch metadata table updates for new walogs > ------------------------------------------- > > Key: ACCUMULO-2889 > URL: https://issues.apache.org/jira/browse/ACCUMULO-2889 > Project: Accumulo > Issue Type: Improvement > Affects Versions: 1.5.1, 1.6.0 > Reporter: Jonathan Park > Assignee: Jonathan Park > Attachments: ACCUMULO-2889.0.patch.txt, ACCUMULO-2889.1.patch, > accumulo-2889-withpatch.png, accumulo-2889_withoutpatch.png, > batch_perf_test.sh, run_all.sh, start-ingest.sh > > > Currently, when we update the Metadata table with new loggers, we will update > the metadata for each tablet serially. We could optimize this to instead use > a batchwriter to send all metadata updates for all tablets in a batch. > A few special cases include: > - What if the !METADATA tablet was included in the batch? > - What about the root tablet? > Benefit: > In one of our clusters, we're experiencing particularly slow HDFS operations > leading to large oscillations in ingest performance. We haven't isolated the > cause in HDFS but when we profile the tservers, we noticed that they were > waiting for metadata table operations to complete. This would target the > waiting. > Potential downsides: > Given the existing locking scheme, it looks like we may have to lock a tablet > for slightly longer (we'll lock for the duration of the batch). -- This message was sent by Atlassian JIRA (v6.2#6252)