[
https://issues.apache.org/jira/browse/KUDU-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wong resolved KUDU-2047.
-------------------------------
Fix Version/s: 1.12.0
Assignee: Andrew Wong (was: William Berkeley)
Resolution: Fixed
I addressed this issue in the context of KUDU-1625. See commits
[f4508ff|https://github.com/apache/kudu/commit/f4508ff] and
[58f7a30|https://github.com/apache/kudu/commit/58f7a30], which keep delta stats
in memory when flushing a DMS and major/minor delta compacting respectively.
I didn't do the plumbing for merge compactions:
{code:java}
I considered also plumbing stats into merge compaction outputs, but
opted not to. The plumbing for compactions seems more significant, and
chances are, large bulk deletes will come after data has had time to be
compacted (e.g. deleting old ranges of keyspace).
{code}
> Lazy cfile open and maintenance op stat caching cause fruitful delta
> compaction ops to never run
> ------------------------------------------------------------------------------------------------
>
> Key: KUDU-2047
> URL: https://issues.apache.org/jira/browse/KUDU-2047
> Project: Kudu
> Issue Type: Bug
> Components: compaction, perf, tablet
> Affects Versions: 1.4.0
> Reporter: Todd Lipcon
> Assignee: Andrew Wong
> Priority: Major
> Fix For: 1.12.0
>
>
> I was just looking at a cluster which has a large amount of REDO data on some
> of its tablets, and wasn't sure why it wasn't ever compacting it. The issue
> appears to be the following:
> - in DiskRowSet::DeltaStoresCompactionPerfImprovementScore(), we call through
> to GetColumnIdsWithUpdates() to see which columns may need compaction
> -- if the REDO delta block is not open (eg when the server has recently
> started), this will skip the unopened delta file stats and not include them
> in the result
> -- we thus determine that the compaction is not fruitful
> This was a conscious decision to avoid the MM from eagerly opening every
> delta on its first pass through computing compaction stats. We figured that,
> if it were worth compacting, then probably someone would scan the data,
> forcing the deltas to get opened and thus made eligible for compaction.
> However, the MM tries to be smart about caching the statistics (see
> e7fe0c1a94cac364522c09b8208c98480947d794). In particular, if it sees that the
> tablet has not run any flushes or compactions, it won't bother to recalculate
> the stats, assuming they haven't changed.
> So, if you have a completely read-only tablet with some uncompacted deltas,
> the MM op will never run.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
