Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24452
to look at the new patch set (#2).
Change subject: KUDU-3783 Extend clean-superblock guarantee into
LoadFromSuperBlock
......................................................................
KUDU-3783 Extend clean-superblock guarantee into LoadFromSuperBlock
Patches under KUDU-1060 only guarantee a clean superblock if cleanup
happens through DeleteTabletData, ignoring crash-recovery path:
1. A crash occurs after a superblock was written with orphaned block
entries, but before cleanup Flush() that re-syncs a superblock with
empty list of orphaned block entries.
2. On the next startup LoadFromSuperBlock() calls DeleteOrphanedBlocks()
that deletes the orphaned blocks from disk as well as in-memory list.
But it doesn't sync a clean superblock afterwards.
3. For a tombstoned tablet that doesn't receive any writes(i.e., flush),
on-disk superblock always has those stale orphaned block entries.
Every subsequent restart would trigger parsing of those block entries
and repeated call to DeleteOrphanedBlocks() that is a per-restart
cost proportional to:
No. of tombstoned tablets x No. of orphaned blocks per tablet
The patch adds a fix by calling Flush() after DeleteOrphanedBlocks,
during load-time to persist a clean superblock (gated by
--enable_orphaned_block_cleanup_on_load). This converts the recurring
per-restart reprocessing into a one-time cost on the first restart,
which is precisely what the newly added unit test i.e.,
BenchmarkCleanStartupAfterOrphanCleanup demonstrates.
Some benchmarking stats:
=== KUDU-3783 Startup Overhead Benchmark ===
Tablets : 1000
Orphaned blocks / tablet : 1000000
With cleanup + Flush() : 613387 ms
Without cleanup + Flush() : 253331 ms
Extra overhead (Flush x N) : 360056 ms
Avg overhead per tablet : 360 ms
=== KUDU-3783 One-Time-Cost Benchmark ===
Tablets : 1000
Orphaned blocks / tablet : 1000000
Pass 1 - 1st restart (fix ON, dirty) : 629216 ms [cleanup+Flush for
all tablets]
Pass 2 - 2nd restart (fix ON, clean) : 114 ms [superblocks already
clean]
Pass 3 - clean baseline (fix OFF, clean) : 112 ms [no orphaned blocks to
parse]
Pass 4 - legacy baseline (fix OFF, dirty) : 262499 ms [parses orphan IDs,
no Flush]
Pass1 overhead vs Pass4 legacy : 366717 ms
Pass2 vs Pass3 clean baseline : 2 ms (should be ~0)
Change-Id: I729395085fb845daa667b1374dc1ac4291a0c30b
---
M src/kudu/tablet/tablet_metadata-test.cc
M src/kudu/tablet/tablet_metadata.cc
M src/kudu/tablet/tablet_metadata.h
3 files changed, 503 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/52/24452/2
--
To view, visit http://gerrit.cloudera.org:8080/24452
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I729395085fb845daa667b1374dc1ac4291a0c30b
Gerrit-Change-Number: 24452
Gerrit-PatchSet: 2
Gerrit-Owner: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
