Ashwani Raina has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24125
Change subject: [WIP] Add online gc op to clean orphaned blocks ...................................................................... [WIP] Add online gc op to clean orphaned blocks This patch adds a background manintenance GC operation that makes use of persisted information about set of orphaned blocks that could not be deleted earlier due to transient errors, and tries to delete those in the background. Main highlights: - Add RetryOrphanedBlockDeletion as the main driving function that does fetching of orphaned blocks, call DeleteOrphanedBlocks for the set of orphaned blocks. - Ensure DeleteOrphanedBlocks doesn't erase all the block entries from the set of orphaned blocks when either partial or nothing from the set was successfully deleted. The blocks with failed deletions are to be used in next iteration by either GC or LoadSuperBlock. - With this change, ensure when superblock load kicks in, we try to delete the stale orphaned blocks only once. Irrespective of what the outcome is from deletion, erase all the block entries from in-memory set. This avoids possibility of those blocks being retried for delete unnecessarily. - Add unit tests to: * Test that OrphanedGCOp runs when there orphaned blocks to be deleted. * Measure performance benchmark for the KUDU-1060 startup overhead. * Test startup overhead with many tombstoned tablets. TODO: - Explore viability of background GC logic for persistent I/O errors. If possible find ways to avoid indefinite retries in such cases. - Add more details in the commit message. - Run more benchmarking tests. Change-Id: I732133b460df2fe4c91d05f420dda6f0274d440e --- M src/kudu/integration-tests/ts_recovery-itest.cc M src/kudu/tablet/tablet.cc M src/kudu/tablet/tablet_metadata-test.cc M src/kudu/tablet/tablet_metadata.cc M src/kudu/tablet/tablet_metadata.h M src/kudu/tablet/tablet_metrics.cc M src/kudu/tablet/tablet_metrics.h M src/kudu/tablet/tablet_mm_ops.cc M src/kudu/tablet/tablet_mm_ops.h 9 files changed, 881 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/25/24125/1 -- To view, visit http://gerrit.cloudera.org:8080/24125 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I732133b460df2fe4c91d05f420dda6f0274d440e Gerrit-Change-Number: 24125 Gerrit-PatchSet: 1 Gerrit-Owner: Ashwani Raina <[email protected]>
