Adar Dembo has submitted this change and it was merged. Change subject: log block manager: detect and repair unpunched holes ......................................................................
log block manager: detect and repair unpunched holes This patch adds detection and repair of "unpunched holes" to the LBM. These are deleted blocks whose space in the container data files is still "live", either because hole punching failed, or because the server crashed before the holes could be punched. The newly added container accounting is used as a heuristic to decide when to repunch holes: if a container's data size exceeds the size we think it should have (after alignment), we'll repunch all holes and truncate any preallocated space off the end. The heuristic uses some (configurable) slop to work around various filesystem accounting issues. An alternative is to use the container's extent map to figure this out (and to provide greater precision on where to punch), but testing on el6.6 showed that calling the FS_IOC_FIEMAP ioctl() on every full container increased LBM startup time by about 50%. That's bad enough that we shouldn't do it willy nilly. It could be gated on the above heuristic and used to drive more precise hole repunching, but given the complexity involved and given the amount of tilting I've done at this particular windmill, it'll remain a problem for another day. Testing is done in three ways: - A new unit test that exercises new LBMCorruptor functionality. - Inclusion in block_manager-stress-test via the change to LBMCorruptor::InjectRandomNonFatalInconsistency(). - Inclusion in BlockManagerTest.TestMetadataOkayDespiteFailedWrites via the generalization of the env_inject_io_error gflag. Change-Id: I016ea401380f4c8c7b1fd907ff67cb595f377dd1 Reviewed-on: http://gerrit.cloudera.org:8080/6717 Tested-by: Adar Dembo <[email protected]> Reviewed-by: David Ribeiro Alves <[email protected]> --- M src/kudu/fs/block_manager-stress-test.cc M src/kudu/fs/block_manager-test.cc M src/kudu/fs/log_block_manager-test-util.cc M src/kudu/fs/log_block_manager-test-util.h M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc M src/kudu/fs/log_block_manager.h M src/kudu/util/env_posix.cc 8 files changed, 297 insertions(+), 81 deletions(-) Approvals: David Ribeiro Alves: Looks good to me, approved Adar Dembo: Verified -- To view, visit http://gerrit.cloudera.org:8080/6717 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: I016ea401380f4c8c7b1fd907ff67cb595f377dd1 Gerrit-PatchSet: 9 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Adar Dembo <[email protected]> Gerrit-Reviewer: Adar Dembo <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: David Ribeiro Alves <[email protected]> Gerrit-Reviewer: Todd Lipcon <[email protected]>
