Todd Lipcon has submitted this change and it was merged. Change subject: KUDU-1856: always truncate containers when they get full ......................................................................
KUDU-1856: always truncate containers when they get full On a production cluster, I found that XFS's speculative preallocation feature often winds up permanently growing LBM containers without increasing their file sizes. It's not clear why this happens; I don't think we're running afoul of any of the conditions spelled out in the XFS FAQ [1]. Nevertheless, on this cluster the preallocated space accounted for almost a third of the total data consumption, so we need to address it. This patch changes the existing LBM container truncation behavior such that at startup, the container's file size no longer determines whether to truncate. Instead, all full containers are truncated unconditionally. The assumption here is that a no-op truncate should be free. We don't always run on XFS, and triggering its speculative preallocation is hard. But, we can approximate it by passing FALLOC_FL_KEEP_SIZE to fallocate(), which grows the file without changing its size. 1. http://xfs.org/index.php/XFS_FAQ#Q:_Is_speculative_preallocation_permanent.3F Change-Id: Ic959c59489a08f92efa2df5c85b22e56740f1346 Reviewed-on: http://gerrit.cloudera.org:8080/5852 Tested-by: Kudu Jenkins Reviewed-by: Mike Percy <mpe...@apache.org> Reviewed-by: Todd Lipcon <t...@apache.org> --- M src/kudu/fs/block_manager-test.cc M src/kudu/fs/data_dirs.cc M src/kudu/fs/fs_manager.cc M src/kudu/fs/log_block_manager.cc M src/kudu/util/env.h M src/kudu/util/env_posix.cc M src/kudu/util/file_cache.cc 7 files changed, 74 insertions(+), 30 deletions(-) Approvals: Mike Percy: Looks good to me, approved Todd Lipcon: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/5852 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ic959c59489a08f92efa2df5c85b22e56740f1346 Gerrit-PatchSet: 4 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Todd Lipcon <t...@apache.org> Gerrit-Reviewer: Will Berkeley <wdberke...@gmail.com>
