Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/5403
to look at the new patch set (#4).
Change subject: KUDU-1508: enforce block limit on lbm containers
......................................................................
KUDU-1508: enforce block limit on lbm containers
This patch introduces the following machinery:
1. LBM containers can now be limited to a certain number of blocks using
log_container_max_blocks. This is a soft limit; if the limit is reduced
and the server restarted, existing containers may exceed it.
The default value means "no limit except when vulnerable to KUDU-1508".
2. When we construct a block manager, we check whether it's an el6 kernel
(that is, whether it's vulnerable to KUDU-1508).
3. At data directory opening time, we figure out whether it's vulnerable to
KUDU-1508. If it is, and if it's running on an ext4 filesystem, we
prescribe a particular block limit that depends on the filesystem's
block size. These limits were arrived at via experimentation (see commit
4923a74). They are likely more conservative than they need to be (they
assume one extent per filesystem block with every other extent punched
out), but with the addition of the file cache, the scalability overhead
should be minimal.
4. The prescribed limit is passed into each container as it is instantiated.
I ran TestContainerWithManyHoles on several loopback ext4 filesystems on
el6.6. I tested block sizes of 1024, 2048, and 4096, each with and without
the fix. Without the fix, the filesystems needed to be repaired 100% of the
time. With the fix, they never needed repair.
Change-Id: I63576d2bf2cc61b2deb70f7e166f08e0d4c66543
---
M src/kudu/experiments/KUDU-1508/run_test.sh
M src/kudu/fs/block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
M src/kudu/fs/log_block_manager.h
M src/kudu/util/env.h
M src/kudu/util/env_posix.cc
6 files changed, 325 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/03/5403/4
--
To view, visit http://gerrit.cloudera.org:8080/5403
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I63576d2bf2cc61b2deb70f7e166f08e0d4c66543
Gerrit-PatchSet: 4
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
