Hello Dan Burkert, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6662
to look at the new patch set (#11).
Change subject: KUDU-1970: node density integration test
......................................................................
KUDU-1970: node density integration test
This patch introduces a new itest that simulates a storage-dense Kudu
deployment. The idea is simple: rather than actually generating and storing
lots of data (which is both time intensive and developer unfriendly), let's
run a workload that produces a lot of metadata with a minimal amount of
data. This is cheaper, and the metadata can proxy for data in areas we care
about (such as start up time, thread count, memory usage, etc.). The test
itself isn't that interesting; most of the challenge was in running it
repeatedly to determine which flag values yielded the most metadata.
In a run of the test on a 48 core el6.6 machine (max_blocks_per_container=8,
num_tablets=1000, num_seconds=240), I produced ~110K blocks across ~21k LBM
containers, which yielded a subsequent ~100s LBM startup time.
I made the following modifications elsewhere to make this work:
- TestWorkload now supports arbitrary schemas.
- EMC-based tests can configure the amount of time they wait on each daemon
process to start as the server info file isn't dumped until after FS
startup is complete (maybe that should be changed?)
- The benchmarks.sh script runs the test with some customized parameters.
I also snuck in changes to remove an unused variable from random.h and to
switch TestWorkload from kudu::Thread to std::thread.
Change-Id: Ie9b5d01557eb41d386ce92f576ed01ec658e8e7d
---
M src/kudu/integration-tests/CMakeLists.txt
A src/kudu/integration-tests/dense_node-itest.cc
M src/kudu/integration-tests/external_mini_cluster-itest-base.cc
M src/kudu/integration-tests/external_mini_cluster-itest-base.h
M src/kudu/integration-tests/external_mini_cluster.cc
M src/kudu/integration-tests/external_mini_cluster.h
M src/kudu/integration-tests/test_workload.cc
M src/kudu/integration-tests/test_workload.h
M src/kudu/scripts/benchmarks.sh
M src/kudu/tools/data_gen_util.cc
M src/kudu/tools/data_gen_util.h
M src/kudu/tools/kudu-ts-cli-test.cc
M src/kudu/util/random.h
13 files changed, 355 insertions(+), 48 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/62/6662/11
--
To view, visit http://gerrit.cloudera.org:8080/6662
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie9b5d01557eb41d386ce92f576ed01ec658e8e7d
Gerrit-PatchSet: 11
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <danburk...@apache.org>
Gerrit-Reviewer: David Ribeiro Alves <davidral...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Tidy Bot
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
