Adar Dembo has submitted this change and it was merged. Change subject: KUDU-1970: node density integration test ......................................................................
KUDU-1970: node density integration test This patch introduces a new itest that simulates a storage-dense Kudu deployment. The idea is simple: rather than actually generating and storing lots of data (which is both time intensive and developer unfriendly), let's run a workload that produces a lot of metadata with a minimal amount of data. This is cheaper, and the metadata can proxy for data in areas we care about (such as start up time, thread count, memory usage, etc.). The test itself isn't that interesting; most of the challenge was in running it repeatedly to determine which flag values yielded the most metadata. In a run of the test on a 48 core el6.6 machine (max_blocks_per_container=8, num_tablets=1000, num_seconds=240), I produced ~110K blocks across ~21k LBM containers, which yielded a subsequent ~100s LBM startup time. I made the following modifications elsewhere to make this work: - TestWorkload now supports arbitrary schemas. - EMC-based tests can configure the amount of time they wait on each daemon process to start as the server info file isn't dumped until after FS startup is complete (maybe that should be changed?) - The benchmarks.sh script runs the test with some customized parameters. I also snuck in changes to remove an unused variable from random.h and to switch TestWorkload from kudu::Thread to std::thread. Change-Id: Ie9b5d01557eb41d386ce92f576ed01ec658e8e7d Reviewed-on: http://gerrit.cloudera.org:8080/6662 Tested-by: Kudu Jenkins Reviewed-by: David Ribeiro Alves <davidral...@gmail.com> --- M src/kudu/integration-tests/CMakeLists.txt A src/kudu/integration-tests/dense_node-itest.cc M src/kudu/integration-tests/external_mini_cluster-itest-base.cc M src/kudu/integration-tests/external_mini_cluster-itest-base.h M src/kudu/integration-tests/external_mini_cluster.cc M src/kudu/integration-tests/external_mini_cluster.h M src/kudu/integration-tests/test_workload.cc M src/kudu/integration-tests/test_workload.h M src/kudu/scripts/benchmarks.sh M src/kudu/tools/data_gen_util.cc M src/kudu/tools/data_gen_util.h M src/kudu/tools/kudu-ts-cli-test.cc M src/kudu/util/random.h 13 files changed, 355 insertions(+), 48 deletions(-) Approvals: David Ribeiro Alves: Looks good to me, approved Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/6662 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Ie9b5d01557eb41d386ce92f576ed01ec658e8e7d Gerrit-PatchSet: 12 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Dan Burkert <danburk...@apache.org> Gerrit-Reviewer: David Ribeiro Alves <davidral...@gmail.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
