Mike Percy has submitted this change and it was merged. Change subject: external minicluster: expand EMC dir usage ......................................................................
external minicluster: expand EMC dir usage In order to test different disk configurations, it is becoming increasingly important to have end-to-end testing with nodes backed by multiple directories. External miniclusters by default use a single directory for each daemon's data (i.e. wals and data dirs fall under a single /cluster/daemon/data directory). This patch adds multi-directory support via a new 'num_data_dirs' parameter to ExternalMiniClusterOptions. Additionally, a 'wal_dir' parameter is added to ExternalDaemonOptions to separate the wal location from the data directories. If 'num_data_dirs' is greater than 1, each daemon will generate multiple paths, appending each with a numeric suffix, up to the number specified. E.g. EMCs that would have used the path /cluster/data, if specifying 'num_data_dirs' as 3, will now generate multiple data directories /cluster/data-0, /cluster/data-1, /cluster/data-2. The wal will be added to /cluster/wal. The new test multidir-cluster-itest demonstrates this. This test has been run via dist-test 2000 with no flakes. Results here: http://dist-test.cloudera.org/job?job_id=awong.1496701042.24862 Change-Id: Id2f5def6980ad394c8558ad97ba830f1b0257332 Reviewed-on: http://gerrit.cloudera.org:8080/6845 Reviewed-by: Adar Dembo <a...@cloudera.com> Tested-by: Kudu Jenkins Reviewed-by: Todd Lipcon <t...@apache.org> Reviewed-by: Mike Percy <mpe...@apache.org> --- M src/kudu/integration-tests/CMakeLists.txt M src/kudu/integration-tests/delete_table-itest.cc M src/kudu/integration-tests/disk_reservation-itest.cc M src/kudu/integration-tests/external_mini_cluster-itest-base.cc M src/kudu/integration-tests/external_mini_cluster-itest-base.h M src/kudu/integration-tests/external_mini_cluster.cc M src/kudu/integration-tests/external_mini_cluster.h M src/kudu/integration-tests/external_mini_cluster_fs_inspector.cc M src/kudu/integration-tests/log_verifier.cc M src/kudu/integration-tests/master_failover-itest.cc M src/kudu/integration-tests/master_migration-itest.cc A src/kudu/integration-tests/multidir_cluster-itest.cc M src/kudu/integration-tests/open-readonly-fs-itest.cc M src/kudu/integration-tests/ts_itest-base.h M src/kudu/integration-tests/ts_recovery-itest.cc 15 files changed, 231 insertions(+), 61 deletions(-) Approvals: Mike Percy: Looks good to me, approved Adar Dembo: Looks good to me, but someone else must approve Todd Lipcon: Looks good to me, but someone else must approve Kudu Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/6845 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: merged Gerrit-Change-Id: Id2f5def6980ad394c8558ad97ba830f1b0257332 Gerrit-PatchSet: 25 Gerrit-Project: kudu Gerrit-Branch: master Gerrit-Owner: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Adar Dembo <a...@cloudera.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: David Ribeiro Alves <davidral...@gmail.com> Gerrit-Reviewer: Kudu Jenkins Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Tidy Bot Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
