Mike Percy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13430 )
Change subject: [backup] KUDU-2786 Parallelize tables for backup and restore ...................................................................... [backup] KUDU-2786 Parallelize tables for backup and restore This patch adds a hidden, experimental option to run backups and restores in parallel across tables. Managing resources across parallel backups and restores is very difficult: the sizes of tables in terms of number of tablets and size of tables can vary by orders of magnitude across a cluster, and there are many resources which may be constrained depending on many factors: CPU, memory, disk I/O, network, number of executors available. This patch doesn't do resource management. It will kick off the jobs in parallel, and it's up to Spark to manage the resources of parallel jobs. Maybe this will work well, maybe it won't... that's why this is just experimental. I tested manually on a Spark cluster to verify that jobs are actually run in parallel. Change-Id: I02f0a818a6fa372ab3c696c11882284877ce207e Reviewed-on: http://gerrit.cloudera.org:8080/13430 Reviewed-by: Mike Percy <mpe...@apache.org> Tested-by: Mike Percy <mpe...@apache.org> --- M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduBackup.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/KuduRestore.scala M java/kudu-backup/src/main/scala/org/apache/kudu/backup/Options.scala M java/kudu-backup/src/test/scala/org/apache/kudu/backup/TestKuduBackup.scala 4 files changed, 69 insertions(+), 14 deletions(-) Approvals: Mike Percy: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13430 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I02f0a818a6fa372ab3c696c11882284877ce207e Gerrit-Change-Number: 13430 Gerrit-PatchSet: 3 Gerrit-Owner: Will Berkeley <wdberke...@gmail.com> Gerrit-Reviewer: Grant Henke <granthe...@apache.org> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Mike Percy <mpe...@apache.org> Gerrit-Reviewer: Will Berkeley <wdberke...@gmail.com>
