Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18152 )
Change subject: KUDU-3344: catalog manager clean up metadata for deleted tables/tablets ...................................................................... KUDU-3344: catalog manager clean up metadata for deleted tables/tablets Kudu masters now retain metadata for deleted tables and tablets forever, and the leader master loads all of them into memory when starts. If we have a lot of tables and tablets in a cluster, memory usage of the leader master will be large and it will take a long time to start the leader master. Consider that in many cases users drop tables and partitions, useless metadata should be cleaned up in backgroud tasks. But it's hard to decide when we should clean them up, because the deletion of tablet replicas is asynchronous. If metadata is deleted before the tablet data is deleted, the unknown tablet reported by a tablet server will not be processed by catalog manager and we must delete it manually. So this patch add a new flag 'deleted_table_and_tablet_reserved_secs', its default value is the same as 'unresponsive_ts_rpc_timeout_ms', we can roughly assume that after this amount of time the tablet data will be actually deleted and it's safe to delete its metadata entry from sys.catalog table and in-memory map. Change-Id: Idefa2ee2f5108ba913fe0057a4061c3c28351547 Reviewed-on: http://gerrit.cloudera.org:8080/18152 Tested-by: Alexey Serbin <aser...@cloudera.com> Reviewed-by: Alexey Serbin <aser...@cloudera.com> --- M src/kudu/client/client.cc M src/kudu/integration-tests/master-stress-test.cc M src/kudu/integration-tests/ts_tablet_manager-itest.cc M src/kudu/master/auto_rebalancer.cc M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test.cc M src/kudu/master/master.proto 8 files changed, 267 insertions(+), 24 deletions(-) Approvals: Alexey Serbin: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/18152 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Idefa2ee2f5108ba913fe0057a4061c3c28351547 Gerrit-Change-Number: 18152 Gerrit-PatchSet: 8 Gerrit-Owner: Yifan Zhang <chinazhangyi...@163.com> Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com> Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yifan Zhang <chinazhangyi...@163.com> Gerrit-Reviewer: yejiabao <yejia...@huawei.com>
