[
https://issues.apache.org/jira/browse/KUDU-3486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798592#comment-17798592
]
ASF subversion and git services commented on KUDU-3486:
-------------------------------------------------------
Commit fab3569f074954f9ba0a69dcc00f96bf2217ef60 in kudu's branch
refs/heads/master from 宋家成
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=fab3569f0 ]
KUDU-3486 Periodically send tombstoned replica report to delete it.
A tombstoned tablet replica might never be deleted since the
delete-type deletion could only occur when the tablet is deleted.
And the requests will only be sent to the voters, not including
the tombstoned ones.
The data of tombstone replica is deleted, but metadata is persisted
in memory, which will occupy a lot of memory, especially for the
tablets with big SchemaPB.
Tombstoned replicas can be deleted while processing heartbeats. So
making the tservers periodically send some reports with tombstoned
replicas will delete the useless tombstoned replicas. See KUDU-3486
for details.
Change-Id: I92d3b3e44f49ad24a95fa10f351c6bd55c6eca7b
Reviewed-on: http://gerrit.cloudera.org:8080/20600
Tested-by: Kudu Jenkins
Reviewed-by: Yifan Zhang <[email protected]>
> Tserver: Too many tombstone tablet may lead to high memory usage.
> -----------------------------------------------------------------
>
> Key: KUDU-3486
> URL: https://issues.apache.org/jira/browse/KUDU-3486
> Project: Kudu
> Issue Type: Bug
> Components: tserver
> Affects Versions: 1.14.0
> Reporter: Song Jiacheng
> Priority: Minor
> Attachments: image-2023-07-06-15-59-44-181.png
>
>
> There are two kinds of tablet replica deletion: tombstone and delete. A
> tombstone tablet replica might never be deleted since the delete-type
> deletion could only occur when the tablet is deleted, and the requests will
> be sent to the voters, not including the tombstone ones.
> Here is a example:
> Tablet T:
> replica A
> replica B
> replica C
> After rebalance:
> replica A
> replica B
> replica C(Tombstone)
> replica D
> When the tablet T is deleted, A B D are deleted, and C exists forever.
> Like this picture, the tablet had already been deleted at 3:00 am 13th Jun,
> but the tombstone replica still exists.
> !image-2023-07-06-15-59-44-181.png|width=568,height=261!
> The data of tombstone replica is deleted, but metadata is persisted in
> memory, especially the biggest one SchemaPB will occupy a lot of memory.
> In some of our clusters, tombstone replicas of each tserver could reach 50k ~
> 100k, which takes about 10G.
> It takes too much resource if adds a vector for each tablet to store the
> history tablet servers that used to hold a replica of the tablet. So I think
> periodically heartbeat might be a good way to solve the problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
