justinmclean opened a new issue, #10275:
URL: https://github.com/apache/gravitino/issues/10275
### What would you like to be improved?
PolicyMetaService.deletePolicyVersionsByRetentionCount(versionRetentionCount,
limit) applies limit inside a loop for each policy. In production, this is
called by the relational garbage collector with
GARBAGE_COLLECTOR_SINGLE_DELETION_LIMIT (100). With multiple policies, one GC
iteration can delete up to limit * numberOfPolicies, exceeding the intended
single-batch cap and causing unexpectedly large cleanup operations.
### How should we improve?
Enforce limit as a global budget per method call. Track remaining = limit,
and for each policy call softDeletePolicyVersionsByRetentionLine(...,
remaining), subtracting deleted rows from remaining; stop when remaining <= 0.
This preserves bounded batch behavior while still progressing cleanup across
runs.
here a test to help:
```
@TestTemplate
public void testDeletePolicyVersionsByRetentionCountRespectGlobalLimit()
throws IOException {
BaseMetalake metalake = createAndInsertMakeLake(METALAKE_NAME +
"_global_limit");
PolicyEntity policy1 =
createPolicy(
RandomIdGenerator.INSTANCE.nextId(),
NamespaceUtil.ofPolicy(metalake.name()),
"policy_limit_1",
AUDIT_INFO);
PolicyEntity policy2 =
createPolicy(
RandomIdGenerator.INSTANCE.nextId(),
NamespaceUtil.ofPolicy(metalake.name()),
"policy_limit_2",
AUDIT_INFO);
backend.insert(policy1, false);
backend.insert(policy2, false);
PolicyEntity policy1V2 =
PolicyEntity.builder()
.withId(policy1.id())
.withNamespace(policy1.namespace())
.withName(policy1.name())
.withPolicyType(policy1.policyType())
.withComment(policy1.comment())
.withEnabled(!policy1.enabled())
.withContent(policy1.content())
.withAuditInfo(AUDIT_INFO)
.build();
PolicyEntity policy2V2 =
PolicyEntity.builder()
.withId(policy2.id())
.withNamespace(policy2.namespace())
.withName(policy2.name())
.withPolicyType(policy2.policyType())
.withComment(policy2.comment())
.withEnabled(!policy2.enabled())
.withContent(policy2.content())
.withAuditInfo(AUDIT_INFO)
.build();
backend.update(policy1.nameIdentifier(), Entity.EntityType.POLICY, e ->
policy1V2);
backend.update(policy2.nameIdentifier(), Entity.EntityType.POLICY, e ->
policy2V2);
int deletedCount =
PolicyMetaService.getInstance().deletePolicyVersionsByRetentionCount(1L, 1);
assertEquals(1, deletedCount);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
