matanper opened a new issue, #70504: URL: https://github.com/apache/doris/issues/70504
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4. master (`e125ea85867`) has the same code. ### What's Wrong? In cloud mode, when object storage throttles the recycler's deletes (S3 `503 SlowDown`), `recycle_rowsets` keeps the recycle keys of every rowset in the batch, including the rowsets whose objects it did delete. The next round lists and deletes them again. Under sustained throttling, the rowset recycle backlog stops shrinking: each round re-sends the same deletes to a bucket that is already throttling, and the oldest expired rowsets stay queued indefinitely. ### Why `InstanceRecycler::recycle_rowsets` (`cloud/src/recycler/recycler.cpp:5772`) groups expired rowsets into batches of up to `recycler_max_tasks_per_batch` (1000) and submits one worker per batch (`:6018-6037`): ```cpp if (delete_rowset_data(rowsets_to_delete, RowsetRecyclingState::FORMAL_ROWSET, metrics_context, &versioned_delete_bitmap_key_groups) != 0) { LOG(WARNING) << "failed to delete rowset data, instance_id=" << instance_id_; return; } ... if (txn_remove(txn_kv_.get(), rowset_keys_to_delete) != 0) { ``` `delete_rowset_data` returns non-zero when any delete request in the batch fails, and the worker then returns before `txn_remove`. A single failed `DeleteObjects` request therefore keeps all of the batch's recycle keys, even though most of its objects are already gone. Since #63776 and #65508 the recycler's S3 client no longer retries `SlowDown` itself, so every throttled request now surfaces as a failed batch. #63377 fixed the same all-or-nothing pattern for `recycle_tablets` only. `recycle_tmp_rowsets` (`:2641-2660`) has the same behaviour. There is also no pacing after a throttled response, so the next batches are sent immediately and are throttled too. ### What You Expected? The recycler removes the recycle keys of the rowsets whose data it deleted, keeps only the failed ones for the next round, and slows down after the object store throttles it, so a throttled round still makes progress. ### How to Reproduce? 1. A cloud-mode cluster with an S3 storage vault, and enough compaction that each recycle round has many batches of expired rowsets. 2. Make some `DeleteObjects` requests fail, for example with an S3 request-rate limit on the bucket, or a fault injected in `delete_rowset_data` for some of a batch's objects. 3. Compare `num_expired` and `num_recycled` in the `recycle rowsets finished` log line across rounds: the rowsets of every batch that had one failure are counted as expired again in the next round, although their objects were deleted. A recycler unit test with one batch of 10 rowsets, where the delete of 2 rowsets' objects fails, shows it: after one round all 10 recycle keys are still present. ### Anything Else? Proposed fix, following #63377: - Have the object-store delete report which keys it could not delete (per-key `Errors` of a `DeleteObjects` response, every key of a request that failed as a whole, and the keys of chunks never sent). - Have `delete_rowset_data` report the rowsets it deleted completely, and have `recycle_rowsets` / `recycle_tmp_rowsets` remove only those rowsets' keys. - Add a bounded, jittered backoff per object-store client after a `SlowDown`/`503`/`429`, with mutable min/max configs, so later requests wait instead of being throttled too. Until then, enabling the Meta Service's S3 rate limiter (`enable_s3_rate_limiter`, `s3_put_token_per_second`) at a low PUT rate keeps the recycler under the throttling threshold. We have a patch with unit tests and can open a PR. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
