github-actions[bot] commented on code in PR #70119:
URL: https://github.com/apache/doris/pull/70119#discussion_r4462589441
##########
cloud/src/recycler/s3_accessor.cpp:
##########
@@ -613,4 +623,80 @@ int S3Accessor::check_versioning() {
return obj_client_->check_versioning(conf_.bucket).status.code;
}
+int GcsAccessor::delete_prefix_impl(const std::string& path_prefix, int64_t
expiration_time) {
+ LOG_INFO("begin delete prefix").tag("uri", to_uri(path_prefix));
+
+ int ret = 0;
+ int cnt = 0;
+ int skip = 0;
+ int64_t del_nonexisted = 0;
+ int del = 0;
+ auto iter = obj_client_->list_objects({.bucket = conf_.bucket, .prefix =
get_key(path_prefix)});
+ for (;;) {
+ auto result = iter->next();
+ if (!result.object.has_value()) {
+ if (result.resp.status.code != ObjStorageStatus::NOT_FOUND &&
+ result.resp.status.code != ObjStorageStatus::OK) {
+ ret = result.resp.status.code;
+ }
+ break;
+ }
+ auto& obj = *result.object;
+ if (!(++cnt % 100)) {
+ LOG_INFO("loop delete prefix")
+ .tag("uri", to_uri(path_prefix))
+ .tag("total_obj_cnt", cnt)
+ .tag("deleted", del)
+ .tag("del_nonexisted", del_nonexisted)
+ .tag("skipped", skip);
+ }
+ if (expiration_time > 0 && obj.mtime_s > expiration_time) {
+ skip++;
+ continue;
+ }
+ del++;
+
+ // FIXME(plat1ko): Delete objects by batch with genuine GCS client
+ int del_ret =
+ obj_client_->delete_object({.bucket = conf_.bucket, .key =
obj.key}).status.code;
+ del_nonexisted += (del_ret == ObjStorageStatus::NOT_FOUND);
+ static_assert(ObjStorageStatus::OK == 0);
+ if (del_ret != ObjStorageStatus::OK && del_ret !=
ObjStorageStatus::NOT_FOUND) {
+ ret = del_ret;
Review Comment:
[P2] Stop scanning the prefix after a failed DELETE
If LIST succeeds while DELETE consistently returns AccessDenied or SlowDown,
recording the error here still proceeds through every remaining object and
subsequent listing page. Thus delete_all/delete_directory perform N synchronous
failed requests before their caller can observe the failure. The waiting
SyncExecutor cannot cancel queued recycling based on this result until the
callback returns, and recycler shutdown waits for these workers. This loses
delete_objects_recursively's explicit stop-after-a-failed-batch behavior and
defeats the intended fast failure on SlowDown. Break or return on the first
DELETE error other than NOT_FOUND, or use bounded cancellable individual-delete
tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
