capistrant commented on code in PR #20382:
URL: https://github.com/apache/druid/pull/20382#discussion_r4073443569
##########
extensions-core/google-extensions/src/main/java/org/apache/druid/storage/google/GoogleDataSegmentKiller.java:
##########
@@ -61,19 +61,48 @@ public void kill(DataSegment segment) throws
SegmentLoadingException
Map<String, Object> loadSpec = segment.getLoadSpec();
final String bucket = MapUtils.getString(loadSpec, "bucket");
final String indexPath = MapUtils.getString(loadSpec, "path");
- final String descriptorPath = DataSegmentKiller.descriptorPath(indexPath);
try {
- deleteIfPresent(bucket, indexPath);
- // descriptor.json is a file to store segment metadata in deep storage.
This file is deprecated and not stored
- // anymore, but we still delete them if exists.
- deleteIfPresent(bucket, descriptorPath);
+ if (indexPath.endsWith("/")) {
+ // segment was pushed unzipped, so the path names a directory of
objects; delete every one of them
+ deleteObjectsInPath(bucket, indexPath);
+ } else {
+ deleteIfPresent(bucket, indexPath);
+ // descriptor.json is a file to store segment metadata in deep
storage. This file is deprecated and not stored
+ // anymore, but we still delete them if exists.
+ deleteIfPresent(bucket, DataSegmentKiller.descriptorPath(indexPath));
+ }
}
catch (StorageException e) {
throw new SegmentLoadingException(e, "Couldn't kill segment[%s]: [%s]",
segment.getId(), e.getMessage());
}
}
+ private void deleteObjectsInPath(String bucket, String pathPrefix) throws
SegmentLoadingException
+ {
+ try {
+ GoogleUtils.deleteObjectsInPath(
+ storage,
+ inputDataConfig,
+ bucket,
+ pathPrefix,
+ Predicates.alwaysTrue()
+ );
+ }
+ catch (StorageException e) {
+ throw e;
+ }
+ catch (Exception e) {
+ throw new SegmentLoadingException(
Review Comment:
feels like an odd exception type for a failed delete, but I see there is
precedent for using it like this in S3DataSegmentKiller so I will not object
outright
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
