vttranlina commented on code in PR #2514:
URL: https://github.com/apache/james-project/pull/2514#discussion_r1855796686
##########
server/blob/blob-postgres/src/main/java/org/apache/james/blob/postgres/PostgresBlobStoreDAO.java:
##########
@@ -151,9 +165,20 @@ public Flux<BucketName> listBuckets() {
@Override
public Flux<BlobId> listBlobs(BucketName bucketName) {
+ return Flux.range(0, OFFSET_UNLIMITED)
+ .concatMap(offsetIndex -> listBlobsBatch(bucketName, offsetIndex *
queryBatchSize, queryBatchSize))
+ .takeUntil(List::isEmpty)
+ .flatMapIterable(Function.identity());
+ }
+
+ private Mono<List<BlobId>> listBlobsBatch(BucketName bucketName, int
offset, int batchSize) {
return postgresExecutor.executeRows(dsl ->
Flux.from(dsl.select(BLOB_ID)
.from(TABLE_NAME)
- .where(BUCKET_NAME.eq(bucketName.asString()))))
- .map(record -> blobIdFactory.parse(record.get(BLOB_ID)));
+ .where(BUCKET_NAME.eq(bucketName.asString()))
+ .limit(batchSize)
+ .offset(offset)))
Review Comment:
> We are missing ordering to ensure the stability of the transformation.
updated
> Also here we have a massive issue: the use case is to delete as we
iterate. This means that, without snapshot reads, the writes would impact the
reads and cause entries to be skept.
Indeed, that's correct. However, this is a GC feature. While batch commands
may not result in thorough cleanup initially, with repeated GC runs, the data
will gradually become more accurate.
What do you think? `collectList` or NOT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
