Glad to hear that!
However, if you have an idea how to handle these listBlobs, please feel
free to send a PR with your proposed fix.
Regards,
Omar
On Tue, Jan 26, 2021 at 2:14 PM Mark Andreev <mark.andr...@gmail.com> wrote:
> I found a solution for this issue. When I use Azure Api directly and
> operate with Page all works. It is useful when your azure blob storage
> contains a lot of blobs (in my case ~250k).
>
> from("direct:list")
> .routeId("listBlobs")
> .process(exchange -> {
> BlobServiceClient blobServiceClient = new
> BlobServiceClientBuilder()
> .endpoint(String
> .format("https://%s.blob.core.windows.net";,
> AzureBlobConfigHelper.ACCOUNT))
> .credential(new
> StorageSharedKeyCredential(AzureBlobConfigHelper.ACCOUNT,
> AzureBlobConfigHelper.ACCESS_KEY))
> .buildClient();
> BlobContainerClient containerClient =
>
>
> blobServiceClient.getBlobContainerClient(AzureBlobConfigHelper.BLOB_CONTAINER_NAME);
> ListBlobsOptions blobsOptions = new
> ListBlobsOptions().setMaxResultsPerPage(5);
> Duration duration = Duration.of(10, ChronoUnit.SECONDS);
>
> Iterator<BlobItem>
> iterator = containerClient.listBlobs(blobsOptions,
> duration).iterator();
> exchange.getIn().setBody(iterator);
> })
> .loopDoWhile(exchange -> {
> Iterator<BlobItem> iterator =
> (Iterator<BlobItem>)
> exchange.getIn().getBody(Iterator.class);
> return iterator.hasNext();
> })
> .process(exchange -> {
> Iterator<BlobItem> iterator =
> (Iterator<BlobItem>)
> exchange.getIn().getBody(Iterator.class);
> exchange.getIn().setBody(iterator.next());
> })
> .process(exchange -> {
> BlobItem item = exchange.getIn().getBody(BlobItem.class);
> log.info(item);
> })
> .end();
>
> On Tue, 26 Jan 2021 at 13:57, Omar Al-Safi <o...@oalsafi.com> wrote:
>
> > Hi,
> >
> > Actually you are doing nothing wrong here. Apparently when you set
> > maxResultsPerPage, in Azure client, that means it will return pageable
> > list, now in every pagelist it will return the max blobs that you set
> > earlier in `maxResultsPerPage` which is IMO pretty weird from Azure's
> side.
> > Now in the Camel component side, we just collect all that list of blobs
> in
> > the pages into one list and return that in the body and hence the reason
> > why you see no effects. I hope Azure will improve the API in the future
> to
> > allow alternative options as max results regardless of the pageable list.
> >
> > Regards,
> > Omar
> >
> > On Mon, Jan 25, 2021 at 8:04 PM Mark Andreev <mark.andr...@gmail.com>
> > wrote:
> >
> > > Hi, I use Spring 2.4.2 with camel-azure-storage-blob 3.7.0.
> > >
> > > When I set BlobConstants.LIST_BLOB_OPTIONS as "new
> > > ListBlobsOptions().setMaxResultsPerPage(5)" driver returns all entities
> > > from blob.
> > >
> > > from("direct:list")
> > > .routeId("listBlobs")
> > > .setHeader(BlobConstants.LIST_BLOB_OPTIONS)
> > >
> > > .constant(new ListBlobsOptions().setMaxResultsPerPage(5))
> > > .to(
> > > String.format(
> > > "
> > >
> > >
> >
> azure-storage-blob://%s/%s?accessKey=RAW(%s)&operation=listBlobs&synchronous=true
> > > ",
> > > ACCOUNT,
> > > BLOB_CONTAINER_NAME,
> > > ACCESS_KEY
> > > )
> > > )
> > > .process(exchange -> {
> > > int itemsCount = exchange.getIn().getBody(List.class).size();
> > > log.info(String.format("Items count = '%d'", itemsCount));
> > > });
> > >
> > > How should I use this API to fetch only 5 items?
> > >
> > > Full example:
> > >
> > >
> >
> https://github.com/mrk-andreev/example-camel-azure-blob-pagination-bug/blob/master/src/main/java/name/mrkandreev/camel/MyApplication.java
> > > --
> > > Best regards,
> > > Mark Andreev
> > >
> >
>
>
> --
> Best regards,
> Mark Andreev
>
