[ https://issues.apache.org/jira/browse/NIFI-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450272#comment-15450272 ]
James Wing commented on NIFI-2631: ---------------------------------- [~jgresock], this looks like a great idea, thanks for contributing. I have a couple of thoughts: I was able to reproduce the large bucket problem you brought up here all too easily. I aimed ListS3 at a bucket of log files and started it. ListS3 ran for about 1 minute solid without producing flowfiles or bulletins, then dumped ~90,000 flowfiles in the success queue. 90,000 objects really isn't big in S3 terms, so aligning the ListS3 processor to the paging pattern of the S3 API seems ideal. Also, I noticed a [comment in NIFI-840|https://issues.apache.org/jira/browse/NIFI-840?focusedCommentId=15270156&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15270156], where ListS3 was added, that suggests this was a known issue at the time. From [~adamonduty]: bq. ListS3 also persists state after each trigger. I didn't think about this until after submitting this PR, but really we should be committing flowfiles and persisting state after every batch (1000 objects by default) to avoid unbounded memory growth on large buckets. So in some cases the initial restore is only one of potentially many roundtrips to the state manager per trigger. But now that I'm on board, I'm curious why this should be optional behavior? If users have small buckets, they probably will not notice or be inconvenienced by a small number of session.commit() calls. Anyone with a large bucket will almost certainly be frustrated by a long delay followed by a massive dump of flowfiles. The optional parameter would safely preserve the current behavior, but I'm not a big fan of adding potentially confusing options for what now seems very normal. What is the case for not committing each page of results from S3? > ListS3 improvements: "Use versions" and "Commit mode" > ----------------------------------------------------- > > Key: NIFI-2631 > URL: https://issues.apache.org/jira/browse/NIFI-2631 > Project: Apache NiFi > Issue Type: Improvement > Affects Versions: 0.7.0 > Reporter: Joseph Gresock > Assignee: Joseph Gresock > Priority: Minor > Fix For: 1.1.0, 0.8.0 > > > Our team needs to be able to list individual versions in S3. We also ran > into a use case where a bucket with many objects (over 1 million in our case) > seemed to cause ListS3 to run forever. The S3 list command finished in a few > minutes, but we believe it was taking a very long time for NiFi to commit all > the flow files at once. > To handle this use case, we added a Commit Mode property to ListS3 that > allows you specify that you want to commit "Per page" vs. "Once". This has > proven to correctly emit the flow files as the S3 paging progresses. > We also implemented support for S3 List Versions, which includes the > "s3.version" and "s3.isLatest" attributes if applicable. The "s3.version" > attribute can in turn be used in the FetchS3 processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)