Hi Martin, Thanks for reporting the issue. I have filed a JIRA [1] and a fix with additional testing coverage [2].
[1] https://issues.apache.org/jira/browse/NIFI-15685 [2] https://github.com/apache/nifi/pull/10983 Thanks, Pierre Le lun. 9 mars 2026 à 11:57, Fulcher, Tim via users <[email protected]> a écrit : > > I’m pretty sure I came across similar during my upgrade. > > I couldn’t work out why the ListS3 generated *millions* of items in the out > queue for a relatively small set of files in the bucket. > > I did 2 things that seemed to work better (for me) – move to V2 of List > Objects and then decided to use a RecordWriter approach. > > I’m a bit amazed there isn’t a test case that’s failing for nifi2! > > > > Tim > > > > From: Hengesbach, Martin <[email protected]> > Sent: 09 March 2026 10:34 > To: [email protected] > Subject: [EXTERNAL] Pagination bug in ListS3 > > > > CAUTION: This email originated from outside of the organisation. Do not click > links or open attachments unless you recognise the sender and know the > content is safe. > > > > Hi, > > > > I think I found a critical pagination bug in the ListS3 processor. I just > upgraded from 1.28 to 2.8. With 1.28 everything runs well, the problems > occurs with 2.8. > > > > The problem is independent from the “Listing Strategy” but depends on the > “List Type”. The problem occurs only with “List Type = List Objects V1”, the > “List Objects V2” is working correctly so can be used as a workaround. If you > have a bucket with less than 1000 objects (or have restricted the number of > objects with a “Prefix” to less than 1000), everything works fine. > > > > But if there is a bucket with more than 1000 objects and you use “List > Objects V1” ListS3 fetches the first 1000 objects and then gets the first > 1000 again and again. The loop never stops. I turned on TRACE level logging, > there it’s easy to see that it starts after 1000 objects again with the first > one and not with the next ones. > > > > I assume that the ContinuationToken is not passed correctly to subsequent > requests causing infinite “while(isTruncated())” loops for buckets with >1000 > objects. > > > > I have this problem with 2 totally different AWS buckets. One is a private > and the other one the publicly available openalex bucket. I added a test case > (see attached JSON). With “List Objects V1” the loops runs indefinitely, if > you change to “List Objects V2” you get the correct number of 5997 flowfiles. > Possibly you need to add a proxy. > > > > Can anyone confirm this problem? How to proceed (I don’t have a Jira account). > > > > > > Thanks > > Martin > > ________________________________ > > FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH. > Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim HRB > 101892. > Geschäftsführer: Prof. Dr. Wolfram Horstmann. > Vorsitzende des Aufsichtsrats: MinR’in Dr. Elise Grauer. > > FIZ Karlsruhe ist zertifiziert mit dem Siegel "audit berufundfamilie". > > This e-mail (and any attachments) may contain privileged and/or confidential > information which may be protected by copyright or other intellectual > property rights. If you are not the intended recipient please do not > disclose, copy, distribute, disseminate or take any action in reliance on it. > If you have received this e-mail in error please reply to the sender and then > immediately delete it (including, any attachments). Should you wish to > communicate with us by e-mail we cannot guarantee the security of any data > outside our own computer systems or that any e-mail will be virus free. > > Any information contained in this e-mail may be subject to applicable terms > and conditions and must not be construed as giving investment advice within > or outside the United Kingdom or the Republic of Ireland. > > Telephone conversations and calls via other telecommunication facilities may > be recorded, including to comply with our legal and/or regulatory > requirements and/or to monitor the quality of our service. For information > about how we use your personal data, including your legal rights, please > refer to our privacy policy at: https://am.landg.com/privacy-policy/ . > > Legal & General Investment Management Limited (Company number 02091894), LGIM > Real Assets (Operator) Limited (Company number 05522016), Legal & General > (Unit Trust Managers) Limited (Company number 01009418), GO ETF Solutions LLP > (Company number OC329482) and LGIM Corporate Director Limited (Company number > 07105051) are each authorised and regulated by the Financial Conduct > Authority. All are registered in England & Wales with a registered office at > One Coleman Street, London, EC2R 5AA. > > Legal and General Assurance (Pensions Management) Limited (Company number > 01006112) is authorised by the Prudential Regulation Authority and regulated > by the Financial Conduct Authority and the Prudential Regulation Authority. > It is registered in England & Wales with a registered office at One Coleman > Street, London, EC2R 5AA. > > Legal & General Property Limited (Registration number 02091897) is authorised > and regulated by the Financial Conduct Authority for insurance mediation > activities. It is registered in England & Wales with a registered office at > One Coleman Street, London, EC2R 5AA. > > LGIM Managers (Europe) Limited is authorised and regulated by the Central > Bank of Ireland (Reference No C173733). It is registered in the Republic of > Ireland (Number 609677) with its principal business address at 33/34 Sir John > Rogerson's Quay, Dublin 2, D02 XK09. > > The ultimate parent company is Legal & General Group PLC (Company number > 01417162) which is registered in England & Wales and has a registered office > at One Coleman Street, London, EC2R 5AA.
