I’m pretty sure I came across similar during my upgrade. I couldn’t work out why the ListS3 generated *millions* of items in the out queue for a relatively small set of files in the bucket. I did 2 things that seemed to work better (for me) – move to V2 of List Objects and then decided to use a RecordWriter approach. I’m a bit amazed there isn’t a test case that’s failing for nifi2!
Tim From: Hengesbach, Martin <[email protected]> Sent: 09 March 2026 10:34 To: [email protected] Subject: [EXTERNAL] Pagination bug in ListS3 CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the sender and know the content is safe. Hi, I think I found a critical pagination bug in the ListS3 processor. I just upgraded from 1.28 to 2.8. With 1.28 everything runs well, the problems occurs with 2.8. The problem is independent from the “Listing Strategy” but depends on the “List Type”. The problem occurs only with “List Type = List Objects V1”, the “List Objects V2” is working correctly so can be used as a workaround. If you have a bucket with less than 1000 objects (or have restricted the number of objects with a “Prefix” to less than 1000), everything works fine. But if there is a bucket with more than 1000 objects and you use “List Objects V1” ListS3 fetches the first 1000 objects and then gets the first 1000 again and again. The loop never stops. I turned on TRACE level logging, there it’s easy to see that it starts after 1000 objects again with the first one and not with the next ones. I assume that the ContinuationToken is not passed correctly to subsequent requests causing infinite “while(isTruncated())” loops for buckets with >1000 objects. I have this problem with 2 totally different AWS buckets. One is a private and the other one the publicly available openalex bucket. I added a test case (see attached JSON). With “List Objects V1” the loops runs indefinitely, if you change to “List Objects V2” you get the correct number of 5997 flowfiles. Possibly you need to add a proxy. Can anyone confirm this problem? How to proceed (I don’t have a Jira account). Thanks Martin ________________________________ FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH. Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim HRB 101892. Geschäftsführer: Prof. Dr. Wolfram Horstmann. Vorsitzende des Aufsichtsrats: MinR’in Dr. Elise Grauer. FIZ Karlsruhe ist zertifiziert mit dem Siegel "audit berufundfamilie". This e-mail (and any attachments) may contain privileged and/or confidential information which may be protected by copyright or other intellectual property rights. If you are not the intended recipient please do not disclose, copy, distribute, disseminate or take any action in reliance on it. If you have received this e-mail in error please reply to the sender and then immediately delete it (including, any attachments). Should you wish to communicate with us by e-mail we cannot guarantee the security of any data outside our own computer systems or that any e-mail will be virus free. Any information contained in this e-mail may be subject to applicable terms and conditions and must not be construed as giving investment advice within or outside the United Kingdom or the Republic of Ireland. Telephone conversations and calls via other telecommunication facilities may be recorded, including to comply with our legal and/or regulatory requirements and/or to monitor the quality of our service. For information about how we use your personal data, including your legal rights, please refer to our privacy policy at: https://am.landg.com/privacy-policy/ . Legal & General Investment Management Limited (Company number 02091894), LGIM Real Assets (Operator) Limited (Company number 05522016), Legal & General (Unit Trust Managers) Limited (Company number 01009418), GO ETF Solutions LLP (Company number OC329482) and LGIM Corporate Director Limited (Company number 07105051) are each authorised and regulated by the Financial Conduct Authority. All are registered in England & Wales with a registered office at One Coleman Street, London, EC2R 5AA. Legal and General Assurance (Pensions Management) Limited (Company number 01006112) is authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. It is registered in England & Wales with a registered office at One Coleman Street, London, EC2R 5AA. Legal & General Property Limited (Registration number 02091897) is authorised and regulated by the Financial Conduct Authority for insurance mediation activities. It is registered in England & Wales with a registered office at One Coleman Street, London, EC2R 5AA. LGIM Managers (Europe) Limited is authorised and regulated by the Central Bank of Ireland (Reference No C173733). It is registered in the Republic of Ireland (Number 609677) with its principal business address at 33/34 Sir John Rogerson's Quay, Dublin 2, D02 XK09. The ultimate parent company is Legal & General Group PLC (Company number 01417162) which is registered in England & Wales and has a registered office at One Coleman Street, London, EC2R 5AA.
