bamaer opened a new issue, #9549: URL: https://github.com/apache/hop/issues/9549
### What would you like to happen? `PAGE_NUMBER`, `OFFSET_LIMIT` and `CURSOR` pagination stop when a page has no payload. `exchangeHadPayload` ([Rest.java:1179](https://github.com/apache/hop/blob/39b3d636b2e003ab9e857eb385f549cbc3e9248c/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java#L1179)) decides that: - with a result split path, from the number of items the path returns; - without one, from whether the raw response body is empty. APIs that wrap results in an envelope never return an empty body. The Ironclad public API returns `{"page":7,"pageSize":20,"count":87,"list":[]}` past the last page. Without a split path, Hop keeps requesting pages until the "maximum pages" safeguard (default 256) is reached. Setting the split path to `$.list` avoids this, but only when the rows are meant to be split; a pipeline that wants one row per page has no way to say which part of the body holds the items. Proposal: - Add an optional "Items JSON path" (and XPath for XML) to the REST connection. When set, a page whose items expression returns nothing ends pagination. When empty, the current behaviour stays. - Optionally add a "Total count JSON path" (for example `$.count`), so pagination stops after the last full page instead of fetching one extra, empty page. - Add both options to the REST connection editor and the docs. Tests: an integration test against an envelope endpoint in the `paging` nginx mock, without a result split path, that expects the configured number of pages and not the safeguard. ### Issue Priority Priority: 2 ### Issue Component Component: Metadata, Component: Transforms -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
