bamaer opened a new issue, #9549:
URL: https://github.com/apache/hop/issues/9549

   ### What would you like to happen?
   
   `PAGE_NUMBER`, `OFFSET_LIMIT` and `CURSOR` pagination stop when a page has 
no payload. `exchangeHadPayload` 
([Rest.java:1179](https://github.com/apache/hop/blob/39b3d636b2e003ab9e857eb385f549cbc3e9248c/plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java#L1179))
 decides that:
   
   - with a result split path, from the number of items the path returns;
   - without one, from whether the raw response body is empty.
   
   APIs that wrap results in an envelope never return an empty body. The 
Ironclad public API returns `{"page":7,"pageSize":20,"count":87,"list":[]}` 
past the last page. Without a split path, Hop keeps requesting pages until the 
"maximum pages" safeguard (default 256) is reached. Setting the split path to 
`$.list` avoids this, but only when the rows are meant to be split; a pipeline 
that wants one row per page has no way to say which part of the body holds the 
items.
   
   Proposal:
   
   - Add an optional "Items JSON path" (and XPath for XML) to the REST 
connection. When set, a page whose items expression returns nothing ends 
pagination. When empty, the current behaviour stays.
   - Optionally add a "Total count JSON path" (for example `$.count`), so 
pagination stops after the last full page instead of fetching one extra, empty 
page.
   - Add both options to the REST connection editor and the docs.
   
   Tests: an integration test against an envelope endpoint in the `paging` 
nginx mock, without a result split path, that expects the configured number of 
pages and not the safeguard.
   
   ### Issue Priority
   
   Priority: 2
   
   ### Issue Component
   
   Component: Metadata, Component: Transforms
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to