bamaer commented on code in PR #9573:
URL: https://github.com/apache/hop/pull/9573#discussion_r4625257125


##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -927,6 +930,7 @@ protected void runPaginationLoop(Object[] rowData) throws 
HopException {
         connection != null ? connection.getPaginationType() : 
RestPaginationType.NONE;
     PaginationState state = new PaginationState();
     state.effectiveLimit = resolvePagingLimit(connection);
+    state.pageNumber = resolveFirstPageNumber(connection);

Review Comment:
   Fixed in ee02dbf509: the first page number is only read for `PAGE_NUMBER` 
now. Test: `aFirstPageNumberLeftOverFromPageNumberPagingIsIgnoredByOtherTypes`.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {
+      return !Utils.isEmpty(NVL(ex.body, "").trim());
+    }
+    List<String> items = splitResultItems(ex.body, countingExpression);

Review Comment:
   Fixed in ee02dbf509: an empty element (no attributes, child elements or 
text) no longer counts as an item, so `/envelope/list` stops on the empty 
`<list/>`. It still counts one item per page, which never stops the total count 
too early, only not early at all. The tooltip and docs say to select the items 
(`/envelope/list/item`) for the total count to work. Tests: 
`anItemsXPathToTheContainerStopsOnTheEmptyPage`, 
`anItemsXPathToTheContainerNeverStopsOnTheTotalCountTooEarly`.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {

Review Comment:
   The total count is now only used together with an items path (a split path 
can give several rows per item, so counting its rows could stop too early). 
Without one, the transform logs once that the total count path is not used. 
Test: `aTotalCountWithoutAnItemsPathIsLogged`.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {
+      return !Utils.isEmpty(NVL(ex.body, "").trim());
+    }
+    List<String> items = splitResultItems(ex.body, countingExpression);
+    if (items.isEmpty()) {
+      return false;
+    }
+    state.itemsRead += items.size();
+
+    Long totalCount = readTotalCount(conn, ex.body);
+    if (totalCount != null && state.itemsRead >= totalCount) {
+      if (isDetailed()) {
+        logDetailed(
+            BaseMessages.getString(
+                PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead, 
totalCount));
+      }
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * Picks the JsonPath or the XPath of a pair of paging expressions, the way 
the cursor and
+   * next-URL expressions are picked: the JsonPath when the response is read 
as JSON, otherwise the
+   * XPath for an XML response. A JsonPath with no XPath next to it is always 
used, so a mistake in
+   * it fails loudly rather than being ignored. Returns {@code null} when 
neither applies.
+   */
+  private String pagingExpression(String jsonPath, String xPath) {

Review Comment:
   Logged once now, naming the XPath and the application type, for the items 
and the total count XPath. Test: `anXPathWithANonXmlApplicationTypeIsLogged`.



##########
plugins/transforms/rest/src/main/java/org/apache/hop/pipeline/transforms/rest/Rest.java:
##########
@@ -1173,17 +1210,111 @@ private boolean continuePaginationAfterExchange(
   }
 
   /**
-   * @return whether this response appears to carry at least one item (whole 
body when no
-   *     split-path, or extracted elements when splitting).
+   * Whether page, offset and cursor paging should ask for another page after 
this one.
+   *
+   * <p>A page is empty when the connection's items path finds nothing in it. 
Without an items path
+   * the result split path decides, and without either an empty body does — 
which an API that wraps
+   * its results in an envelope never sends (#9549). The items path wins over 
the split path because
+   * it describes the API, while the split path describes the rows this 
transform wants.
+   *
+   * <p>With a total count path as well, paging also stops once that many 
items were read, so the
+   * last, empty, page is not fetched at all.
    */
-  private boolean exchangeHadPayload(RestExchangeResult ex, String 
resolvedSplitPath)
+  private boolean pageHadItemsAndMoreRemain(
+      RestConnection conn, PaginationState state, RestExchangeResult ex, 
String resolvedSplitPath)
       throws HopException {
-    List<String> items =
-        Utils.isEmpty(resolvedSplitPath) ? null : splitResultItems(ex.body, 
resolvedSplitPath);
-    if (items != null) {
-      return !items.isEmpty();
+    String itemsExpression = pagingExpression(conn.getItemsJsonPath(), 
conn.getItemsXPath());
+    String countingExpression =
+        Utils.isEmpty(itemsExpression) ? resolvedSplitPath : itemsExpression;
+    if (Utils.isEmpty(countingExpression)) {
+      return !Utils.isEmpty(NVL(ex.body, "").trim());
+    }
+    List<String> items = splitResultItems(ex.body, countingExpression);
+    if (items.isEmpty()) {
+      return false;
+    }
+    state.itemsRead += items.size();
+
+    Long totalCount = readTotalCount(conn, ex.body);
+    if (totalCount != null && state.itemsRead >= totalCount) {
+      if (isDetailed()) {
+        logDetailed(
+            BaseMessages.getString(
+                PKG, "Rest.Log.PaginationTotalCountReached", state.itemsRead, 
totalCount));
+      }
+      return false;
+    }
+    return true;
+  }
+
+  /**
+   * Picks the JsonPath or the XPath of a pair of paging expressions, the way 
the cursor and
+   * next-URL expressions are picked: the JsonPath when the response is read 
as JSON, otherwise the
+   * XPath for an XML response. A JsonPath with no XPath next to it is always 
used, so a mistake in
+   * it fails loudly rather than being ignored. Returns {@code null} when 
neither applies.
+   */
+  private String pagingExpression(String jsonPath, String xPath) {
+    String appType = NVL(resolve(meta.getApplicationType()), "");
+    String json = resolveSplitPathOrPagingExpression(jsonPath);
+    String xml = resolveSplitPathOrPagingExpression(xPath);
+    if (!Utils.isEmpty(json) && (shouldParseResponseAsJson(appType, json) || 
Utils.isEmpty(xml))) {
+      return json;
+    }
+    if (!Utils.isEmpty(xml) && isXmlApplicationType(appType)) {
+      return xml;
+    }
+    return null;
+  }
+
+  private static boolean isXmlApplicationType(String appType) {
+    return RestMeta.APPLICATION_TYPE_XML.equals(appType)
+        || RestMeta.APPLICATION_TYPE_TEXT_XML.equals(appType)
+        || RestMeta.APPLICATION_TYPE_ATOM_XML.equals(appType)
+        || RestMeta.APPLICATION_TYPE_SVG_XML.equals(appType)
+        || RestMeta.APPLICATION_TYPE_XHTML.equals(appType);
+  }
+
+  /**
+   * The total number of items the API reports through the connection's total 
count path, or {@code
+   * null} when no path is set or the page does not hold a whole number there.
+   */
+  private Long readTotalCount(RestConnection conn, String body) throws 
HopException {
+    String expression = pagingExpression(conn.getTotalCountJsonPath(), 
conn.getTotalCountXPath());
+    if (Utils.isEmpty(expression)) {
+      return null;
+    }
+    String value;
+    try {
+      // The same JSON-or-XML rule as splitResultItems(), so the count and the 
items agree.
+      if (shouldParseResponseAsJson(NVL(resolve(meta.getApplicationType()), 
""), expression)) {
+        Object raw =
+            JsonPath.using(JSON_PATH_CONFIGURATION).parse(NVL(body, 
"{}")).read(expression);
+        if (raw instanceof Number number) {
+          return number.longValue();
+        }
+        value = raw == null ? null : raw.toString();
+      } else {
+        Document doc;
+        try (StringReader sr = new StringReader(NVL(body, ""))) {
+          doc = documentBuilderFactory().newDocumentBuilder().parse(new 
InputSource(sr));

Review Comment:
   A total count that cannot be read is now logged, and paging goes on as 
without a count. Since the page is parsed once (see below), the items path has 
already read the same body by then, so in practice this covers a broken count 
expression. Test: `aTotalCountThatCannotBeReadIsLoggedAndPagingGoesOn`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to