This is an automated email from the ASF dual-hosted git repository.
davsclaus pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel.git
The following commit(s) were added to refs/heads/main by this push:
new 2fefcb707405 CAMEL-25806: camel-jira - the newIssues consumer no
longer loses the issues beyond maxResults of a poll (#28198)
2fefcb707405 is described below
commit 2fefcb70740504989ff0b6d9ae626b9d58e1a0de
Author: allthingssecurity <[email protected]>
AuthorDate: Thu Oct 22 00:59:13 2026 +0530
CAMEL-25806: camel-jira - the newIssues consumer no longer loses the issues
beyond maxResults of a poll (#28198)
* CAMEL-25806: camel-jira - the newIssues consumer no longer loses the
issues beyond maxResults of a poll
A poll of the newIssues consumer searched for the issues created after the
latest issue it had routed, newest first, and kept only the first maxResults
(default 50). The latest issue then became the newest one, so the older
issues of that poll were never returned again: with more than maxResults
issues created between two polls (a burst, or the first poll after Jira was
unreachable for a while) the older ones were silently lost.
The poll now reads every page of that search; maxResults is its page size.
* CAMEL-25806: camel-jira - address review: page with the next page token
on Jira Cloud
jira-rest-java-client 6.0.4 gives an AsynchronousCloudSearchRestClient for
Jira Cloud, whose searchJql throws UnsupportedOperationException for a
startAt
other than 0. The newIssues consumer, which now reads every page, therefore
failed at every poll with more than 50 new issues on Jira Cloud, and the
other
consumers failed the same way with maxResults above 50. On Jira Cloud the
pages
are now read with enhancedSearchJql and the next page token of the previous
page, until there is none; Jira Server / Data Center still pages by startAt.
The component doc and the upgrade guide now say that there is no upper bound
on the pages a poll reads.
* CAMEL-25806: camel-jira - address review: wrap the maxResults DSL javadoc
as the generator does
The endpoint DSL javadoc of the maxResults description was wrapped by hand
one column too wide; CI regenerates it with lines of at most 80 columns.
* CAMEL-25806: camel-jira - address review: say in the javadoc that
getIssues serves every Jira consumer
The Cloud paging by next page token in AbstractJiraConsumer.getIssues
applies to newIssues, newComments and watchUpdates alike.
* CAMEL-25806: camel-jira - address review: issues that leave the result
set while paging, Cloud note, at most 50
On Jira Server / Data Center the pages are read by start index. When
issues leave the result set while paging (deleted, moved, or edited out
of the JQL), the later ones move back and the first issue of the next
page was never read. When the total of a page is lower than the one of
the previous page, the search is read again from that many issues
before; the set drops the repeats.
The component doc now says that the search of Jira Cloud is eventually
consistent, so an issue that becomes searchable after a newer one was
consumed is not consumed, and the maxResults option says that the page
size is at most 50.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
---
.../org/apache/camel/catalog/components/jira.json | 2 +-
.../apache/camel/catalog/docs/jira-component.adoc | 8 +
.../org/apache/camel/component/jira/jira.json | 2 +-
.../camel-jira/src/main/docs/jira-component.adoc | 8 +
.../apache/camel/component/jira/JiraEndpoint.java | 3 +-
.../jira/consumer/AbstractJiraConsumer.java | 48 +++++-
.../component/jira/consumer/NewIssuesConsumer.java | 13 +-
...wIssuesConsumerMoreThanMaxResultsCloudTest.java | 65 ++++++++
.../NewIssuesConsumerMoreThanMaxResultsTest.java | 170 +++++++++++++++++++++
.../NewIssuesConsumerShrinkingResultTest.java | 65 ++++++++
.../ROOT/pages/camel-4x-upgrade-guide-4_24.adoc | 11 ++
.../endpoint/dsl/JiraEndpointBuilderFactory.java | 8 +-
12 files changed, 385 insertions(+), 18 deletions(-)
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/jira.json
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/jira.json
index 2b0b518a69fe..af18afb6b83b 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/jira.json
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/components/jira.json
@@ -65,7 +65,7 @@
"delay": { "index": 1, "kind": "parameter", "displayName": "Delay",
"group": "common", "label": "", "required": false, "type": "integer",
"javaType": "java.lang.Integer", "deprecated": false, "autowired": false,
"secret": false, "defaultValue": 6000, "configurationClass":
"org.apache.camel.component.jira.JiraConfiguration", "configurationField":
"configuration", "description": "Time in milliseconds to elapse for the next
poll." },
"jiraUrl": { "index": 2, "kind": "parameter", "displayName": "Jira Url",
"group": "common", "label": "", "required": true, "type": "string", "javaType":
"java.lang.String", "deprecated": false, "deprecationNote": "", "autowired":
false, "secret": false, "configurationClass":
"org.apache.camel.component.jira.JiraConfiguration", "configurationField":
"configuration", "description": "The Jira server url, for example
http:\/\/my_jira.com:8081." },
"jql": { "index": 3, "kind": "parameter", "displayName": "Jql", "group":
"consumer", "label": "consumer", "required": false, "type": "string",
"javaType": "java.lang.String", "deprecated": false, "autowired": false,
"secret": false, "description": "JQL is the query language from JIRA which
allows you to retrieve the data you want. For example jql=project=MyProject
Where MyProject is the product key in Jira. It is important to use the RAW()
and set the JQL inside it to prevent camel p [...]
- "maxResults": { "index": 4, "kind": "parameter", "displayName": "Max
Results", "group": "consumer", "label": "consumer", "required": false, "type":
"integer", "javaType": "java.lang.Integer", "deprecated": false, "autowired":
false, "secret": false, "defaultValue": 50, "description": "Max number of
issues to search for" },
+ "maxResults": { "index": 4, "kind": "parameter", "displayName": "Max
Results", "group": "consumer", "label": "consumer", "required": false, "type":
"integer", "javaType": "java.lang.Integer", "deprecated": false, "autowired":
false, "secret": false, "defaultValue": 50, "description": "Max number of
issues to search for. The newIssues consumer reads every issue created since
its previous poll, page by page, with this as the page size (at most 50)." },
"sendEmptyMessageWhenIdle": { "index": 5, "kind": "parameter",
"displayName": "Send Empty Message When Idle", "group": "consumer", "label":
"consumer", "required": false, "type": "boolean", "javaType": "boolean",
"deprecated": false, "autowired": false, "secret": false, "defaultValue":
false, "description": "If the polling consumer did not poll any files, you can
enable this option to send an empty message (no body) instead." },
"sendOnlyUpdatedField": { "index": 6, "kind": "parameter", "displayName":
"Send Only Updated Field", "group": "consumer", "label": "consumer",
"required": false, "type": "boolean", "javaType": "boolean", "deprecated":
false, "autowired": false, "secret": false, "defaultValue": true,
"description": "Indicator for sending only changed fields in exchange body or
issue object. By default consumer sends only changed fields." },
"watchedFields": { "index": 7, "kind": "parameter", "displayName":
"Watched Fields", "group": "consumer", "label": "consumer", "required": false,
"type": "string", "javaType": "java.lang.String", "deprecated": false,
"autowired": false, "secret": false, "defaultValue": "Status,Priority",
"description": "Comma separated list of fields to watch for changes.
Status,Priority are the defaults." },
diff --git
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/jira-component.adoc
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/jira-component.adoc
index 62980d533155..861c4d0f3d21 100644
---
a/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/jira-component.adoc
+++
b/catalog/camel-catalog/src/generated/resources/org/apache/camel/catalog/docs/jira-component.adoc
@@ -141,6 +141,14 @@ automatically set the JQL as:
This is in order to optimize startup processing, rather than having to index
every single
issue in the project.
+Each poll reads every page of that search, so the issues created since the
previous poll are consumed also when
+there are more than `maxResults` of them (for example after Jira was
unreachable for a while); `maxResults` is then
+the page size of the search (at most 50). There is no upper bound: after a
long outage or a large import, one poll
+reads all the new issues, one search request per page, keeps them in memory
and routes them as one batch. On Jira
+Cloud the pages are read with the next page token of the search, as Jira Cloud
cannot page by start index. The search
+of Jira Cloud is eventually consistent, so a new issue can become searchable
only after a newer one: when a poll has
+already consumed the newer one, the late issue is not consumed.
+
Another note is that, similarly, the newComments consumer will have to
index every single issue *and* comment on the project. Therefore, for
large projects, it's *vital* to optimize the JQL expression as much as
diff --git
a/components/camel-jira/src/generated/resources/META-INF/org/apache/camel/component/jira/jira.json
b/components/camel-jira/src/generated/resources/META-INF/org/apache/camel/component/jira/jira.json
index 2b0b518a69fe..af18afb6b83b 100644
---
a/components/camel-jira/src/generated/resources/META-INF/org/apache/camel/component/jira/jira.json
+++
b/components/camel-jira/src/generated/resources/META-INF/org/apache/camel/component/jira/jira.json
@@ -65,7 +65,7 @@
"delay": { "index": 1, "kind": "parameter", "displayName": "Delay",
"group": "common", "label": "", "required": false, "type": "integer",
"javaType": "java.lang.Integer", "deprecated": false, "autowired": false,
"secret": false, "defaultValue": 6000, "configurationClass":
"org.apache.camel.component.jira.JiraConfiguration", "configurationField":
"configuration", "description": "Time in milliseconds to elapse for the next
poll." },
"jiraUrl": { "index": 2, "kind": "parameter", "displayName": "Jira Url",
"group": "common", "label": "", "required": true, "type": "string", "javaType":
"java.lang.String", "deprecated": false, "deprecationNote": "", "autowired":
false, "secret": false, "configurationClass":
"org.apache.camel.component.jira.JiraConfiguration", "configurationField":
"configuration", "description": "The Jira server url, for example
http:\/\/my_jira.com:8081." },
"jql": { "index": 3, "kind": "parameter", "displayName": "Jql", "group":
"consumer", "label": "consumer", "required": false, "type": "string",
"javaType": "java.lang.String", "deprecated": false, "autowired": false,
"secret": false, "description": "JQL is the query language from JIRA which
allows you to retrieve the data you want. For example jql=project=MyProject
Where MyProject is the product key in Jira. It is important to use the RAW()
and set the JQL inside it to prevent camel p [...]
- "maxResults": { "index": 4, "kind": "parameter", "displayName": "Max
Results", "group": "consumer", "label": "consumer", "required": false, "type":
"integer", "javaType": "java.lang.Integer", "deprecated": false, "autowired":
false, "secret": false, "defaultValue": 50, "description": "Max number of
issues to search for" },
+ "maxResults": { "index": 4, "kind": "parameter", "displayName": "Max
Results", "group": "consumer", "label": "consumer", "required": false, "type":
"integer", "javaType": "java.lang.Integer", "deprecated": false, "autowired":
false, "secret": false, "defaultValue": 50, "description": "Max number of
issues to search for. The newIssues consumer reads every issue created since
its previous poll, page by page, with this as the page size (at most 50)." },
"sendEmptyMessageWhenIdle": { "index": 5, "kind": "parameter",
"displayName": "Send Empty Message When Idle", "group": "consumer", "label":
"consumer", "required": false, "type": "boolean", "javaType": "boolean",
"deprecated": false, "autowired": false, "secret": false, "defaultValue":
false, "description": "If the polling consumer did not poll any files, you can
enable this option to send an empty message (no body) instead." },
"sendOnlyUpdatedField": { "index": 6, "kind": "parameter", "displayName":
"Send Only Updated Field", "group": "consumer", "label": "consumer",
"required": false, "type": "boolean", "javaType": "boolean", "deprecated":
false, "autowired": false, "secret": false, "defaultValue": true,
"description": "Indicator for sending only changed fields in exchange body or
issue object. By default consumer sends only changed fields." },
"watchedFields": { "index": 7, "kind": "parameter", "displayName":
"Watched Fields", "group": "consumer", "label": "consumer", "required": false,
"type": "string", "javaType": "java.lang.String", "deprecated": false,
"autowired": false, "secret": false, "defaultValue": "Status,Priority",
"description": "Comma separated list of fields to watch for changes.
Status,Priority are the defaults." },
diff --git a/components/camel-jira/src/main/docs/jira-component.adoc
b/components/camel-jira/src/main/docs/jira-component.adoc
index 62980d533155..861c4d0f3d21 100644
--- a/components/camel-jira/src/main/docs/jira-component.adoc
+++ b/components/camel-jira/src/main/docs/jira-component.adoc
@@ -141,6 +141,14 @@ automatically set the JQL as:
This is in order to optimize startup processing, rather than having to index
every single
issue in the project.
+Each poll reads every page of that search, so the issues created since the
previous poll are consumed also when
+there are more than `maxResults` of them (for example after Jira was
unreachable for a while); `maxResults` is then
+the page size of the search (at most 50). There is no upper bound: after a
long outage or a large import, one poll
+reads all the new issues, one search request per page, keeps them in memory
and routes them as one batch. On Jira
+Cloud the pages are read with the next page token of the search, as Jira Cloud
cannot page by start index. The search
+of Jira Cloud is eventually consistent, so a new issue can become searchable
only after a newer one: when a poll has
+already consumed the newer one, the late issue is not consumed.
+
Another note is that, similarly, the newComments consumer will have to
index every single issue *and* comment on the project. Therefore, for
large projects, it's *vital* to optimize the JQL expression as much as
diff --git
a/components/camel-jira/src/main/java/org/apache/camel/component/jira/JiraEndpoint.java
b/components/camel-jira/src/main/java/org/apache/camel/component/jira/JiraEndpoint.java
index aeb9797330d7..64ab0d802a75 100644
---
a/components/camel-jira/src/main/java/org/apache/camel/component/jira/JiraEndpoint.java
+++
b/components/camel-jira/src/main/java/org/apache/camel/component/jira/JiraEndpoint.java
@@ -267,7 +267,8 @@ public class JiraEndpoint extends ScheduledPollEndpoint
implements EndpointServi
}
/**
- * Max number of issues to search for
+ * Max number of issues to search for. The newIssues consumer reads every
issue created since its previous poll,
+ * page by page, with this as the page size (at most 50).
*/
public void setMaxResults(Integer maxResults) {
this.maxResults = maxResults;
diff --git
a/components/camel-jira/src/main/java/org/apache/camel/component/jira/consumer/AbstractJiraConsumer.java
b/components/camel-jira/src/main/java/org/apache/camel/component/jira/consumer/AbstractJiraConsumer.java
index a249a02c9203..017d4f67fe78 100644
---
a/components/camel-jira/src/main/java/org/apache/camel/component/jira/consumer/AbstractJiraConsumer.java
+++
b/components/camel-jira/src/main/java/org/apache/camel/component/jira/consumer/AbstractJiraConsumer.java
@@ -114,6 +114,10 @@ public abstract class AbstractJiraConsumer extends
ScheduledBatchPollingConsumer
return getIssues(jql, SEARCH_START_AT, SEARCH_MAX_PER_QUERY,
maxResults);
}
+ /**
+ * Runs the search of every Jira consumer (newIssues, newComments,
watchUpdates): on Jira Cloud the pages are read
+ * with the next page token (see getCloudIssues), on Jira Server / Data
Center by start index.
+ */
// Ignore maxResults if it's <= 0.
protected Queue<Issue> getIssues(String jql, int start, int maxPerQuery,
int maxResults) {
LOG.debug("Start indexing current JIRA issues...");
@@ -122,8 +126,13 @@ public abstract class AbstractJiraConsumer extends
ScheduledBatchPollingConsumer
maxPerQuery = maxResults;
}
+ if (endpoint.getClient().getSearchClient() instanceof
AsynchronousCloudSearchRestClient cloudSearchRestClient) {
+ return getCloudIssues(cloudSearchRestClient, jql, maxPerQuery,
maxResults);
+ }
+
// Avoid duplicates
Set<Issue> issues = new LinkedHashSet<>();
+ int previousTotal = -1;
while (true) {
SearchRestClient searchRestClient =
endpoint.getClient().getSearchClient();
// *navigable should be the default value, but it does not seem to
be true with 6.0.2 client
@@ -136,15 +145,17 @@ public abstract class AbstractJiraConsumer extends
ScheduledBatchPollingConsumer
// Note: the total # the query would return *without* pagination,
effectively telling us
// we've reached the end. Also exit early if we're limiting the #
of results or
// if total # of returned issues is lower than the actual page
size.
- int total;
- if (searchRestClient instanceof AsynchronousCloudSearchRestClient)
{
- // calling searchResult.getTotal() on
AsynchronousCloudSearchRestClient throws an exception:
- // Total is not available in the Cloud version of the new
Search API response.
- // Please use `SearchRestClient.totalCount` instead to fetch
the estimated count of the issues for a given JQL
- total = searchRestClient.totalCount(jql).claim().getCount();
- } else {
- total = searchResult.getTotal();
+ int total = searchResult.getTotal();
+ if (previousTotal >= 0 && total < previousTotal && start <
previousTotal) {
+ // issues left the result set while paging (deleted, moved, or
edited out of the JQL), so the later
+ // ones moved back and the first ones of this page may have
been on the previous page: read again from
+ // that many issues before, the set drops the repeats. Not
needed when the previous page was the last
+ // one. An issue created at the same time as one leaves keeps
the total and is not seen here.
+ start = Math.max(0, start - (previousTotal - total));
+ previousTotal = total;
+ continue;
}
+ previousTotal = total;
if (maxPerQuery >= total || start >= total || maxResults > 0 &&
issues.size() >= maxResults) {
break;
}
@@ -155,6 +166,27 @@ public abstract class AbstractJiraConsumer extends
ScheduledBatchPollingConsumer
return new LinkedList<>(issues);
}
+ /**
+ * Jira Cloud cannot page by startAt: its search client throws
UnsupportedOperationException for a startAt other
+ * than 0. The pages are read with the next page token of the previous
page instead, from the first issue, until the
+ * last page (no next page token) or maxResults (ignored if it's <= 0).
+ */
+ private Queue<Issue> getCloudIssues(
+ AsynchronousCloudSearchRestClient searchRestClient, String jql,
int maxPerQuery, int maxResults) {
+ Set<Issue> issues = new LinkedHashSet<>();
+ String nextPageToken = null;
+ do {
+ SearchResult searchResult
+ = searchRestClient.enhancedSearchJql(jql, maxPerQuery,
nextPageToken, Set.of("*navigable"), null).claim();
+ for (Issue issue : searchResult.getIssues()) {
+ issues.add(issue);
+ }
+ nextPageToken = searchResult.getNextPageToken();
+ } while (nextPageToken != null && !(maxResults > 0 && issues.size() >=
maxResults));
+ LOG.debug("End indexing current JIRA issues. {} issues indexed.",
issues.size());
+ return new LinkedList<>(issues);
+ }
+
protected JiraRestClient client() {
return endpoint.getClient();
}
diff --git
a/components/camel-jira/src/main/java/org/apache/camel/component/jira/consumer/NewIssuesConsumer.java
b/components/camel-jira/src/main/java/org/apache/camel/component/jira/consumer/NewIssuesConsumer.java
index 87b4095df965..b091f1b43e1a 100644
---
a/components/camel-jira/src/main/java/org/apache/camel/component/jira/consumer/NewIssuesConsumer.java
+++
b/components/camel-jira/src/main/java/org/apache/camel/component/jira/consumer/NewIssuesConsumer.java
@@ -129,16 +129,19 @@ public class NewIssuesConsumer extends
AbstractJiraConsumer {
* @return queue of new issues
*/
private Queue<Issue> getNewIssues() {
- String jqlFilter;
+ Queue<Issue> issues;
// If we have processed an issue before, use its timestamp to get only
the newer ones
if (latestIssue != null) {
- jqlFilter = String.format(NEW_ISSUES_JQL_FORMAT,
getServerTimestamp(), getEndpoint().getJql());
+ String jqlFilter = String.format(NEW_ISSUES_JQL_FORMAT,
getServerTimestamp(), getEndpoint().getJql());
+ // all of them: the newest one becomes the latest issue, so a new
issue left out by maxResults would never
+ // be returned again; maxResults is the page size here
+ int maxResults = getEndpoint().getMaxResults();
+ int pageSize = maxResults > 0 ? Math.min(maxResults,
SEARCH_MAX_PER_QUERY) : SEARCH_MAX_PER_QUERY;
+ issues = getIssues(jqlFilter, SEARCH_START_AT, pageSize,
Integer.MAX_VALUE);
} else {
- jqlFilter = getEndpoint().getJql();
+ issues = getIssues(getEndpoint().getJql());
}
- Queue<Issue> issues = getIssues(jqlFilter);
-
if (!issues.isEmpty()) {
if (latestIssue != null) {
// remove all issues that are older than the latestIssue
(including), because those were processed by previous polls
diff --git
a/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerMoreThanMaxResultsCloudTest.java
b/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerMoreThanMaxResultsCloudTest.java
new file mode 100644
index 000000000000..642865615844
--- /dev/null
+++
b/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerMoreThanMaxResultsCloudTest.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.jira.consumer;
+
+import com.atlassian.jira.rest.client.api.SearchRestClient;
+import com.atlassian.jira.rest.client.api.domain.SearchResult;
+import com.atlassian.jira.rest.client.api.domain.TotalCount;
+import
com.atlassian.jira.rest.client.internal.async.AsynchronousCloudSearchRestClient;
+import io.atlassian.util.concurrent.Promise;
+import io.atlassian.util.concurrent.Promises;
+import org.mockito.Mock;
+
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.lenient;
+
+/**
+ * The same as {@link NewIssuesConsumerMoreThanMaxResultsTest} on Jira Cloud,
whose search client cannot page by
+ * startAt: like the real client of jira-rest-java-client 6.0.4, it throws
UnsupportedOperationException for a startAt
+ * other than 0, and pages with the next page token instead.
+ */
+public class NewIssuesConsumerMoreThanMaxResultsCloudTest extends
NewIssuesConsumerMoreThanMaxResultsTest {
+
+ @Mock
+ private AsynchronousCloudSearchRestClient cloudSearchRestClient;
+
+ @Override
+ protected SearchRestClient searchClient() {
+ // lenient: which of these the consumer calls is what the test is about
+ lenient().when(cloudSearchRestClient.searchJql(any(), any(), any(),
any())).then(invocation -> {
+ Integer startAt = invocation.getArgument(2);
+ if (startAt != null && startAt != 0) {
+ throw new UnsupportedOperationException(
+ "This search API is not supported anymore. Please use
this enhancedSearchJql");
+ }
+ return enhancedSearch(invocation.getArgument(0),
invocation.getArgument(1), null);
+ });
+ lenient().when(cloudSearchRestClient.enhancedSearchJql(any(), any(),
any(), any(), any()))
+ .then(invocation -> enhancedSearch(invocation.getArgument(0),
invocation.getArgument(1),
+ invocation.getArgument(2)));
+ lenient().when(cloudSearchRestClient.totalCount(any()))
+ .then(invocation -> Promises.promise(new
TotalCount(total(invocation.getArgument(0)))));
+ return cloudSearchRestClient;
+ }
+
+ private Promise<SearchResult> enhancedSearch(String jql, int maxPerQuery,
String nextPageToken) {
+ int startAt = nextPageToken == null ? 0 :
Integer.parseInt(nextPageToken);
+ int total = total(jql);
+ String next = startAt + maxPerQuery < total ? String.valueOf(startAt +
maxPerQuery) : null;
+ return Promises.promise(new SearchResult(page(jql, startAt,
maxPerQuery), next));
+ }
+}
diff --git
a/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerMoreThanMaxResultsTest.java
b/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerMoreThanMaxResultsTest.java
new file mode 100644
index 000000000000..4f51d6f65e05
--- /dev/null
+++
b/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerMoreThanMaxResultsTest.java
@@ -0,0 +1,170 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.jira.consumer;
+
+import java.net.URI;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.atlassian.jira.rest.client.api.JiraRestClient;
+import com.atlassian.jira.rest.client.api.JiraRestClientFactory;
+import com.atlassian.jira.rest.client.api.SearchRestClient;
+import com.atlassian.jira.rest.client.api.UserRestClient;
+import com.atlassian.jira.rest.client.api.domain.Issue;
+import com.atlassian.jira.rest.client.api.domain.SearchResult;
+import com.atlassian.jira.rest.client.api.domain.User;
+import io.atlassian.util.concurrent.Promises;
+import org.apache.camel.CamelContext;
+import org.apache.camel.EndpointInject;
+import org.apache.camel.builder.RouteBuilder;
+import org.apache.camel.component.jira.JiraComponent;
+import org.apache.camel.component.mock.MockEndpoint;
+import org.apache.camel.spi.Registry;
+import org.apache.camel.test.junit6.CamelTestSupport;
+import org.joda.time.DateTime;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+import static org.apache.camel.component.jira.JiraConstants.JIRA;
+import static
org.apache.camel.component.jira.JiraConstants.JIRA_REST_CLIENT_FACTORY;
+import static
org.apache.camel.component.jira.JiraTestConstants.JIRA_CREDENTIALS;
+import static org.apache.camel.component.jira.JiraTestConstants.PROJECT;
+import static
org.apache.camel.component.jira.Utils.createIssueWithCreationDate;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.when;
+
+/**
+ * More new issues than maxResults between two polls (a burst, or the polls
after Jira was unreachable for a while):
+ * every one of them must be consumed, not only the newest maxResults.
+ */
+@ExtendWith(MockitoExtension.class)
+public class NewIssuesConsumerMoreThanMaxResultsTest extends CamelTestSupport {
+
+ protected static final int NEW_ISSUES = 60;
+
+ private final DateTime start = DateTime.now().minusHours(2);
+ private final Issue latestAtStart = createIssueWithCreationDate(1000,
start);
+ // newest first, as the JQL of the consumer orders them
+ protected final List<Issue> newIssues = new ArrayList<>();
+ protected final AtomicBoolean served = new AtomicBoolean();
+
+ @Mock
+ private JiraRestClient jiraClient;
+
+ @Mock
+ private JiraRestClientFactory jiraRestClientFactory;
+
+ @Mock
+ private SearchRestClient searchRestClient;
+
+ @Mock
+ private UserRestClient userRestClient;
+
+ @EndpointInject("mock:result")
+ private MockEndpoint mockResult;
+
+ @Override
+ protected void bindToRegistry(Registry registry) {
+ registry.bind(JIRA_REST_CLIENT_FACTORY, jiraRestClientFactory);
+ }
+
+ @Override
+ protected CamelContext createCamelContext() throws Exception {
+ for (int i = NEW_ISSUES; i >= 1; i--) {
+ newIssues.add(createIssueWithCreationDate(1000 + i,
start.plusMinutes(i)));
+ }
+ User user = new User(
+ null, "admin", null, null, true, null,
+ Map.of("48x48", URI.create("")),
DateTime.now().getZone().getID());
+ SearchRestClient searchClient = searchClient();
+ when(jiraClient.getSearchClient()).thenReturn(searchClient);
+ when(jiraClient.getUserClient()).thenReturn(userRestClient);
+ when(jiraRestClientFactory.createWithBasicHttpAuthentication(any(),
any(), any())).thenReturn(jiraClient);
+
when(userRestClient.getUser(any(URI.class))).thenReturn(Promises.promise(user));
+
+ CamelContext camelContext = super.createCamelContext();
+ camelContext.disableJMX();
+ camelContext.addComponent(JIRA, new JiraComponent(camelContext));
+ return camelContext;
+ }
+
+ /**
+ * The search client of Jira Server / Data Center, which pages by startAt.
+ */
+ protected SearchRestClient searchClient() {
+ when(searchRestClient.searchJql(any(), any(), any(),
any())).then(invocation -> {
+ String jql = invocation.getArgument(0);
+ int maxPerQuery = invocation.getArgument(1);
+ int startAt = invocation.getArgument(2);
+ int total = total(jql);
+ return Promises.promise(new SearchResult(startAt, maxPerQuery,
total, page(jql, startAt, maxPerQuery)));
+ });
+ return searchRestClient;
+ }
+
+ /**
+ * The number of issues the given search finds now.
+ */
+ protected int total(String jql) {
+ if (!jql.startsWith("created >")) {
+ return 1;
+ }
+ return served.get() ? 0 : NEW_ISSUES;
+ }
+
+ /**
+ * The issues of the given page of the search: the latest issue at
startup; then the issues created after it, all of
+ * them at the first poll, which pages through them, and none at the next
polls, as they are all older than the
+ * newest one.
+ */
+ protected List<Issue> page(String jql, int startAt, int maxPerQuery) {
+ if (!jql.startsWith("created >")) {
+ return List.of(latestAtStart);
+ }
+ List<Issue> answer = new ArrayList<>();
+ int total = total(jql);
+ for (int i = startAt; i < Math.min(startAt + maxPerQuery, total); i++)
{
+ answer.add(newIssues.get(i));
+ }
+ if (startAt + maxPerQuery >= total) {
+ served.set(true);
+ }
+ return answer;
+ }
+
+ @Override
+ protected RouteBuilder createRouteBuilder() {
+ return new RouteBuilder() {
+ @Override
+ public void configure() {
+ from("jira://newIssues?jiraUrl=" + JIRA_CREDENTIALS +
"&jql=project=" + PROJECT + "&delay=500")
+ .to(mockResult);
+ }
+ };
+ }
+
+ @Test
+ public void everyNewIssueIsConsumed() throws Exception {
+ mockResult.expectedMessageCount(NEW_ISSUES);
+ mockResult.expectedBodiesReceivedInAnyOrder(newIssues.toArray());
+ mockResult.assertIsSatisfied();
+ }
+}
diff --git
a/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerShrinkingResultTest.java
b/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerShrinkingResultTest.java
new file mode 100644
index 000000000000..bff5b0eea44c
--- /dev/null
+++
b/components/camel-jira/src/test/java/org/apache/camel/component/jira/consumer/NewIssuesConsumerShrinkingResultTest.java
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.camel.component.jira.consumer;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import com.atlassian.jira.rest.client.api.domain.Issue;
+
+/**
+ * An issue leaves the result set between two pages of a poll on Jira Server /
Data Center (deleted, moved, or edited
+ * out of the JQL): the later issues move back by one, and the first issue of
the next page must still be consumed.
+ */
+public class NewIssuesConsumerShrinkingResultTest extends
NewIssuesConsumerMoreThanMaxResultsTest {
+
+ private final Set<Issue> returned = new HashSet<>();
+ private boolean removed;
+
+ @Override
+ protected int total(String jql) {
+ if (!jql.startsWith("created >")) {
+ return 1;
+ }
+ if (served.get()) {
+ return 0;
+ }
+ return removed ? NEW_ISSUES - 1 : NEW_ISSUES;
+ }
+
+ @Override
+ protected List<Issue> page(String jql, int startAt, int maxPerQuery) {
+ if (!jql.startsWith("created >")) {
+ return super.page(jql, startAt, maxPerQuery);
+ }
+ List<Issue> current = removed ? newIssues.subList(1, NEW_ISSUES) :
newIssues;
+ int total = served.get() ? 0 : current.size();
+ List<Issue> answer = new ArrayList<>();
+ for (int i = startAt; i < Math.min(startAt + maxPerQuery, total); i++)
{
+ answer.add(current.get(i));
+ }
+ // the newest issue, read on the first page, leaves the result set
before the next page is read
+ removed = true;
+ returned.addAll(answer);
+ if (returned.containsAll(current)) {
+ served.set(true);
+ }
+ return answer;
+ }
+}
diff --git
a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
index d90829087638..697208b5c676 100644
--- a/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
+++ b/docs/user-manual/modules/ROOT/pages/camel-4x-upgrade-guide-4_24.adoc
@@ -1074,6 +1074,17 @@ too early (`messageExpiry=3600` gave 3.6 ms instead of
one hour). The value is n
created before keep their expiry: change it on the Iggy server. If you set
`messageExpiry` in microseconds to work
around this, set it in seconds now, or newly created topics keep their
messages a million times longer than intended.
+=== camel-jira - the newIssues consumer reads every new issue
+
+A poll of the `newIssues` consumer read only the newest `maxResults` (default
50) of the issues created since the
+previous poll, and the older ones were never consumed. A poll now reads every
page of the search and can route more
+than `maxResults` issues; `maxResults` is the page size of that search (at
most 50). There is no upper bound, so the
+first poll after a long outage or a large import can send many search requests
and route many issues in one batch.
+
+On Jira Cloud the Jira consumers now read the pages of a search with the next
page token, as Jira Cloud cannot page by
+start index: with `maxResults` above 50, a search with more than 50 results
failed on the second page with
+`UnsupportedOperationException`.
+
=== camel-hashicorp-vault - properties without engine use the `secret` engine
A `hashicorp:` property without an engine, such as `{{hashicorp:mysecret}}` or
`{{hashicorp:mysecret#field}}`, now
diff --git
a/dsl/camel-endpointdsl/src/generated/java/org/apache/camel/builder/endpoint/dsl/JiraEndpointBuilderFactory.java
b/dsl/camel-endpointdsl/src/generated/java/org/apache/camel/builder/endpoint/dsl/JiraEndpointBuilderFactory.java
index 39b08cf8afce..f4c2c51057ec 100644
---
a/dsl/camel-endpointdsl/src/generated/java/org/apache/camel/builder/endpoint/dsl/JiraEndpointBuilderFactory.java
+++
b/dsl/camel-endpointdsl/src/generated/java/org/apache/camel/builder/endpoint/dsl/JiraEndpointBuilderFactory.java
@@ -108,7 +108,9 @@ public interface JiraEndpointBuilderFactory {
return this;
}
/**
- * Max number of issues to search for.
+ * Max number of issues to search for. The newIssues consumer reads
+ * every issue created since its previous poll, page by page, with this
+ * as the page size (at most 50).
*
* The option is a: <code>java.lang.Integer</code> type.
*
@@ -123,7 +125,9 @@ public interface JiraEndpointBuilderFactory {
return this;
}
/**
- * Max number of issues to search for.
+ * Max number of issues to search for. The newIssues consumer reads
+ * every issue created since its previous poll, page by page, with this
+ * as the page size (at most 50).
*
* The option will be converted to a <code>java.lang.Integer</code>
* type.
