Claus Ibsen created CAMEL-27195:
-----------------------------------

             Summary: camel-mongodb - change streams consumer: resume after 
restart without a bean
                 Key: CAMEL-27195
                 URL: https://issues.apache.org/jira/browse/CAMEL-27195
             Project: Camel
          Issue Type: Improvement
          Components: camel-mongodb
            Reporter: Claus Ibsen


h3. Background

A {{changeStreams}} consumer resumes after a restart only when 
{{changeStreamTokenRepository}} is set. It takes a {{StateRepository<String, 
String>}} object, such as a {{FileStateRepository}}. After each exchange, the 
resume token is stored under {{<routeId>/<collection>}}, and the next start 
resumes after it. Without a repository, the changes made while the route was 
stopped are never consumed.

The other option, {{changeStreamToken}}, is a fixed token: it is used on every 
start and wins over the repository, so each restart consumes the same changes 
again. It is not a way to keep the place.

h3. The problem

The repository can only be given as a bean. That works in a Java or YAML route, 
e.g.:

{code}
mongodb:mongo?...&consumerType=changeStreams&changeStreamTokenRepository=#class:org.apache.camel.support.processor.state.FileStateRepository#fileStateRepository('/var/data/tokens.dat')
{code}

but it cannot be offered as a plain, optional option where only endpoint 
parameters can be set. The Kamelet {{mongodb-changes-stream-source}} cannot 
offer resume after restart because of this: an optional placeholder drops a 
parameter only when the value starts with the placeholder, so the 
{{#class:...('{{?file}}')}} form is never dropped, and a Kamelet that always 
creates the bean forces every user to have a file.

h3. Suggested improvement

Add a simple built-in way to keep the tokens, set from the endpoint URI, for 
example one of:
* {{changeStreamTokenFile=<path>}}: the consumer creates and owns a 
{{FileStateRepository}} on that file (like {{offsetStorageFileName}} in 
camel-debezium);
* {{persistentChangeStreamToken=true}} with an optional {{changeStreamTokenDb}} 
/ {{changeStreamTokenCollection}}: store the token in a MongoDB collection, 
like {{persistentTailTracking}} does for the tailable consumer.

{{changeStreamTokenRepository}}, when set, would still win.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to