Claus Ibsen created CAMEL-15955:
-----------------------------------
Summary: camel-core - API and SPI for Camel components to resume
from offset
Key: CAMEL-15955
URL: https://issues.apache.org/jira/browse/CAMEL-15955
Project: Camel
Issue Type: New Feature
Components: camel-core
Reporter: Claus Ibsen
Fix For: 3.8.0
We need a API and SPI for components to optionally support some way of offset
management. Meaning that a component is able to resume from a given offset to
continue from last known state. For example if a Camel application was
stopped/crashed, then it can restart and resume from last known offset.
Something like Apache Kafka offset has.
This benefits Camel in general, and Camel Kafka Connector also.
An use-case is the famous big files. So the file component is reading a file,
and process it line by line via a splitter. Then we would like to store an
offset how far it has processed. And in case of crash/restart etc then it can
continue from last known position. Today it would restart from top.
Same for SQL,JDBC components with SQL ResultSet and so on.
The offset management should support different quality levels and a SPI so
different implementations can plugin. One could be to store to a plain file and
for each step. And other can have a batch to store after N rows or N seconds.
Or like in Kafka with every N seconds.
Then each Camel component that can support this, can then implement the API and
logic for handling this. For the file component we can maybe store an offset in
the input stream, so when resuming we then forward the stream to that position.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
