This is an automated email from the ASF dual-hosted git repository.
orpiske pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/camel-website.git
The following commit(s) were added to refs/heads/main by this push:
new 6b139f6 Added blog post about the Resume API v2
6b139f6 is described below
commit 6b139f61185de2f8476bb9aa01e359219caad088
Author: Otavio Rodolfo Piske <[email protected]>
AuthorDate: Mon Mar 21 13:32:44 2022 +0100
Added blog post about the Resume API v2
---
content/blog/2022/03/resume-api-v2/index.md | 42 +++++++++++++++++++++++++++++
1 file changed, 42 insertions(+)
diff --git a/content/blog/2022/03/resume-api-v2/index.md
b/content/blog/2022/03/resume-api-v2/index.md
new file mode 100644
index 0000000..b6cabb8
--- /dev/null
+++ b/content/blog/2022/03/resume-api-v2/index.md
@@ -0,0 +1,42 @@
+---
+title: "Introducing the Resume API v2"
+date: 2022-03-22
+draft: false
+authors: [orpiske]
+categories: ["Features", "Camel"]
+preview: "Introducing the Resume API v2"
+---
+
+The need to process larger quantities of data has been a growing concern and
necessity within our community. For the last few releases we have been working
to create mechanisms to simplify how our users can consume data at scale.
+
+Although Camel is no stranger to consuming data at scale, recent trends in
computing and systems architecture introduce the need for behaviors that
improve agility, speed and scalability when doing so. That's why, starting with
Camel 3.11, we included in the Kafka component the ability to restart data
consumption at specific offsets in the topics. Subsequent releases of Camel
expanded this feature to other components and opened up the API to give the
users more flexibility.
+
+With Camel 3.16.0 we are leveraging the work we did in the previous releases
to create a more elegant and flexible API. We are calling it the [Resume API
V2](/components/next/eips/resume-strategies.html).
+
+
+With this API the users can instruct Camel to start consuming data from the
last offset that was processed. For instance, when reading a very large file
Camel can skip reading the file offsets that have been previously consumed. It
can start processing at the last processed offset.
+
+The Resume API should not be confused with the [Idempotent
Consumer](/components/next/eips/idempotentConsumer-eip.html). Although there is
a small overlap between them (in the sense that they avoid processing twice)
they operate at different levels. The Resume API works at the component level,
ensuring that the component-specific lower level input skip the processed data,
whereas the Idempotent Consumer reads it all and filters the duplicates.
Additionally, the Resume API offers a greate [...]
+
+Here's an example of that API in action, taken from the [Resume API File
Offset
Example](https://github.com/apache/camel-examples/tree/main/examples/resume-api/resume-api-file-offset):
+
+[](https://asciinema.org/a/477253)
+
+This example shows a sample Camel integration reading a large file and
publishing the offsets to a Kafka instance that serves as the storaged for the
offsets. It displays it consuming batches of 30 lines, then stopping and then
resuming from where it stopped.
+
+A similar example to the one above, shows the Resume API v2 processing large
directories [Resume API File Set
Example](https://github.com/apache/camel-examples/tree/main/examples/resume-api/resume-api-fileset):
+
+[](https://asciinema.org/a/477253)
+
+The Resume API v2 also works in clustered mode with the master component. This
means that integrations using clustering with this component, can have their
secondary nodes pick up the processing from where the primary left off. The
[Resume API File Set Clusterized
Example](https://github.com/apache/camel-examples/tree/main/examples/resume-api/resume-api-fileset-clusterized)
demonstrates how this:
+
+[](https://asciinema.org/a/479064)
+
+In this release we are starting with a limited set of supported components:
+
+* camel-aws2-kinesis
+* camel-couchdb
+* camel-file
+* camel-kafka
+
+In the subsequent releases we plan to implement support for other components
and continue to mature and stabilize the API. If you find an issue with this
feature, don't hesitate to [report an issue](https://issues.apache.org/jira) or
provide a [contribution](https://github.com/apache/camel) to Apache Camel.
\ No newline at end of file