Hi,
We have developed Apache Came, SpringBoot based application to read data
from Oracle table and do some transformation of record and publish to Kafka
topic. We are using Camel SQL component for DB integration and implemented
Split & parallel processing pattern to parallelize processing to achieve
high throughput. Oracle tables are partitioned, so we are creating multiple
routes, one route per table partition, to speed up the processing. We have
30 partitions in table and so created 30 routes.
from(buildSelectSqlEndpoint(routeId))
.process(new GenericEventProcessor("MessagesReceived"))
.split(body())
.parallelProcessing()
.process(new
GenericEventProcessor("SplitAndParallelProcessing"))
.process(new InventoryProcessor())
.process(new GenericEventProcessor("ConvertedToAvroFormat"))
.to(buildKafkaProducerEndPoint(routeId));
I tested the application in local laptop with an adequate load and it is
processing as expected. But when we deployed the application in PCF, we see
some of the threads are failing. I have enabled the Camel debug log and i
see below debug log line -
Pipeline - Message exchange has failed: so breaking out of pipeline for
exchange Due to these thousands of messages are not published to Kafka.
>From initial analysis, i figured out Camel is creating one thread for each
route and based on *maxMessagePerPoll* configuration, it creates number of
threads equal to *maxMessagePerPoll*. With Split & parallel processing
enabled, Camel creates additional threads equal to *maxMessagePerPoll* for
Kafka publish. with this approach, we will have hundreds of threads created
and not sure any issue with PCF. Also I tried number of routes with 2, to
check the message delivery failures, and still see hundreds of failures and
with only 2 routes, increases total processing time for millions of records.
Could you please let me know, how to use Apache Camel with containers like
PCF? any additional configurations we need to have in PCF or Camel ?
Thanks & Regards
Praveen KG
