Hello, We're using a camel Consumer with an aws-sqs endpoint and running into issues with large queues and slow consumers. The consumer jvm appears to download messages very fast from sqs, onto the jvm heap, even though the camel receive method processes each message very slowly (and Ack's when done, with autoAck = false). The result of this is that the consumer jvm continually fills its heap with an unbounded queue of incoming messages and eventually throws OOME.
To avoid this failure, is there a way to enable backpressure when using aws-sqs?—e.g. an on-heap bounded queue between the component fetching from sqs and the user-defined actor processing the messages. I didn't see anything relevant in the config: - http://camel.apache.org/aws-sqs.html And I didn't see any relevant jira issues: - https://issues.apache.org/jira/browse/CAMEL-5113?jql=project%20%3D%20CAMEL%20AND%20text%20~%20sqs Looking at the code, I see that SqsConsumer subtypes ScheduledBatchPollingConsumer, which subtypes ScheduledPollConsumer. To get backpressure, should it use something like EventDrivenPollingConsumer somewhere, which uses a blocking queue to avoid the heap blowup above? - https://github.com/apache/camel/blob/camel-2.14.1/components/camel-aws/src/main/java/org/apache/camel/component/aws/sqs/SqsConsumer.java#L52-L57 - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledBatchPollingConsumer.java#L31-L34 - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/ScheduledPollConsumer.java#L40-L43 - https://github.com/apache/camel/blob/camel-2.14.1/camel-core/src/main/java/org/apache/camel/impl/EventDrivenPollingConsumer.java#L35-L42 Thanks, Dan -- View this message in context: http://camel.465427.n5.nabble.com/Backpressure-with-aws-sqs-Consumer-tp5761348.html Sent from the Camel - Users mailing list archive at Nabble.com.
