I have a problem of efficiency in my project which uses Camel with the Esper component <http://camel.apache.org/esper.html> .
I have several external datasources feeding information to camel endpoints. Each Camel endpoint that receives data transfers it to a route that processes it and then delivers it at an Esper endpoint. The image below illustrates this behavior: <http://camel.465427.n5.nabble.com/file/n5747836/cPVWQ.png> The efficiency problem is that all of this is done by a single Java thread. Thus if I have many sources, there is a huge bottleneck. The following code accurately illustrates what is going on with the image: *To fix this problem, I have to handle this situation with a pool of threads or using some sort of parallel processing.* I cannot use patterns like multicast, recipient list, etc because all of those send the same message to multiple endpoints / clients, which is not the case in my examples. A possible solution would be having 1 thread per each "Datasource endpoint -> Route -> Esper endpoint" combination, like the image bellow: <http://camel.465427.n5.nabble.com/file/n5747836/hHg6A.png> Another possible solution is to have 1 thread receive everything from the datasources, and then dispatch it to multiple threads handling the route processing together with the other endpoint: <http://camel.465427.n5.nabble.com/file/n5747836/L2FgW.png> *PS:* I am open to any other possible suggestions you may have. To achieve one of these I have considered using the Camel SEDA component <http://camel.apache.org/seda.html> component, however, this one does not seem to allow me to have dynamic thread pools, because the `concurrentConsumers` property is static. Furthermore, I was alerted in StackOverflow for loss of messages using SEDA, so I am not sure how trustworthy this component really is. Can someone confirm? So, at this point I am quite lost and I don't know what to do: - Is SEDA the solution I am looking for? - If yes, how do I integrate it with the Esper endpoint given the syntax problem? - Are there any other solutions / Camel components that could fix my problem? -- View this message in context: http://camel.465427.n5.nabble.com/How-to-improve-efficiency-with-Camel-tp5747836.html Sent from the Camel - Users mailing list archive at Nabble.com.
