Similar to Quinn's response, but instead of splunk, we stood up a statsd + graphite server (pretty quick via docker). We're using Spring Boot, so it was natural to use its metric support with a statsd client to push metrics to the server over UDP -- very, very little impact to app performance.
Note that we made metrics collection part of the design of our system from the start as we were integrating across many SaaS and on-prem commercial apps via web services and needed to identify bottlenecks caused by each service call. We collect metrics mostly from our own code that's called from Camel routes (i.e beans and processors) vs. using Camel's metric component[1]. I never quite figured out how to get the two to work together (i.e. metric component pushing to statsd) but I didn't spend too much time trying :/ Bottom line: your messaging server wasn't designed to keep track of metrics beyond it's own operations. There are many systems better suited to the task, though all come with some additional overhead (setup, maintenance, client code, etc.) Have a look at the metric component[1] as it might provide all that you need for now. [1] http://camel.apache.org/metrics-component.html On Thu, Jan 25, 2018 at 2:28 PM, Quinn Stevenson < qu...@pronoia-solutions.com> wrote: > We do this by posting an auditing message to Splunk over HTTP. By doing > that, we can not only get counts over a particular timeframe, but we can > derive message rates as well. > > > > On Jan 25, 2018, at 2:56 PM, Olivier Eymere <oeym...@gmail.com> wrote: > > > > I am looking for a way to keep running counts of all of the processed > > messages. The last part of the camel application is a route that posts > the > > message to a web service, we need to keep counts of the various responses > > so that when the entire process is complete and the application is > exiting > > it will log the number of successful submissions and the numbers of > > unsuccessful submissions by the error type. So for example we have > > something like this: > > > > Success: 12345678 > > Error A: 2345 > > Error B: 3456 > > ... > > > > One idea was to send an empty message to a counter topic based on the > > respose. e.g. COUNTER.SUCCESS, COUNTER.ERRA, COUNTER.ERRB, etc. Then on > > exit I can use the enqueue count as the counter. I don't know if that's > > the best idea but it worked with one problem. The problem is that if the > > application is killed or has some sort of serious problems such as a full > > disk or we run out of memory, all of which have happened while testing, > the > > counts are completely lost. > > > > I could do the same thing with queues instead of topics and use the queue > > size for the count but I would have to persist the messages. It would > work > > but with additional overhead and I/O for persisting essentially > nothing. I > > could just log the response results and count later but that does not > seem > > much better than using persistent queues. > > > > Is there some way that people are using to keep a running count of > various > > message processing results that can reliably survive an application > restart? > >
