Thanks for the responses. I agree that this isn't really something to expect from a message server. At first I was looking at the metrics component but it didn't quite match our requirements (e.g. the counter doesn't persist on restarts afaik). Unfortunately we have a number of constraints about how and where we are running and we are limited in our choices; splunk would have been great but isn't an option. I haven't worked with statsd before but I am looking at it now and it looks to be perfect. I am going to give that a try.
On Thu, Jan 25, 2018 at 3:43 PM, Doug Douglass <douglass.d...@gmail.com> wrote: > Similar to Quinn's response, but instead of splunk, we stood up a statsd + > graphite server (pretty quick via docker). We're using Spring Boot, so it > was natural to use its metric support with a statsd client to push metrics > to the server over UDP -- very, very little impact to app performance. > > Note that we made metrics collection part of the design of our system from > the start as we were integrating across many SaaS and on-prem commercial > apps via web services and needed to identify bottlenecks caused by each > service call. We collect metrics mostly from our own code that's called > from Camel routes (i.e beans and processors) vs. using Camel's metric > component[1]. I never quite figured out how to get the two to work together > (i.e. metric component pushing to statsd) but I didn't spend too much time > trying :/ > > Bottom line: your messaging server wasn't designed to keep track of metrics > beyond it's own operations. There are many systems better suited to the > task, though all come with some additional overhead (setup, maintenance, > client code, etc.) Have a look at the metric component[1] as it might > provide all that you need for now. > > [1] http://camel.apache.org/metrics-component.html > > On Thu, Jan 25, 2018 at 2:28 PM, Quinn Stevenson < > qu...@pronoia-solutions.com> wrote: > > > We do this by posting an auditing message to Splunk over HTTP. By doing > > that, we can not only get counts over a particular timeframe, but we can > > derive message rates as well. > > > > > > > On Jan 25, 2018, at 2:56 PM, Olivier Eymere <oeym...@gmail.com> wrote: > > > > > > I am looking for a way to keep running counts of all of the processed > > > messages. The last part of the camel application is a route that posts > > the > > > message to a web service, we need to keep counts of the various > responses > > > so that when the entire process is complete and the application is > > exiting > > > it will log the number of successful submissions and the numbers of > > > unsuccessful submissions by the error type. So for example we have > > > something like this: > > > > > > Success: 12345678 > > > Error A: 2345 > > > Error B: 3456 > > > ... > > > > > > One idea was to send an empty message to a counter topic based on the > > > respose. e.g. COUNTER.SUCCESS, COUNTER.ERRA, COUNTER.ERRB, etc. Then > on > > > exit I can use the enqueue count as the counter. I don't know if > that's > > > the best idea but it worked with one problem. The problem is that if > the > > > application is killed or has some sort of serious problems such as a > full > > > disk or we run out of memory, all of which have happened while testing, > > the > > > counts are completely lost. > > > > > > I could do the same thing with queues instead of topics and use the > queue > > > size for the count but I would have to persist the messages. It would > > work > > > but with additional overhead and I/O for persisting essentially > > nothing. I > > > could just log the response results and count later but that does not > > seem > > > much better than using persistent queues. > > > > > > Is there some way that people are using to keep a running count of > > various > > > message processing results that can reliably survive an application > > restart? > > > > >
