Hi Gary,
Elasticsearch is one backend that you can use with Decanter, but not the
only one. You can use JDBC, Cassandra, or even your own appender.
You can also filter the metrics that you want to store. By default, we
retrieve all metrics, that means large data set.
It's what we do most of the time in production: collecting the data that
we really want (to implement BAM, and define the SLA for instance).
Regards
JB
On 10/13/2015 11:46 PM, garyhodgson wrote:
Hi,
I was wondering how people are using decanter as a metrics platform? In
particular using Elastic Search to store metrics. I had previously thought
that ES wasn't deemed suitable for time-series data, but I recently read up
on aggregations and how to use them in Kibana, and it seems that ES + Kibana
via Decanter would be a reasonable, all-java, way of doing metrics.
My concern however is how to handle data volume. Running a pair of basic
collectors (camel and activemq) overnight, with the default scheduler
settings, seems to produce quite a bit of data (over 1GB for ~1M documents).
Whilst consumer disk storage is quite cheap, it will be difficult for me to
justify huge amounts of enterprise storage at $work for what is often seen
as a "nice to have" feature. Ideally I would like to mimic the storage
aggregation feature of Graphite which aggregates the metrics into ever
coarser blocks over time. My googling so far hasn't yielded any pre-made
solutions, and I suspect I may have to roll my own, but I wanted to ask how
others are dealing with this, if indeed it's deemed to be also a problem
elsewhere.
Cheers,
Gary
--
View this message in context:
http://karaf.922171.n3.nabble.com/Decanter-handling-metric-data-volumes-tp4043063.html
Sent from the Karaf - User mailing list archive at Nabble.com.
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
