Hi JB, Thanks for the response. It sounds like filtering is a sensible first step, as there are only a handlfull of metrics we really need to track. That might make an ES storage aggregation feature easier to implement too.
Cheers, Gary On 14 October 2015 at 06:09, jbonofre [via Karaf] < [email protected]> wrote: > Hi Gary, > > Elasticsearch is one backend that you can use with Decanter, but not the > only one. You can use JDBC, Cassandra, or even your own appender. > > You can also filter the metrics that you want to store. By default, we > retrieve all metrics, that means large data set. > It's what we do most of the time in production: collecting the data that > we really want (to implement BAM, and define the SLA for instance). > > Regards > JB > > On 10/13/2015 11:46 PM, garyhodgson wrote: > > > Hi, > > > > I was wondering how people are using decanter as a metrics platform? In > > particular using Elastic Search to store metrics. I had previously > thought > > that ES wasn't deemed suitable for time-series data, but I recently read > up > > on aggregations and how to use them in Kibana, and it seems that ES + > Kibana > > via Decanter would be a reasonable, all-java, way of doing metrics. > > > > My concern however is how to handle data volume. Running a pair of > basic > > collectors (camel and activemq) overnight, with the default scheduler > > settings, seems to produce quite a bit of data (over 1GB for ~1M > documents). > > Whilst consumer disk storage is quite cheap, it will be difficult for me > to > > justify huge amounts of enterprise storage at $work for what is often > seen > > as a "nice to have" feature. Ideally I would like to mimic the storage > > aggregation feature of Graphite which aggregates the metrics into ever > > coarser blocks over time. My googling so far hasn't yielded any > pre-made > > solutions, and I suspect I may have to roll my own, but I wanted to ask > how > > others are dealing with this, if indeed it's deemed to be also a problem > > elsewhere. > > > > Cheers, > > Gary > > > > > > > > -- > > View this message in context: > http://karaf.922171.n3.nabble.com/Decanter-handling-metric-data-volumes-tp4043063.html > > Sent from the Karaf - User mailing list archive at Nabble.com. > > > > -- > Jean-Baptiste Onofré > [hidden email] <http:///user/SendEmail.jtp?type=node&node=4043066&i=0> > http://blog.nanthrax.net > Talend - http://www.talend.com > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > > http://karaf.922171.n3.nabble.com/Decanter-handling-metric-data-volumes-tp4043063p4043066.html > To unsubscribe from Decanter - handling metric data volumes?, click here > <http://karaf.922171.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4043063&code=Y29udGFjdEBnYXJ5aG9kZ3Nvbi5jb218NDA0MzA2M3w4NTU3ODE3OA==> > . > NAML > <http://karaf.922171.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > -- View this message in context: http://karaf.922171.n3.nabble.com/Decanter-handling-metric-data-volumes-tp4043063p4043071.html Sent from the Karaf - User mailing list archive at Nabble.com.
