[uknof] Bandwidth graphs

[Ed Butler]

We are introducing a new bandwidth collection model, where instead of using
off the shelf tools like RRD etc, we are bringing data into a database. The
challenge we have currently with this is how to display the data to clients
in as pretty a way as possible.

We've found the libraries nvd3.org and Google Charts, both which are pretty
decent but I have a niggling feeling there is something truly whizz-bang
out there. We're happy to pay for something decent.

Has anyone got suggestions?

-- 
Ed Butler

Re: [uknof] Bandwidth graphs

[Simon Lockhart]

On Fri May 02, 2014 at 02:33:42PM +0100, Ed Butler wrote:
> We are introducing a new bandwidth collection model, where instead of using
> off the shelf tools like RRD etc, we are bringing data into a database. The
> challenge we have currently with this is how to display the data to clients
> in as pretty a way as possible.

There's only so much you can sex up a bandwidth graph - particularly if you
want to be able to include it in an email, or print onto a invoice. There's
probably more benefit in analysing the graph - peak times, unexpected peaks,
min, max, 95th percentile, average, etc.

There are various graphing tools out there, most of which will generate pretty
good looking graphs, with various levels of work required to achieve. Much
depends on what language you're working in, and in what format you want the
graphs out.

Simon

Re: [uknof] Bandwidth graphs

[Keith Mitchell]

On 05/02/2014 09:33 AM, Ed Butler wrote:
> We are introducing a new bandwidth collection model, where instead
> of using off the shelf tools like RRD etc, we are bringing data into
> a database. The challenge we have currently with this is how to
> display the data to clients in as pretty a way as possible.
> 
> We've found the libraries nvd3.org  and Google
> Charts, both which are pretty decent but I have a niggling feeling
> there is something truly whizz-bang out there. We're happy to pay for
> something decent.
> 
> Has anyone got suggestions?

It's possible that offerings from the likes of Splunk and Guavas might
be what you are looking for.

Keith

Re: [uknof] Bandwidth graphs

[Charl Tintinger]

Logstash is also worth considering, Splunk is good but gets very pricy when
you start working with lots of data.


On Fri, May 2, 2014 at 4:47 PM, Keith Mitchell  wrote:

> On 05/02/2014 09:33 AM, Ed Butler wrote:
> > We are introducing a new bandwidth collection model, where instead
> > of using off the shelf tools like RRD etc, we are bringing data into
> > a database. The challenge we have currently with this is how to
> > display the data to clients in as pretty a way as possible.
> >
> > We've found the libraries nvd3.org  and Google
> > Charts, both which are pretty decent but I have a niggling feeling
> > there is something truly whizz-bang out there. We're happy to pay for
> > something decent.
> >
> > Has anyone got suggestions?
>
> It's possible that offerings from the likes of Splunk and Guavas might
> be what you are looking for.
>
> Keith
>
>

Re: [uknof] Bandwidth graphs

[Graeme Fowler]

On 2 May 2014 17:01:09 Charl Tintinger  wrote:

Logstash is also worth considering


+1 to this. Have recently been introduced to logstash and it it is, 
frankly, brilliant.


Caution needed however: you really need to understand your data stream. And 
be good with regex.


Graeme

Re: [uknof] Bandwidth graphs

[Philip Gaw]

On 02/05/2014 21:14, Graeme Fowler wrote:


On 2 May 2014 17:01:09 Charl Tintinger  wrote:
> Logstash is also worth considering

+1 to this. Have recently been introduced to logstash and it it is, 
frankly, brilliant.


Caution needed however: you really need to understand your data 
stream. And be good with regex.




+1. Logstash & Kibana are a good combination, have also been looking at 
collectd/graphite.


Graeme

Re: [uknof] Bandwidth graphs

[Rob Greenwood]

Quite a big fan of highcharts (http://www.highcharts.com/) for any custom 
graphing I need to do. 

Are you using a relational database for this data? - a word of warning that 
time series databases exist for a reason :). Take a look at Graphite / Grafana 
for a good (almost) out-of-the-box solution.

-Rob

On 2 May 2014 at 14:39:47, Ed Butler (e...@tn13.com) wrote:

We are introducing a new bandwidth collection model, where instead of using off 
the shelf tools like RRD etc, we are bringing data into a database. The 
challenge we have currently with this is how to display the data to clients in 
as pretty a way as possible.

We've found the libraries nvd3.org and Google Charts, both which are pretty 
decent but I have a niggling feeling there is something truly whizz-bang out 
there. We're happy to pay for something decent.

Has anyone got suggestions?

--
Ed Butler

Re: [uknof] Bandwidth graphs

[Andy Davidson]

Hi,

Graeme Fowler wrote:
> On 2 May 2014 17:01:09 Charl Tintinger  wrote:
> > Logstash is also worth considering
> +1 to this. Have recently been introduced to logstash and it it is, frankly, 
> brilliant.

+1 to the +1.  If you have a lot of log events and you need to search them 
quickly, the ElasticSearch integration ("ELK Stack") is what you are looking 
for.

This is another quite good thing to look at as well :  http://www.graylog2.org/

Maybe doesn't answer the original question about graphing from Ed which I can 
summarise of "I'm storing data in a database, now help me turn it into a 
product." :-P  You need to know how it's going to scale so you should be 
looking at Graphite: http://graphite.readthedocs.org/en/latest/ as platform for 
this data that can scale to a modern production sized network.  This tool is 
going to let you store enough data to be interesting (you probably want to 
change the default granularity settings if you are going to use this for 
billing so that you keep detailed poll averages for three months or more, you 
definitely have to carefully ensure that the 
carbon/graphite-web/twisted/whisper software versions are a known good set 
because its very very beta at this stage.  It has a render url api that will 
let you integrate the data into your applications.  You are going to need to 
secure it so that customers can't render each others' data.

I love graphs.

Andy

Re: [uknof] Bandwidth graphs

[Rob Greenwood]

There’s a couple of interesting alternatives to Graphite (if you’re interested 
in scalability/performance..):
 
OpenTSDB: http://opentsdb.net/
InfluxDB: http://influxdb.org/

In terms of dashboards and graph rendering, I’d also recommend Grafana: 
http://grafana.org/, which has support for Graphite & InfluxDB, and an active 
pull request for OpenTSDB.

-Rob
On 7 May 2014 at 19:58:25, Andy Davidson (a...@nosignal.org) wrote:

Hi,  

Graeme Fowler wrote:  
> On 2 May 2014 17:01:09 Charl Tintinger  wrote:  
> > Logstash is also worth considering  
> +1 to this. Have recently been introduced to logstash and it it is, frankly, 
> brilliant.  

+1 to the +1. If you have a lot of log events and you need to search them 
quickly, the ElasticSearch integration ("ELK Stack") is what you are looking 
for.  

This is another quite good thing to look at as well : http://www.graylog2.org/  

Maybe doesn't answer the original question about graphing from Ed which I can 
summarise of "I'm storing data in a database, now help me turn it into a 
product." :-P You need to know how it's going to scale so you should be looking 
at Graphite: http://graphite.readthedocs.org/en/latest/ as platform for this 
data that can scale to a modern production sized network. This tool is going to 
let you store enough data to be interesting (you probably want to change the 
default granularity settings if you are going to use this for billing so that 
you keep detailed poll averages for three months or more, you definitely have 
to carefully ensure that the carbon/graphite-web/twisted/whisper software 
versions are a known good set because its very very beta at this stage. It has 
a render url api that will let you integrate the data into your applications. 
You are going to need to secure it so that customers can't render each others' 
data.  

I love graphs.  

Andy  



signature.asc
Description: Message signed with OpenPGP using AMPGpg

Re: [uknof] Bandwidth graphs

[Ed Butler]

Thanks everyone for their suggestions. We looked at pretty much everything,
and ended up using NVD3. An example of its output in our deployment is here:

https://www.dropbox.com/s/ox6vsy111ichccd/NVD3.png

(Power usage for a rack in our DC)


On 2 May 2014 14:33, Ed Butler  wrote:

> We are introducing a new bandwidth collection model, where instead of
> using off the shelf tools like RRD etc, we are bringing data into a
> database. The challenge we have currently with this is how to display the
> data to clients in as pretty a way as possible.
>
> We've found the libraries nvd3.org and Google Charts, both which are
> pretty decent but I have a niggling feeling there is something truly
> whizz-bang out there. We're happy to pay for something decent.
>
> Has anyone got suggestions?
>
> --
> Ed Butler
>



-- 
Ed Butler

Re: [uknof] Bandwidth graphs

[Nat Morris]

On 5 June 2014 07:25, Ed Butler  wrote:
> Thanks everyone for their suggestions. We looked at pretty much everything,
> and ended up using NVD3. An example of its output in our deployment is here:
>
> https://www.dropbox.com/s/ox6vsy111ichccd/NVD3.png

Out of curiosity are you monitoring the power using in-rack IP enabled
PDUs, DIN rail mounted power meters or something else?

Thanks,

-- 
Nat

https://nat.ms
+44 7531 750292

Re: [uknof] Bandwidth graphs

[Ed Butler]

We're using C-Matic's strips which are installed in the distribution
boards. I had a look but can't find their website, the closest I can get to
is this:

http://www.schneider-electric.co.uk/sites/uk/en/company/brands/cmatic.page


On 5 June 2014 09:59, Nat Morris  wrote:

> On 5 June 2014 07:25, Ed Butler  wrote:
> > Thanks everyone for their suggestions. We looked at pretty much
> everything,
> > and ended up using NVD3. An example of its output in our deployment is
> here:
> >
> > https://www.dropbox.com/s/ox6vsy111ichccd/NVD3.png
>
> Out of curiosity are you monitoring the power using in-rack IP enabled
> PDUs, DIN rail mounted power meters or something else?
>
> Thanks,
>
> --
> Nat
>
> https://nat.ms
> +44 7531 750292
>



-- 
Ed Butler

Re: [uknof] Bandwidth graphs

[Chris Russell]

On 05/06/2014 07:25, Ed Butler wrote:

Thanks everyone for their suggestions. We looked at pretty much
everything, and ended up using NVD3. An example of its output in our
deployment is here:

https://www.dropbox.com/s/ox6vsy111ichccd/NVD3.png [2]


 Nice :)

 IIRC when we were putting our DC together, we were quoted something 
like $25k + $25k maintenance for ultimately the same thing




Chris