Re: [rrd-users] Question regarding missing data points

[Buescher, Friedhelm]

> Let me elaborate on this:
>
> > rrdtool update test.rrd 1477433700:15
> > # rrdtool update test.rrd 1477434000:NaN   # here is a missed measuring
> > point
> > rrdtool update test.rrd 1477434300:8
>
> > Can anyone advise how to setup the rrd, so that one missing PDP gets
> > exactly one NaN ?
>
> The problem is that you get 2, not 1, unknown PDPs. That is because the
> update at timestamp 1477434300 will fill all the time between the last update
> (1477433700) and itself. The rate is unknown because of your heartbeat
> settings.
>
> As far as I know you cannot do something in RRDtool to say that updates are
> for 300 seconds only and if there is more time not updated, it should be
> processed as if it was updated with U. But who knows, maybe I'm missing
> something.
>
> Unless someone else comes up with a better solution, you will have to
> update using U (for unknown) or accept that one more PDP will also be
> unknown.  If the monitoring is offline for e.g. 55 minutes, the next update
> will try to fill 12 PDPs which will all be unknown.
>
> A quick hack, if you can spare the resources, is to update 300 seconds ago
> with rate U and just ignore the error RRDtool will give you if that update was
> already successfully done.
>
> A less quick hack is to modify your scripts so that it remembers the last
> update time (or queries it from the RRD) and looks at the difference with
> now. If that is more than 300 seconds, then update with rate U and time
> now-300s.
>
Yes, i implemented the last approach and it worked as expected:

Everytime I get a new value to update, I ask the rrd to for the last timestamp 
(rrd->last). If this timestamp is older than $now -monitoring_intervall, I 
update the timestamp $now - $monitoring_intervall with U. After that, I update 
$now with the current value. This gives exactly one missing block in RRD as 
expected.


Thanks for your support in thinking!

Best regards,
Friedhelm







CEWE Stiftung & Co. KGaA mit Sitz in Oldenburg; Registergericht Oldenburg HR B 
208214; Persönlich haftende geschäftsführende und vertretungsberechtigte 
Gesellschafterin:
Neumüller CEWE COLOR Stiftung, Sitz: Oldenburg;
Stiftungsverzeichnis der rechtsfähigen Stiftungen des bürgerlichen Rechts im 
Bezirk der Regierungsvertretung Oldenburg Nummer 15(034).
Stiftungsvorstand:
Dr. Rolf Hollander (Vorsitzender), Patrick Berkhouwer, Dr. Reiner Fageth, Dr. 
Christian Friege, Carsten Heitkamp, Dr. Olaf Holzkämper, Thomas Mehls, Frank 
Zweigle
Aufsichtsrat:
Otto Korte (Vorsitzender)
UST ID-Nr. DE815453806; St.-Nr. 64/200/38999; GLN 40 04468 0 4
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Question regarding missing data points

[Alex van den Bogaerdt]

Let me elaborate on this:

> rrdtool update test.rrd 1477433700:15
> # rrdtool update test.rrd 1477434000:NaN   # here is a missed measuring
> point
> rrdtool update test.rrd 1477434300:8

> Can anyone advise how to setup the rrd, so that one missing PDP gets
> exactly one NaN ?

The problem is that you get 2, not 1, unknown PDPs. That is because the
update at timestamp 1477434300 will fill all the time between the last
update (1477433700) and itself. The rate is unknown because of your
heartbeat settings.

As far as I know you cannot do something in RRDtool to say that updates
are for 300 seconds only and if there is more time not updated, it should
be processed as if it was updated with U. But who knows, maybe I'm missing
something.

Unless someone else comes up with a better solution, you will have to
update using U (for unknown) or accept that one more PDP will also be
unknown.  If the monitoring is offline for e.g. 55 minutes, the next
update will try to fill 12 PDPs which will all be unknown.

A quick hack, if you can spare the resources, is to update 300 seconds ago
with rate U and just ignore the error RRDtool will give you if that update
was already successfully done.

A less quick hack is to modify your scripts so that it remembers the last
update time (or queries it from the RRD) and looks at the difference with
now. If that is more than 300 seconds, then update with rate U and time
now-300s.



___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Question regarding missing data points

[Alex van den Bogaerdt]

> Alex van den Bogaerdt  wrote:
>
>> You are not writing points, you are writing rates which are measured
>> during some interval.
>> The rate "8" is for time 1477433700..1477434300. This is however not
>> allowed because of your heartbeat setting and gets lost.
>>
>> What happens if you do update at time 1477434000, using value U  ?
>
> Would that make a difference ?

Yes.

> I know that with a counter type, the second interval would still be
> unknown because the (b-a)/t calculation can't be done if a is unknown -
> but how does RRD handle the gauge type ? As "everything is a rate", does
> this limitation still apply with the gauge type ?

This is a different problem.

He is updating 600 seconds apart (because of the missing update) which is
more than heartbeat.




___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Question regarding missing data points

[Buescher, Friedhelm]

> > Hi,
> >
> > we use rrd to get an overview of our server farm. Every data point
> > that is monitored (disk usage, cpu, mem, temp., iops, bandwidth and so
> > on) gets written to rrd (and then gets visualized via drraw).
> >
> > Every now and then we get some timeout from the monitoring system
> > (read: a missing data point), but i’m unable to implement this in RRD.
> >
> > Minimal compiling sample:
> >
> > --
> > -
> > rm test.rrd
> > rrdtool create test.rrd --step 300 --start 1477432800
> > DS:testds:GAUGE:300:U:U RRA:LAST:0:1:10 rrdtool update test.rrd
> > 1477433100:12 rrdtool update test.rrd 1477433400:13 rrdtool update
> > test.rrd 1477433700:15
> > # rrdtool update test.rrd 1477434000:NaN   # here is a missed measuring
> > point
> > rrdtool update test.rrd 1477434300:8
> > rrdtool update test.rrd 1477434600:12
> > --
> > -
> >
> > As you can see, the data point for the timestamp 1477434000 will not
> > be written to rrd, but the following data point is available, again.
>
> You are not writing points, you are writing rates which are measured during
> some interval.
> The rate "8" is for time 1477433700..1477434300. This is however not allowed
> because of your heartbeat setting and gets lost.
>
> What happens if you do update at time 1477434000, using value U  ?
>
>

Hi, thanks for the quick Reply!

If i insert the missing value for timestamp 1477434000 with U, everything is OK.

But this is not really the desired (my desired)  behaviour. I'd expect a 
missing point or rate for one interval to be stored as one missing interval. 
Not two.

Or am I misunderstood the whole thing?


Best regards,
Friedhelm





CEWE Stiftung & Co. KGaA mit Sitz in Oldenburg; Registergericht Oldenburg HR B 
208214; Persönlich haftende geschäftsführende und vertretungsberechtigte 
Gesellschafterin:
Neumüller CEWE COLOR Stiftung, Sitz: Oldenburg;
Stiftungsverzeichnis der rechtsfähigen Stiftungen des bürgerlichen Rechts im 
Bezirk der Regierungsvertretung Oldenburg Nummer 15(034).
Stiftungsvorstand:
Dr. Rolf Hollander (Vorsitzender), Patrick Berkhouwer, Dr. Reiner Fageth, Dr. 
Christian Friege, Carsten Heitkamp, Dr. Olaf Holzkämper, Thomas Mehls, Frank 
Zweigle
Aufsichtsrat:
Otto Korte (Vorsitzender)
UST ID-Nr. DE815453806; St.-Nr. 64/200/38999; GLN 40 04468 0 4
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Question regarding missing data points

[Simon Hobson]

Alex van den Bogaerdt  wrote:

> You are not writing points, you are writing rates which are measured
> during some interval.
> The rate "8" is for time 1477433700..1477434300. This is however not
> allowed because of your heartbeat setting and gets lost.
> 
> What happens if you do update at time 1477434000, using value U  ?

Would that make a difference ?
I know that with a counter type, the second interval would still be unknown 
because the (b-a)/t calculation can't be done if a is unknown - but how does 
RRD handle the gauge type ? As "everything is a rate", does this limitation 
still apply with the gauge type ?
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

Re: [rrd-users] Question regarding missing data points

[Alex van den Bogaerdt]

> Hi,
>
> we use rrd to get an overview of our server farm. Every data point that is
> monitored (disk usage, cpu, mem, temp., iops, bandwidth and so on) gets
> written to rrd (and then gets visualized via drraw).
>
> Every now and then we get some timeout from the monitoring system (read: a
> missing data point), but i’m unable to implement this in RRD.
>
> Minimal compiling sample:
>
> ---
> rm test.rrd
> rrdtool create test.rrd --step 300 --start 1477432800
> DS:testds:GAUGE:300:U:U RRA:LAST:0:1:10
> rrdtool update test.rrd 1477433100:12
> rrdtool update test.rrd 1477433400:13
> rrdtool update test.rrd 1477433700:15
> # rrdtool update test.rrd 1477434000:NaN   # here is a missed measuring
> point
> rrdtool update test.rrd 1477434300:8
> rrdtool update test.rrd 1477434600:12
> ---
>
> As you can see, the data point for the timestamp 1477434000 will not be
> written to rrd, but the following data point is available, again.

You are not writing points, you are writing rates which are measured
during some interval.
The rate "8" is for time 1477433700..1477434300. This is however not
allowed because of your heartbeat setting and gets lost.

What happens if you do update at time 1477434000, using value U  ?


___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users

[rrd-users] Question regarding missing data points

[Buescher, Friedhelm]

Hi,

we use rrd to get an overview of our server farm. Every data point that is 
monitored (disk usage, cpu, mem, temp., iops, bandwidth and so on) gets written 
to rrd (and then gets visualized via drraw).

Every now and then we get some timeout from the monitoring system (read: a 
missing data point), but i’m unable to implement this in RRD.

Minimal compiling sample:

---
rm test.rrd
rrdtool create test.rrd --step 300 --start 1477432800 DS:testds:GAUGE:300:U:U 
RRA:LAST:0:1:10
rrdtool update test.rrd 1477433100:12
rrdtool update test.rrd 1477433400:13
rrdtool update test.rrd 1477433700:15
# rrdtool update test.rrd 1477434000:NaN   # here is a missed measuring point
rrdtool update test.rrd 1477434300:8
rrdtool update test.rrd 1477434600:12
---

As you can see, the data point for the timestamp 1477434000 will not be written 
to rrd, but the following data point is available, again.


But if you dump the rrd, you can notice some strange behaviour:

rrdtool dump test.rrd  | grep "2016-10-26 00"

 
NaN
 
1.20e+01
 
1.30e+01
 
1.50e+01
 
NaN
 
NaN
 
1.20e+01

1477432800 is missing, because we did not populate it with data.
1477434000 is missing, because there was a timeout

But why the heck is 1477434300 missing? We populated it with value 8.

Considerations regarding the definitions:
Step=300, because we check every 5 minutes
Heartbeat=300, because rrd shall not interpolate
Xff=0, if one PDP is missing, I want to see an NaN

Can anyone advise how to setup the rrd, so that one missing PDP gets exactly 
one NaN ?

Thank you very much!

Best regards,
Friedhelm








CEWE Stiftung & Co. KGaA mit Sitz in Oldenburg; Registergericht Oldenburg HR B 
208214; Persönlich haftende geschäftsführende und vertretungsberechtigte 
Gesellschafterin:
Neumüller CEWE COLOR Stiftung, Sitz: Oldenburg;
Stiftungsverzeichnis der rechtsfähigen Stiftungen des bürgerlichen Rechts im 
Bezirk der Regierungsvertretung Oldenburg Nummer 15(034).
Stiftungsvorstand:
Dr. Rolf Hollander (Vorsitzender), Patrick Berkhouwer, Dr. Reiner Fageth, Dr. 
Christian Friege, Carsten Heitkamp, Dr. Olaf Holzkämper, Thomas Mehls, Frank 
Zweigle
Aufsichtsrat:
Otto Korte (Vorsitzender)
UST ID-Nr. DE815453806; St.-Nr. 64/200/38999; GLN 40 04468 0 4
___
rrd-users mailing list
rrd-users@lists.oetiker.ch
https://lists.oetiker.ch/cgi-bin/listinfo/rrd-users