Re: [collectd] Illegal attemp to update ...

[Anh K. Huynh]

On Wed, 29 Sep 2010 08:36:56 +0200
Florian Forster  wrote:

> Hi,
> 
> On Tue, Sep 28, 2010 at 05:46:24PM +0700, Anh K. Huynh wrote:
> > I see the following messages from collectd:
> > 
> > > illegal attempt to update using time 1285669914 when last update
> > > time is 1285669915 
> 
> > What's kind of problem?
> 
> somehow a value with the timestamp "1285669914" is submitted to the
> daemon. The daemon checks its internal cache and finds out that the
> last data received for this metric is "1285669915" -- i.e. the
> "new" value is older than the value before that.
> 
> The most common cause for this is that the metric is collected
> twice. Maybe the name isn't unique or maybe you're running two
> instances of collectd on a client machine.

Exactly:  Two instances of collectd were running on my client.

Thank you so much :)

-- 
Anh Ky Huynh

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] MySQL plugin reports wrong numbers on mysqld restart

[Bostjan Skufca]

This COUNTER->DERIVE does the trick. Transition script complete,
thanks for all your help.

b.


On 29 September 2010 20:15, Bostjan Skufca  wrote:
> Hey, by my calculations in six hours I should receive a reply with
> even easier solution to all issues mentioned and maybe some other ones
> too. :)
> Serious business below. :)
>
> On 29 September 2010 17:56, Florian Forster  wrote:
>> Hi Bostjan, Mariusz and everybody else ;)
>>
>> On Wed, Sep 29, 2010 at 02:47:08PM +0200, XANi wrote:
>>> Dnia 2010-09-29, śro o godzinie 03:09 +0200, Bostjan Skufca pisze:
>>> > Maybe it's time to make a script for automated dump/change obviously
>>> > invalid data to NaN/import of rrds.
>>
>> What's obvious to you is not obvious for a script. So I'm afraid you'll
>> be stuck with some manual tuning or write a script specifically suited
>> for your needs.
>
> Probably, but I'll have to do it anyway to fix the spikes.
>
>>> > Do you know how  why it was not DERIVE specified at rrd creation?
>>> Datatype was not DERIVE probably because mysql plugin (as many other)
>>> is much older than actual DERIVE support in collectd :)
>>
>> 100% right: Support for DERIVE was added relatively late – much later
>> than the MySQL plugin. The next major version, version 5.0, will likely
>> convert most of the COUNTER data sources to DERIVE – it's the better
>> default in most cases.
>
> Can you think of any example where it would be bad/improper to convert?
>
>>> > Probably I should consult google and manuals for this, but: is there
>>> > an easy way to change from COUNTER to DERIVE or do I have to dump
>>> > and reload the data?
>>> As for changing counter to derive, simple rrdtool dump to xml file and
>>> find/replace should be enougth
>>
>> That's too complicated. You can use "rrdtool tune" (rrdtune(1)) to do
>> this without much hassle:
>>
>>  rrdtool tune mysql_commands-select.rrd --data-source-type value:DERIVE
>
> From The Incredibles: "That's the way they do it!" Thanks for the hint!
>
>
> Best regards,
> b.
>

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] collectd plugin-specific intervals

[Bostjan Skufca]

By my experience separate collectd daemons are possible. I am
compiling it from scratch and all it needs besides the installed files
(binaries etc. ) is config file, pid file and potentially unix socket
file. Now if you create 2 separate configs which both point to
/var/lib/collectd (example) but each handle it's own rrd files I
believe it should not be a problem.

Managing two separate daemons with the same name by startup scripts is
another "happy hacking hour" but it is not impossible, actually far
from it:)

b.

PS: You can specify distinguishing config files and pid files on the
command line.


On 30 September 2010 02:46, Amos Shapira  wrote:
> +1 from me too.
>
> I'd even ask for a different collection interval per instance, e.g.
> maybe a "df" change on /home or /boot is less important to check on
> every second than a "df" change on /var/log.
>
> While we are on this topic - until this gets implemented - would it be
> possible to work around this by having separate collectd servers for
> different intervals?
>
> --Amos
>
> On 30 September 2010 08:53, Bostjan Skufca  wrote:
>> +1 for this feature
>>
>> It could really help with IO "hell" also.
>>
>> Nicolas: maybe you should currently apply some of the generic plugins
>> (exec, libperl, java) and collect the actual data only every 360th
>> time (if you use 10s interval and want to collect this specific item
>> only once every hour).
>>
>> b.
>>
>>
>> On 29 September 2010 09:24, Nicolas Michael  wrote:
>>> Hi,
>>>
>>> I recently came across collect and was impressed by its functionality and
>>> modular design. Great work!
>>>
>>> There is just one piece missing, which actually would be very important for 
>>> me
>>> to make use of it: A possibility to specifiy intervals on a per-plugin 
>>> basis.
>>>
>>> There are some counters which I need quite often (let's say, most OS 
>>> statistics
>>> as cpu, network etc.), and some statistics which are rather expensive to 
>>> collect
>>> and only change infrequently anyway (retrieved through some Oracle DB 
>>> queries).
>>> For the first set of data I would need at least 10 second intervals, while 
>>> the
>>> DB stuff should not be queried more often than once an hour. Those two 
>>> intervals
>>> are so wide apart that it's impossible to find any useful "compromise" here.
>>>
>>> Browsing through your archives I've found some discussions from February 
>>> this
>>> year about the exact same topic. Do you have any plans to implement such a
>>> feature in the near future (somehow generic for all plugins)? For my current
>>> purpose it might be sufficient to have such intervals for the oracle (and 
>>> maybe
>>> Exec, GenericJMX and Java) plugins, but of course a generic solution would 
>>> be
>>> even better.
>>>
>>> Thanks for your help,
>>> and again my respect for this great tool!
>>>
>>> Nick.
>>>
>>>
>>> ___
>>> collectd mailing list
>>> collectd@verplant.org
>>> http://mailman.verplant.org/listinfo/collectd
>>>
>>
>> ___
>> collectd mailing list
>> collectd@verplant.org
>> http://mailman.verplant.org/listinfo/collectd
>>
>

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] collectd plugin-specific intervals

[Amos Shapira]

+1 from me too.

I'd even ask for a different collection interval per instance, e.g.
maybe a "df" change on /home or /boot is less important to check on
every second than a "df" change on /var/log.

While we are on this topic - until this gets implemented - would it be
possible to work around this by having separate collectd servers for
different intervals?

--Amos

On 30 September 2010 08:53, Bostjan Skufca  wrote:
> +1 for this feature
>
> It could really help with IO "hell" also.
>
> Nicolas: maybe you should currently apply some of the generic plugins
> (exec, libperl, java) and collect the actual data only every 360th
> time (if you use 10s interval and want to collect this specific item
> only once every hour).
>
> b.
>
>
> On 29 September 2010 09:24, Nicolas Michael  wrote:
>> Hi,
>>
>> I recently came across collect and was impressed by its functionality and
>> modular design. Great work!
>>
>> There is just one piece missing, which actually would be very important for 
>> me
>> to make use of it: A possibility to specifiy intervals on a per-plugin basis.
>>
>> There are some counters which I need quite often (let's say, most OS 
>> statistics
>> as cpu, network etc.), and some statistics which are rather expensive to 
>> collect
>> and only change infrequently anyway (retrieved through some Oracle DB 
>> queries).
>> For the first set of data I would need at least 10 second intervals, while 
>> the
>> DB stuff should not be queried more often than once an hour. Those two 
>> intervals
>> are so wide apart that it's impossible to find any useful "compromise" here.
>>
>> Browsing through your archives I've found some discussions from February this
>> year about the exact same topic. Do you have any plans to implement such a
>> feature in the near future (somehow generic for all plugins)? For my current
>> purpose it might be sufficient to have such intervals for the oracle (and 
>> maybe
>> Exec, GenericJMX and Java) plugins, but of course a generic solution would be
>> even better.
>>
>> Thanks for your help,
>> and again my respect for this great tool!
>>
>> Nick.
>>
>>
>> ___
>> collectd mailing list
>> collectd@verplant.org
>> http://mailman.verplant.org/listinfo/collectd
>>
>
> ___
> collectd mailing list
> collectd@verplant.org
> http://mailman.verplant.org/listinfo/collectd
>

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] collectd plugin-specific intervals

[Bostjan Skufca]

+1 for this feature

It could really help with IO "hell" also.

Nicolas: maybe you should currently apply some of the generic plugins
(exec, libperl, java) and collect the actual data only every 360th
time (if you use 10s interval and want to collect this specific item
only once every hour).

b.


On 29 September 2010 09:24, Nicolas Michael  wrote:
> Hi,
>
> I recently came across collect and was impressed by its functionality and
> modular design. Great work!
>
> There is just one piece missing, which actually would be very important for me
> to make use of it: A possibility to specifiy intervals on a per-plugin basis.
>
> There are some counters which I need quite often (let's say, most OS 
> statistics
> as cpu, network etc.), and some statistics which are rather expensive to 
> collect
> and only change infrequently anyway (retrieved through some Oracle DB 
> queries).
> For the first set of data I would need at least 10 second intervals, while the
> DB stuff should not be queried more often than once an hour. Those two 
> intervals
> are so wide apart that it's impossible to find any useful "compromise" here.
>
> Browsing through your archives I've found some discussions from February this
> year about the exact same topic. Do you have any plans to implement such a
> feature in the near future (somehow generic for all plugins)? For my current
> purpose it might be sufficient to have such intervals for the oracle (and 
> maybe
> Exec, GenericJMX and Java) plugins, but of course a generic solution would be
> even better.
>
> Thanks for your help,
> and again my respect for this great tool!
>
> Nick.
>
>
> ___
> collectd mailing list
> collectd@verplant.org
> http://mailman.verplant.org/listinfo/collectd
>

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

[collectd] collectd plugin-specific intervals

[Nicolas Michael]

Hi,

I recently came across collect and was impressed by its functionality and
modular design. Great work!

There is just one piece missing, which actually would be very important for me
to make use of it: A possibility to specifiy intervals on a per-plugin basis. 

There are some counters which I need quite often (let's say, most OS statistics
as cpu, network etc.), and some statistics which are rather expensive to collect
and only change infrequently anyway (retrieved through some Oracle DB queries).
For the first set of data I would need at least 10 second intervals, while the
DB stuff should not be queried more often than once an hour. Those two intervals
are so wide apart that it's impossible to find any useful "compromise" here.

Browsing through your archives I've found some discussions from February this
year about the exact same topic. Do you have any plans to implement such a
feature in the near future (somehow generic for all plugins)? For my current
purpose it might be sufficient to have such intervals for the oracle (and maybe
Exec, GenericJMX and Java) plugins, but of course a generic solution would be
even better.

Thanks for your help,
and again my respect for this great tool!

Nick.


___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] MySQL plugin reports wrong numbers on mysqld restart

[Bostjan Skufca]

Hey, by my calculations in six hours I should receive a reply with
even easier solution to all issues mentioned and maybe some other ones
too. :)
Serious business below. :)

On 29 September 2010 17:56, Florian Forster  wrote:
> Hi Bostjan, Mariusz and everybody else ;)
>
> On Wed, Sep 29, 2010 at 02:47:08PM +0200, XANi wrote:
>> Dnia 2010-09-29, śro o godzinie 03:09 +0200, Bostjan Skufca pisze:
>> > Maybe it's time to make a script for automated dump/change obviously
>> > invalid data to NaN/import of rrds.
>
> What's obvious to you is not obvious for a script. So I'm afraid you'll
> be stuck with some manual tuning or write a script specifically suited
> for your needs.

Probably, but I'll have to do it anyway to fix the spikes.

>> > Do you know how  why it was not DERIVE specified at rrd creation?
>> Datatype was not DERIVE probably because mysql plugin (as many other)
>> is much older than actual DERIVE support in collectd :)
>
> 100% right: Support for DERIVE was added relatively late – much later
> than the MySQL plugin. The next major version, version 5.0, will likely
> convert most of the COUNTER data sources to DERIVE – it's the better
> default in most cases.

Can you think of any example where it would be bad/improper to convert?

>> > Probably I should consult google and manuals for this, but: is there
>> > an easy way to change from COUNTER to DERIVE or do I have to dump
>> > and reload the data?
>> As for changing counter to derive, simple rrdtool dump to xml file and
>> find/replace should be enougth
>
> That's too complicated. You can use "rrdtool tune" (rrdtune(1)) to do
> this without much hassle:
>
>  rrdtool tune mysql_commands-select.rrd --data-source-type value:DERIVE

From The Incredibles: "That's the way they do it!" Thanks for the hint!


Best regards,
b.

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] MySQL plugin reports wrong numbers on mysqld restart

[Florian Forster]

Hi Bostjan, Mariusz and everybody else ;)

On Wed, Sep 29, 2010 at 02:47:08PM +0200, XANi wrote:
> Dnia 2010-09-29, śro o godzinie 03:09 +0200, Bostjan Skufca pisze:
> > Maybe it's time to make a script for automated dump/change obviously
> > invalid data to NaN/import of rrds.

What's obvious to you is not obvious for a script. So I'm afraid you'll
be stuck with some manual tuning or write a script specifically suited
for your needs.

> > Do you know how  why it was not DERIVE specified at rrd creation?
> Datatype was not DERIVE probably because mysql plugin (as many other)
> is much older than actual DERIVE support in collectd :)

100% right: Support for DERIVE was added relatively late – much later
than the MySQL plugin. The next major version, version 5.0, will likely
convert most of the COUNTER data sources to DERIVE – it's the better
default in most cases.

> > Probably I should consult google and manuals for this, but: is there
> > an easy way to change from COUNTER to DERIVE or do I have to dump
> > and reload the data?
> As for changing counter to derive, simple rrdtool dump to xml file and
> find/replace should be enougth

That's too complicated. You can use "rrdtool tune" (rrdtune(1)) to do
this without much hassle:

  rrdtool tune mysql_commands-select.rrd --data-source-type value:DERIVE

Regards,
—octo
-- 
Florian octo Forster
Hacker in training
GnuPG: 0x0C705A15
http://octo.it/


signature.asc
Description: Digital signature
___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] MySQL plugin reports wrong numbers on mysqld restart

[XANi]

Dnia 2010-09-29, śro o godzinie 03:09 +0200, Bostjan Skufca pisze:

> Maybe it's time to make a script for automated dump/change obviously
> invalid data to NaN/import of rrds.
> 
> Anyway, it is indeed COUNTER in my RRD files.
> Do you know how  why it was not DERIVE specified at rrd creation?
> 
> Probably I should consult google and manuals for this, but: is there
> an easy way to change from COUNTER to DERIVE or do I have to dump and
> reload the data?
> 
> Thanks!
> b.
> 

Easiest fix would be stopping collectd for a minute, and then mysql
restart. Datatype was not DERIVE probably because mysql plugin (as many
other) is much older than actual DERIVE support in collectd :)
As for changing counter to derive, simple rrdtool dump to xml file and
find/replace should be enougth

-- 
Mariusz Gronczewski (XANi) 
GnuPG: 0xEA8ACE64
http://devrandom.pl


signature.asc
Description: This is a digitally signed message part
___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] Using SNMP plugin with collectd

[Sen, Kakoli]

Hi,

Thanks for your reply.

The host-resource mib has the following OID for processor usage:

host.hrDevice.hrProcessorTable.hrProcessorEntry.hrProcessorLoad



 But since we have 2 processors in the windows machine, the windows SNMP 
service is returning the following OIDs for the 2 processors which form the 
leafnodes of the MIB tree:

host.hrDevice.hrProcessorTable.hrProcessorEntry.hrProcessorLoad.15

host.hrDevice.hrProcessorTable.hrProcessorEntry.hrProcessorLoad.16



I have put the following block in Collectd.conf for capturing this data:





   Type "cpustats_percent"

   Table true

   Instance "HOST-RESOURCES-MIB::hrProcessorEntry"

   Values "HOST-RESOURCES-MIB::hrProcessorLoad"



And the data for both processors are coming in the same csv file named 
cpustats_percent-0-2010-09-29

Now the question is how do I get the 2 processor values in 2 different files 
without specifying the last numbersin the OID, in this case, 15 and 16, in the 
collectd.conf.



We need the configuration to be generic, applicable to any windows node.

Note that, the windows snmp mib implementation gives these random numbers which 
will vary from machine to machine.



Hope I could clarify,

Thanks & Regards,

Kakoli



-Original Message-
From: Florian Forster [mailto:o...@collectd.org]
Sent: Wednesday, September 29, 2010 12:22 PM
To: Sen, Kakoli
Cc: collectd@verplant.org
Subject: Re: [collectd] Using SNMP plugin with collectd



Hi Kakoli,



On Tue, Sep 28, 2010 at 10:03:26AM +, Sen, Kakoli wrote:

> I need to collect cpu/memory/disk usage of windows node.



if you could use an evaluation version of SSC Serv [0], please let me know. The 
"free edition" is capable of collecting CPU and interface statistics, the full 
version collects memory, disk (among others) as well.



> In collectd.conf, in snmp Data block it is given as Table true Values

> "HOST-RESOURCES-MIB::hrProcessorLoad "

>

> But for a multi-processor node, in the same csv file values of both

> processors are coming.



When configuring a *table*, the plugin expects multiple values to be returned. 
If you do not specify the "Instance" option, the last part of the OID will be 
used as instance, in the hope that this is unique.



If, however, the data is returned as



  hrProcessorLoad.1.0 = 12

  hrProcessorLoad.2.0 = 13



both values will use "0" (the last part of the OID) as instance.



Could you provide an snmpwalk of the entire table? This would make it a lot 
easier to tell you how to configure the plugin.



Best regards,

--octo



[0] 

--

Florian octo Forster

Hacker in training

GnuPG: 0x0C705A15

http://octo.it/
___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd