Re: [collectd] Preventing collectd high memory usage

2015-12-15 Thread Yves Mettier 

Hello again,

For your information, I created a Pull Request in order to improve the 
doc.


https://github.com/collectd/collectd/pull/1471

Regards,
Yves

Le 2015-12-15 10:37, Yves Mettier a écrit :

Hello,

You can read this : https://github.com/collectd/collectd/pull/691

In short :

queue_length
nb values incoming. This is what you want to limit with
WriteQueueLimitHigh and WriteQueueLimitLow.

derive-dropped
this is what you drop when queue_length is greater than the limits you 
set.

If derive-dropped is nul, either everything is going well or you set
WriteQueueLimitHigh and WriteQueueLimitLow too high.
If derive-dropped is never nul, your problem is either that
WriteQueueLimitHigh and WriteQueueLimitLow are too low, or you have a
bottleneck somewhere.

cache_size
same as "echo LISTVAL | nc -u /var/.../collectd.sock | wc -l"

Regards,
Yves

Le 2015-12-14 20:09, Vaughn Wernersbach a écrit :

Hi André Luiz da Silva!  Thanks for the suggestion.  I’ll keep
switching to UDP in mind, though I’m inclined to stick with TCP if at
all possible (largely to minimize the required changes to my
environment).

All - Anyone have info on using CollectInternalStats?  Still need to
know what value to set WriteQueueLimitHigh to and having collectd tell
me definitely seems like the best bet.



Vaughn Wernersbach
BIDS Trading, L.P.

111 Broadway, Suite 1603 | New York, NY 10006
T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>

From: André Luiz da Silva [mailto:andresilvalu...@gmail.com]
Sent: Monday, December 14, 2015 11:18 AM
To: Vaughn Wernersbach <vaughn.wernersb...@bidstrading.com>
Cc: collectd@verplant.org
Subject: Re: [collectd] Preventing collectd high memory usage


Hi, Wenersbach! Did you try to send your metrics over UDP instead of
TCP? It's just a suggestion if you have a reliable network.



Regards.



Em 14/12/2015 1:54 PM, "Vaughn Wernersbach"
<vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>>
escreveu:
Hi,

In order to limit high memory usage (one of my servers hit 94% for
collectd) I’m interested in enabling the
WriteQueueLimitHigh<https://collectd.org/documentation/manpages/collectd.conf.5.shtml#writequeuelimithigh_highnum>
option, but I don’t know what value to use.  The docs suggest enabling
CollectInternalStats<https://collectd.org/documentation/manpages/collectd.conf.5.shtml#collectinternalstats_false_true>,
and that the “write_queue” plugin instance reports the info from
CollectInternalStats that I’m looking for.  However, I don’t see any
“write_queue” plugin nor do I see anything relevant being written by
my existing write plugins (syslog and graphite) after enabling
CollectInternalStats.


• How do I obtain data written collected when setting
CollectInternalStats to true?

• Alternately, are there any guidelines for setting the values
of WriteQueueLimitHigh?



Vaughn Wernersbach
BIDS Trading, L.P.

111 Broadway, Suite 1603 | New York, NY 10006
T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>


___
collectd mailing list
collectd@verplant.org<mailto:collectd@verplant.org>
http://mailman.verplant.org/listinfo/collectd

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd


___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] Preventing collectd high memory usage

2015-12-15 Thread Yves Mettier 

Hello,

You can read this : https://github.com/collectd/collectd/pull/691

In short :

queue_length
nb values incoming. This is what you want to limit with 
WriteQueueLimitHigh and WriteQueueLimitLow.


derive-dropped
this is what you drop when queue_length is greater than the limits you 
set.
If derive-dropped is nul, either everything is going well or you set 
WriteQueueLimitHigh and WriteQueueLimitLow too high.
If derive-dropped is never nul, your problem is either that 
WriteQueueLimitHigh and WriteQueueLimitLow are too low, or you have a 
bottleneck somewhere.


cache_size
same as "echo LISTVAL | nc -u /var/.../collectd.sock | wc -l"

Regards,
Yves

Le 2015-12-14 20:09, Vaughn Wernersbach a écrit :

Hi André Luiz da Silva!  Thanks for the suggestion.  I’ll keep
switching to UDP in mind, though I’m inclined to stick with TCP if at
all possible (largely to minimize the required changes to my
environment).

All - Anyone have info on using CollectInternalStats?  Still need to
know what value to set WriteQueueLimitHigh to and having collectd tell
me definitely seems like the best bet.



Vaughn Wernersbach
BIDS Trading, L.P.

111 Broadway, Suite 1603 | New York, NY 10006
T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>

From: André Luiz da Silva [mailto:andresilvalu...@gmail.com]
Sent: Monday, December 14, 2015 11:18 AM
To: Vaughn Wernersbach <vaughn.wernersb...@bidstrading.com>
Cc: collectd@verplant.org
Subject: Re: [collectd] Preventing collectd high memory usage


Hi, Wenersbach! Did you try to send your metrics over UDP instead of
TCP? It's just a suggestion if you have a reliable network.



Regards.



Em 14/12/2015 1:54 PM, "Vaughn Wernersbach"
<vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>>
escreveu:
Hi,

In order to limit high memory usage (one of my servers hit 94% for
collectd) I’m interested in enabling the
WriteQueueLimitHigh<https://collectd.org/documentation/manpages/collectd.conf.5.shtml#writequeuelimithigh_highnum>
option, but I don’t know what value to use.  The docs suggest enabling
CollectInternalStats<https://collectd.org/documentation/manpages/collectd.conf.5.shtml#collectinternalstats_false_true>,
and that the “write_queue” plugin instance reports the info from
CollectInternalStats that I’m looking for.  However, I don’t see any
“write_queue” plugin nor do I see anything relevant being written by
my existing write plugins (syslog and graphite) after enabling
CollectInternalStats.


• How do I obtain data written collected when setting
CollectInternalStats to true?

• Alternately, are there any guidelines for setting the values
of WriteQueueLimitHigh?



Vaughn Wernersbach
BIDS Trading, L.P.

111 Broadway, Suite 1603 | New York, NY 10006
T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>


___
collectd mailing list
collectd@verplant.org<mailto:collectd@verplant.org>
http://mailman.verplant.org/listinfo/collectd

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd


___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] Preventing collectd high memory usage

2015-12-14 Thread André Luiz da Silva 
Hi, Wenersbach! Did you try to send your metrics over UDP instead of TCP?
It's just a suggestion if you have a reliable network.


Regards.


Em 14/12/2015 1:54 PM, "Vaughn Wernersbach" <
vaughn.wernersb...@bidstrading.com> escreveu:

> Hi,
>
>
>
> In order to limit high memory usage (one of my servers hit 94% for
> collectd) I’m interested in enabling the WriteQueueLimitHigh
> 
> option, but I don’t know what value to use.  The docs suggest enabling
> CollectInternalStats
> ,
> and that the “write_queue” plugin instance reports the info from
> CollectInternalStats that I’m looking for.  However, I don’t see any
> “write_queue” plugin nor do I see anything relevant being written by my
> existing write plugins (syslog and graphite) after enabling
> CollectInternalStats.
>
>
>
> · How do I obtain data written collected when setting
> CollectInternalStats to true?
>
> · Alternately, are there any guidelines for setting the values of
> WriteQueueLimitHigh?
>
>
>
>
>
>
>
> *Vaughn Wernersbach*
>
> *BIDS Trading, L.P.*
>
>
>
> 111 Broadway, Suite 1603 | New York, NY 10006
>
> T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
> vaughn.wernersb...@bidstrading.com
>
>
>
> ___
> collectd mailing list
> collectd@verplant.org
> http://mailman.verplant.org/listinfo/collectd
>
>
___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] Preventing collectd high memory usage

2015-12-14 Thread Vaughn Wernersbach 
Hi André Luiz da Silva!  Thanks for the suggestion.  I’ll keep switching to UDP 
in mind, though I’m inclined to stick with TCP if at all possible (largely to 
minimize the required changes to my environment).

All - Anyone have info on using CollectInternalStats?  Still need to know what 
value to set WriteQueueLimitHigh to and having collectd tell me definitely 
seems like the best bet.



Vaughn Wernersbach
BIDS Trading, L.P.

111 Broadway, Suite 1603 | New York, NY 10006
T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E: 
vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>

From: André Luiz da Silva [mailto:andresilvalu...@gmail.com]
Sent: Monday, December 14, 2015 11:18 AM
To: Vaughn Wernersbach <vaughn.wernersb...@bidstrading.com>
Cc: collectd@verplant.org
Subject: Re: [collectd] Preventing collectd high memory usage


Hi, Wenersbach! Did you try to send your metrics over UDP instead of TCP? It's 
just a suggestion if you have a reliable network.



Regards.



Em 14/12/2015 1:54 PM, "Vaughn Wernersbach" 
<vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>> 
escreveu:
Hi,

In order to limit high memory usage (one of my servers hit 94% for collectd) 
I’m interested in enabling the 
WriteQueueLimitHigh<https://collectd.org/documentation/manpages/collectd.conf.5.shtml#writequeuelimithigh_highnum>
 option, but I don’t know what value to use.  The docs suggest enabling 
CollectInternalStats<https://collectd.org/documentation/manpages/collectd.conf.5.shtml#collectinternalstats_false_true>,
 and that the “write_queue” plugin instance reports the info from 
CollectInternalStats that I’m looking for.  However, I don’t see any 
“write_queue” plugin nor do I see anything relevant being written by my 
existing write plugins (syslog and graphite) after enabling 
CollectInternalStats.


• How do I obtain data written collected when setting 
CollectInternalStats to true?

• Alternately, are there any guidelines for setting the values of 
WriteQueueLimitHigh?



Vaughn Wernersbach
BIDS Trading, L.P.

111 Broadway, Suite 1603 | New York, NY 10006
T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E: 
vaughn.wernersb...@bidstrading.com<mailto:vaughn.wernersb...@bidstrading.com>


___
collectd mailing list
collectd@verplant.org<mailto:collectd@verplant.org>
http://mailman.verplant.org/listinfo/collectd
___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd

Re: [collectd] Preventing collectd high memory usage

2015-12-14 Thread Wilfried Goesgens 
Hi everyone,   

We also found out, that replacing the system allocator for collect with
jemalloc or tcmalloc significantly improves the enduring memory footprint.  


We suffer from a problem called 'heap fragmentation' here to wich the glibc's
allocation algorith is vulnerable.  

It will reserve chunks of memory from the system; and rent it to parts of the
process. Once such a block is full, a new blog will be reserved.  

However, only the very latest block will be used to rent out memory to the
application. So, if you have a single byte valid allocated for a longer time
in such a memory block, it will remain 'forever yours'.  
>  Mon Dec 14 2015 14:09:35 EST from "Vaughn Wernersbach"
><vaughn.wernersb...@bidstrading.com>  Subject: Re: [collectd] Preventing
>collectd high memory usage
>
>
>  
>
>Hi André Luiz da Silva!  Thanks for the suggestion.  I’ll keep
>switching to UDP in mind, though I’m inclined to stick with TCP if at all
>possible (largely to minimize the required changes to my environment).  
>
>   
>
>All - Anyone have info on using CollectInternalStats?  Still need to know
>what value to set WriteQueueLimitHigh to and having collectd tell me
>definitely seems like the best bet.  
>
>   
>
>   
>
>   
>
>*Vaughn Wernersbach*  
>
>*BIDS Trading, L.P.*  
>
>   
>
>111 Broadway, Suite 1603 | New York, NY 10006  
>
>T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
>vaughn.wernersb...@bidstrading.com  
>
>   
>
>*From:* André Luiz da Silva [mailto:andresilvalu...@gmail.com] 
> *Sent:* Monday, December 14, 2015 11:18 AM
> *To:* Vaughn Wernersbach <vaughn.wernersb...@bidstrading.com>
> *Cc:* collectd@verplant.org
> *Subject:* Re: [collectd] Preventing collectd high memory usage  
>
> 
>
>Hi, Wenersbach! Did you try to send your metrics over UDP instead of TCP?
>It's just a suggestion if you have a reliable network.  
>
>   
>
>Regards.  
>
>   
>
>Em 14/12/2015 1:54 PM, "Vaughn Wernersbach"
><vaughn.wernersb...@bidstrading.com> escreveu:
>>  
>>
>>Hi,  
>>
>>   
>>
>>In order to limit high memory usage (one of my servers hit 94% for
>>collectd) I’m interested in enabling the  WriteQueueLimitHigh option, but I
>>don’t know what value to use.  The docs suggest enabling 
>>CollectInternalStats, and that the “write_queue” plugin instance reports
>>the info from CollectInternalStats that I’m looking for.  However, I
>>don’t see any “write_queue” plugin nor do I see anything relevant being
>>written by my existing write plugins (syslog and graphite) after enabling
>>CollectInternalStats.  
>>
>>   
>>
>>· How do I obtain data written collected when setting
>>CollectInternalStats to true?  
>>
>>· Alternately, are there any guidelines for setting the
>>values of WriteQueueLimitHigh?  
>>
>>   
>>
>>   
>>
>>   
>>
>>*Vaughn Wernersbach*  
>>
>>*BIDS Trading, L.P.*  
>>
>>   
>>
>>111 Broadway, Suite 1603 | New York, NY 10006  
>>
>>T: 212.618.2071 |  F: 212.618.2031  |  C: 732.397.8344  | E:
>>vaughn.wernersbach@bidstradingcom  
>>
>>   
>>
>>  
>>
>>  
>>
>>
>> ___
>> collectd mailing list
>> collectd@verplant.org
>> http://mailman.verplant.org/listinfo/collectd  

>  
>
>  
>
>  
>
>
>  
>
>   (, 0 bytes) [View| Download]
>   
>
>  
>
>  
>
>  

  

 

___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd
___
collectd mailing list
collectd@verplant.org
http://mailman.verplant.org/listinfo/collectd