Hello Arie, We use Microsoft Hyper-V as hypervisor.
I believe our problem could be decompressGzip that decreases Graylog throughput. I will let you know about our findings when we disable compression on client side. Another thing that caused a lot of performance problems was RegexExtractor. Throughput wen't down to few hundred per second. Btw I have tested Raw/Plaintext Input on another machine with 'pv haproxy.log -L 50m -i 1 | nc 127.0.0.1 5555' and got 30-35k per second on single virtualized machine :) Cheers On Wednesday, January 28, 2015 at 9:29:14 PM UTC+1, Arie wrote: > > O I forgot (:- > > Are vmware-tools installed? We recently found some systems that where > forgotten, and that has more impact than foreseen. > > Op woensdag 28 januari 2015 21:25:11 UTC+1 schreef Arie: >> >> Petar, >> >> we are running on bare metal, with a low load. Tested to 10k messages >> with the http test input, >> with everything on one (test)server and running well. >> >> I can tell you that in our production systems in our private/local cloud >> we are encountering severe >> network/disk related problems with our systems. All of your network is >> CPU bound. Sometimes there >> are delays that we can count in seconds. All VM Hosts is running @75% CPU. >> >> Must say that we had problems with sending windows eventlogs thru >> UDP/GELF with nxlog, those were >> gone when switching tot TCP. >> >> Have you already done some graylog2 performance tweaks already? >> >> >> >> Op woensdag 28 januari 2015 14:41:43 UTC+1 schreef Petar Koraca: >>> >>> Thanks Arie. I have already tried that yesterday and did not help. >>> >>> I have removed unnecessary TCP input, and I don't have any >>> NettyTransport exceptions now. >>> >>> I still have problem with RecvQ in peaks (as seen in netstat) which >>> should be related to slow processing. >>> >>> Do you have any benchmark data with bare-metal vs VM, and different >>> processors numbers / ring_size in graylog2.conf ? >>> >>> >>> On Wed, Jan 28, 2015 at 11:49 AM, Arie <satya...@gmail.com> wrote: >>> >>>> Maybe this can be helpfull to you: >>>> >>>> >>>> https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html >>>> >>>> or this for more advanced network tuning: >>>> https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php >>>> >>>> hth,, >>>> >>>> Arie >>>> >>>> >>>> >>>> On Tuesday, January 27, 2015 at 6:27:39 PM UTC+1, Petar Koraca wrote: >>>>> >>>>> Hello, >>>>> >>>>> I have some performance issues with graylog2-server 0.92.4 (cannot >>>>> process more than 7-8k per second), and I think it may be related to UDP >>>>> buffers. This is CentOS 6 virtual machine with 16 vCPU. >>>>> >>>>> $ netstat -ulptn|grep 12201 >>>>> tcp 0 0 :::12201 :::* >>>>> LISTEN 2311/java >>>>> udp 75960 0 :::12201 :::* >>>>> 2311/java >>>>> >>>>> I've noticed this in my logs: >>>>> >>>>> 16:31:06,719 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for >>>>> [id: 0x537c78d9, /0:0:0:0:0:0:0:0:12201] should be 1048576 but is 43690. >>>>> >>>>> I've set udp_recvbuffer_sizes=1048576 but no luck. >>>>> >>>>> Also, I've set net.core.rmem_max from 124928 to 26214400. >>>>> >>>>> Any idea where did this 43690 come from? >>>>> >>>>> if you need additional information I am at your disposal. >>>>> >>>>> Kind regards, >>>>> >>>>> Petar Koraca >>>>> >>>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "graylog2" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/graylog2/SR9sqDyZqrU/unsubscribe. >>>> To unsubscribe from this group and all its topics, send an email to >>>> graylog2+u...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
