Hi Petar, Nice you probably find your answer. I'll keep in mind that we are using regex as well on inputs and streams for alerting.
Did you raise the processors in the graylog2.conf? Maybe your are running out on processorbuffer_processors. You could double that for catching up your peak loads. processbuffer_processors = 8 second thing to look at is the "ring_size" Op woensdag 4 februari 2015 16:04:34 UTC+1 schreef Petar Koraca: > > Hello Arie, > > We use Microsoft Hyper-V as hypervisor. > > I believe our problem could be decompressGzip that decreases Graylog > throughput. I will let you know about our findings when we disable > compression on client side. > > Another thing that caused a lot of performance problems > was RegexExtractor. Throughput wen't down to few hundred per second. > > Btw I have tested Raw/Plaintext Input on another machine with 'pv > haproxy.log -L 50m -i 1 | nc 127.0.0.1 5555' and got 30-35k per second on > single virtualized machine :) > > Cheers > > > On Wednesday, January 28, 2015 at 9:29:14 PM UTC+1, Arie wrote: >> >> O I forgot (:- >> >> Are vmware-tools installed? We recently found some systems that where >> forgotten, and that has more impact than foreseen. >> >> Op woensdag 28 januari 2015 21:25:11 UTC+1 schreef Arie: >>> >>> Petar, >>> >>> we are running on bare metal, with a low load. Tested to 10k messages >>> with the http test input, >>> with everything on one (test)server and running well. >>> >>> I can tell you that in our production systems in our private/local cloud >>> we are encountering severe >>> network/disk related problems with our systems. All of your network is >>> CPU bound. Sometimes there >>> are delays that we can count in seconds. All VM Hosts is running @75% >>> CPU. >>> >>> Must say that we had problems with sending windows eventlogs thru >>> UDP/GELF with nxlog, those were >>> gone when switching tot TCP. >>> >>> Have you already done some graylog2 performance tweaks already? >>> >>> >>> >>> Op woensdag 28 januari 2015 14:41:43 UTC+1 schreef Petar Koraca: >>>> >>>> Thanks Arie. I have already tried that yesterday and did not help. >>>> >>>> I have removed unnecessary TCP input, and I don't have any >>>> NettyTransport exceptions now. >>>> >>>> I still have problem with RecvQ in peaks (as seen in netstat) which >>>> should be related to slow processing. >>>> >>>> Do you have any benchmark data with bare-metal vs VM, and different >>>> processors numbers / ring_size in graylog2.conf ? >>>> >>>> >>>> On Wed, Jan 28, 2015 at 11:49 AM, Arie <satya...@gmail.com> wrote: >>>> >>>>> Maybe this can be helpfull to you: >>>>> >>>>> >>>>> https://access.redhat.com/documentation/en-US/JBoss_Enterprise_Web_Platform/5/html/Administration_And_Configuration_Guide/jgroups-perf-udpbuffer.html >>>>> >>>>> or this for more advanced network tuning: >>>>> https://wwwx.cs.unc.edu/~sparkst/howto/network_tuning.php >>>>> >>>>> hth,, >>>>> >>>>> Arie >>>>> >>>>> >>>>> >>>>> On Tuesday, January 27, 2015 at 6:27:39 PM UTC+1, Petar Koraca wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> I have some performance issues with graylog2-server 0.92.4 (cannot >>>>>> process more than 7-8k per second), and I think it may be related to UDP >>>>>> buffers. This is CentOS 6 virtual machine with 16 vCPU. >>>>>> >>>>>> $ netstat -ulptn|grep 12201 >>>>>> tcp 0 0 :::12201 :::* >>>>>> LISTEN 2311/java >>>>>> udp 75960 0 :::12201 :::* >>>>>> 2311/java >>>>>> >>>>>> I've noticed this in my logs: >>>>>> >>>>>> 16:31:06,719 WARN [NettyTransport] receiveBufferSize (SO_RCVBUF) for >>>>>> [id: 0x537c78d9, /0:0:0:0:0:0:0:0:12201] should be 1048576 but is 43690. >>>>>> >>>>>> I've set udp_recvbuffer_sizes=1048576 but no luck. >>>>>> >>>>>> Also, I've set net.core.rmem_max from 124928 to 26214400. >>>>>> >>>>>> Any idea where did this 43690 come from? >>>>>> >>>>>> if you need additional information I am at your disposal. >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> Petar Koraca >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "graylog2" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/graylog2/SR9sqDyZqrU/unsubscribe. >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> graylog2+u...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
