Re: [graylog2] Increase OUTPUT speed
Hi, thats the PermGen Space, the jvm will create special objects in this area. The option is derecated with java8. you could leave it as it is. If you are still running with java7 then I think this option is with 256mb well defined. But thats the wild domain of jvm optimizing. One can perfectly spend a lifetime mastering all these options ;) Graylog will log periodically into its own server log statistics about the garbage collections. If you think you still have a performance problem, don't forget, there is a second jvm running with elasticsearch. Maybe your bottleneck is there? Are you running CPU bound due to expensive extractions taking place on the log events in your input queue. Are you running IO bound due to slow disks. There are a lot of options for performance gains and losses. Regards, Marcus Roberto Carna schrieb am Mi., 27. Juli 2016 um 13:26 Uhr: > Dear Marcus, special thanks for your comments. > > Please just a brief question: I have this line in > /etc/default/graylog-server (considering I defined the HEAP SIZE = > 16GB for Elasticsearch): > > GRAYLOG_SERVER_JAVA_OPTS="-Xms16g -Xmx18g -XX:NewRatio=1 > -XX:PermSize=256m -XX:MaxPermSize=512m -server -XX:+ResizeTLAB > -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled > -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC > -XX:-OmitStackTraceInFastThrow" > > What's the meaning of -XX:PermSize=256m -XX:MaxPermSize=512m ??? Do I > have to increase these options? > > Thanks again, regards. > > 2016-07-27 5:28 GMT-03:00 Marcus Franke : > > Hi Roberto, > > > > If you have performance issues with the processing, then you have two > > choices: > > > > Have a look into your server.conf settings. > > There are several options to possibly speed up the output performance > > output_batch_size defaults to 25? If you have thousands of messages in > the > > incoming then the batches are too small > > outputbuffer_processors .. if you have plenty of idle cpu power raise the > > value > > > > If you still have problems with the output buffer filling up you should > have > > a look at tuning your es cluster for more indexing speed. > > > > Then there is the obvious option, do you have enough RAM in your server? > > Have you raised Xms and Xmx settings for the jvm graylog-server is > running > > in? On RHEL based distributions this can be found in > > /etc/sysconfig/graylog-server. > > > > For example, I have set them to 24G for my setup, be aware of the java > > compressed pointers and thus you should not raise above 31G. > > > > > > regards, > > Marcus > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "Graylog Users" group. > > To unsubscribe from this topic, visit > > https://groups.google.com/d/topic/graylog2/Yz3jmpfqnwQ/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > > graylog2+unsubscr...@googlegroups.com. > > To view this discussion on the web visit > > > https://groups.google.com/d/msgid/graylog2/CAFRuYVcjSWLi1mKs%3DD9CJj5C41n%2BNXqaL2C2F_dfAbV_v7P%3DKw%40mail.gmail.com > . > > > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Graylog Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/CAG2Qp6t0vhgrYoxK81C8snYUzuS9qyEOa_Bg4bTa4gqfo1cTzQ%40mail.gmail.com > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFRuYVeC7kKYtzeuWTUYQx6A_x9FTuMEL4DWwC09dMeAvf-bZw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Increase OUTPUT speed
Dear Marcus, special thanks for your comments. Please just a brief question: I have this line in /etc/default/graylog-server (considering I defined the HEAP SIZE = 16GB for Elasticsearch): GRAYLOG_SERVER_JAVA_OPTS="-Xms16g -Xmx18g -XX:NewRatio=1 -XX:PermSize=256m -XX:MaxPermSize=512m -server -XX:+ResizeTLAB -XX:+UseConcMarkSweepGC -XX:+CMSConcurrentMTEnabled -XX:+CMSClassUnloadingEnabled -XX:+UseParNewGC -XX:-OmitStackTraceInFastThrow" What's the meaning of -XX:PermSize=256m -XX:MaxPermSize=512m ??? Do I have to increase these options? Thanks again, regards. 2016-07-27 5:28 GMT-03:00 Marcus Franke : > Hi Roberto, > > If you have performance issues with the processing, then you have two > choices: > > Have a look into your server.conf settings. > There are several options to possibly speed up the output performance > output_batch_size defaults to 25? If you have thousands of messages in the > incoming then the batches are too small > outputbuffer_processors .. if you have plenty of idle cpu power raise the > value > > If you still have problems with the output buffer filling up you should have > a look at tuning your es cluster for more indexing speed. > > Then there is the obvious option, do you have enough RAM in your server? > Have you raised Xms and Xmx settings for the jvm graylog-server is running > in? On RHEL based distributions this can be found in > /etc/sysconfig/graylog-server. > > For example, I have set them to 24G for my setup, be aware of the java > compressed pointers and thus you should not raise above 31G. > > > regards, > Marcus > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Graylog Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/graylog2/Yz3jmpfqnwQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > graylog2+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/graylog2/CAFRuYVcjSWLi1mKs%3DD9CJj5C41n%2BNXqaL2C2F_dfAbV_v7P%3DKw%40mail.gmail.com. > > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAG2Qp6t0vhgrYoxK81C8snYUzuS9qyEOa_Bg4bTa4gqfo1cTzQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [graylog2] Increase OUTPUT speed
Hi Roberto, If you have performance issues with the processing, then you have two choices: Have a look into your server.conf settings. There are several options to possibly speed up the output performance output_batch_size defaults to 25? If you have thousands of messages in the incoming then the batches are too small outputbuffer_processors .. if you have plenty of idle cpu power raise the value If you still have problems with the output buffer filling up you should have a look at tuning your es cluster for more indexing speed. Then there is the obvious option, do you have enough RAM in your server? Have you raised Xms and Xmx settings for the jvm graylog-server is running in? On RHEL based distributions this can be found in /etc/sysconfig/graylog-server. For example, I have set them to 24G for my setup, be aware of the java compressed pointers and thus you should not raise above 31G. regards, Marcus -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CAFRuYVcjSWLi1mKs%3DD9CJj5C41n%2BNXqaL2C2F_dfAbV_v7P%3DKw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Increase OUTPUT speed
Dear, my OUTPUT is too slow so the journal of my Graylog is increasing time after time. How can I speed up the OUTPUT in order to make it faster than the INPUT always?? Thanks a lot, Roberto -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/a500adc6-ed17-467b-82f1-272e15346a49%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
