On Monday, July 25, 2016 at 1:45:16 PM UTC-4, roberto...@gmail.com wrote: > > People, I have a Graylog 1.3 server in just one Linux box (Debian 8), so I > mean I have one Elasticsearch node. > > Nowadays I'm receiveing about 4000/6000 logs/second. I had to increase the > memory heap size of JVM, and used CPU x 10 and RAM x 40GB and after that > everything seems OK, because I have near 200/800 unprocessed messages as > maximum everytime. > > When do you recommend to scale to more Elasticsearch nodes or to have > diferent MongoDB's or somethinh like that??? > > Is there a logs/seg threshold meaning I have to scale to a distributed > architecture??? > > Thanks a lot!!! > > Roberto >
I can tell you from experience it's unlikely any one server will handle that amount of logs per/sec. I had Graylog installed on a physical server with the same specs as my VMware hosts, except with less memory, and it couldn't stand the load. Your best bet is going to be to do an Elasticsearch cluster of two servers and have your Graylog server be a third node of that cluster. You'll want your Graylog server to be the Elasticsearch master and not store any data or do any indexing. That will push off much of the load and give you some resiliency. You don't have much to worry about with MongoDB. It mostly stores configuration settings, although I think it does store logs that can't be indexed to Elasticsearch (don't hold me to that statement, but I'm pretty sure that's what I've read). You could also load-balance several Graylog servers by running them behind HAProxy, or maybe even PFSense. I don't think you'll get the performance you want without doing so. I'm certainly no Graylog expert, but It would have to be one monster server to do everything with that much load. Another option is to simply limit what you log. If you're logging Windows, you'll get tons of junk log entries. You can have more granular control with the 'auditpol' command. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/781fbb06-cc43-4204-885d-80add6b9f26d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
