All, We currently have graylog deployed like this:
Graylog ServerA (2.0.3) - 40G Ram, 20G Heap, runs MongoDB, currently takes ALL incoming logs. Graylog ServerB (2.0.3) - 40G Ram, 20G Heap, currently takes NO incoming logs. Three ElasticSearch Nodes (2.3.5) - 64G Ram each, ~30G Heap. Statistics: 3500 msgs per second avg We are not using a load balancer for log data yet, so all traffic goes to ServerA right now. We have moved about 60% of our logging traffic into it, and expect when finished to be at about 16k msg per second. While working through the project, we have added various streams and extractors as required. The goal was to keep these as few as possible, but the huge variety of logs input and formats has led to a fair number. We wanted to add a second node to graylog (not elasticsearch, but actual graylog) to prepare to spread out the processing load. The server will be called "ServerB" as listed above. We created the second node, copied relevant config, pointed it at our DBs (elasticsearch and mongo) and started things up. The node starts just fine, and appears to be basically healthy. Health looking things: 1. The In/Out counter at the top of the screen is running, and showing numbers we expect (~3500 per second) - this makes us think our ES connection is fine. 2. Users can login with ldap creds - this makes us think MongoDB connections are fine. 3. Streams - the counters show correctly as streams get messages. Now for the issue(s): 1. We see only incoming log message from a single source when searching the last five minutes. It is always the same source. This happens even we KNOW there are other log data from past five minutes. If we change to the past hour, all logs are there and appear correct. If we search past 15 minutes, we see all log data. Sometimes we log into the second node and can only see messages from this single source. 2. Streams - while the counts are there and appear correct, actually clicking into a stream and searching doesn't show any messages. Again, if you search past the 15 minute mark all messages are visible. Is this normal? I couldn't find a guide or set of specific instructions for what to do on the second node. It all seemed obvious, but I am wondering what we missed. Any pointers? Thanks !!! Dustin Tennill -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/31271ebf-c1d4-4459-9f20-8aa61ca48103%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
