Yesterday we upgraded to Graylog 2.1. As a side effect, our second node now works correctly.
I can't explain why it works yet, but here is what we went through: 1. Time sync checked, both hosts were only a couple of seconds apart. 2. Elasticsearch was indexing very slowly, and we worked most of the day trying to figure out why. In the end, we decided to upgrade to ES 2.4.0. There were various reasons, but essentially we had performance issues and decided to try it. We were already at 2.3.5, and some of the tools we wanted to use to evaluate performance problems needed 2.4.0. This went well, and our indexing problems were resolved. 3. We upgraded Graylog next - this went well. Problems are solved for the moment - we are glad this list is here and people are listening. Thanks !!! On Monday, August 29, 2016 at 9:20:06 PM UTC-4, Dustin Tennill wrote: > > Eric/Jochen - thanks for the feedback. We have been working on > upgrade/moving elasticsearch data to larger nodes, and I feel fairly > confident it's healthy. > > We will check everything out and get back with findings in the next day or > so. > > Thanks !! > > > > On Mon, Aug 29, 2016 at 4:33 PM, Eric Green <eric.lee.gr...@gmail.com> > wrote: > >> >> >> On Friday, August 26, 2016 at 10:54:03 AM UTC-7, Dustin Tennill wrote: >>> >>> >>> Now for the issue(s): >>> 1. We see only incoming log message from a single source when searching >>> the last five minutes. It is always the same source. This happens even we >>> KNOW there are other log data from past five minutes. If we change to the >>> past hour, all logs are there and appear correct. If we search past 15 >>> minutes, we see all log data. Sometimes we log into the second node and can >>> only see messages from this single source. >>> 2. Streams - while the counts are there and appear correct, actually >>> clicking into a stream and searching doesn't show any messages. Again, if >>> you search past the 15 minute mark all messages are visible. >>> >>> Is this normal? I couldn't find a guide or set of specific instructions >>> for what to do on the second node. It all seemed obvious, but I am >>> wondering what we missed. >>> >>> >> Virtually every time I've run into weirdness like this, it has been a >> time issue. All servers should be running on UTC and sync'ed to a common >> NTP time server. Java itself should be configured to UTC with >> -Duser.timezone=UTC , I don't know what OS you're running, on Red Hat >> derived systems it should be in the GRAYLOG_SERVER_JAVA_OPTS= variable in >> the file /etc/sysconfig/graylog-server along with your tweaks to -Xms and >> Xmx . Also, make sure that both nodes are pointing at the exact same Mongo >> server and exact same set of Elasticsearch nodes and that the Elasticsearch >> nodes are similarly configured in UTC with its java opts set to UTC. And >> finally, make sure that your *source* servers are set to UTC and is sync'ed >> via NTP. If your source servers' time is off, then the time can be off in >> the syslog messages that Graylog is receiving. >> >> You can also check your Elasticsearch cluster's health to see if it has >> pending tasks, delayed unassigned shards, etc. that could be holding up >> processing. E.g. curl -XGET ' >> http://localhost:9200/_cluster/health?pretty=true' . But I seriously >> doubt it... >> >> In general, if you're following the rules adding a second Graylog >> instance is pretty easy. The hard part is creating a load balancer to >> spread the syslog messages across them, which is why I have syslog-ng in my >> infrastructure. But of course if your time is off, then everything's going >> to be off. >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Graylog Users" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/graylog2/aY-dDoYGOfw/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> graylog2+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/graylog2/ac9f306e-2ec3-4829-9e52-ef8e37a75585%40googlegroups.com >> >> <https://groups.google.com/d/msgid/graylog2/ac9f306e-2ec3-4829-9e52-ef8e37a75585%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > > ---------------------------------------- > Dustin Tennill > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/cdb85226-edcc-4fab-9d8e-101450ba97b1%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
