This seems like a potential bug to me. Due to an unrelated and undiagnosed issue, my elasticsearch cluster isn't allocating shards successfully every night. So this problem is recurring for me each day in the following scenario:
1. Index rotation set to time based, P1D. 2. New index is created, but shard allocation fails for some shards, so elasticsearch status goes red. 3. Graylog does not move the deflector alias and continues logging in yesterday's index. 4. Relative searches do not show data UNLESS you change the relative search time window to a large enough value to include data from yesterday. Then all data shows up. 5. Manually recalculating index ranges resolves the issue (at least temporarily). Not super familiar w/ the graylog code base, but this feels like there's a bug in the index selection algorithm and/or index range maintenance routines. The query analyzer is excluding yesterday's index when in fact that is the one that has the events we're looking for. Doing anything to trick the query analyzer into including that index makes the query start succeeding. On Tuesday, November 1, 2016 at 2:47:41 PM UTC-4, 123Dev wrote: > > How often "Recalculate Index Ranges" are automatically performed? > What controls that? > > It appears that when I "recalculate the index Ranges", messages are > detected. > but 5 minutes later, messages in the last 5m window are gone, obviously > 15m window still show data. > but as more time passes on, we have to widen the search to a larger window > to get the data. > > Can I (should I?) find a way to trigger "Recalculate Index Ranges" every > minute? > This command doesn't seem to be doing the job > > curl -XPOST http://127.0.0.1:9000/api/system/indices/ranges/rebuild > > > I can tell because I don't see logs in graylog, nor do I see the ui > updated with the current timestamp. > > Our rotation strategy which hasn't changed for the longest time, which is > set to: > Index rotation strategy: Index Time > Rotation period: P1D (1d, a day) > Index retention strategy: Delete > Max number of indices: 28 > > So all of the current data should be present in the latest index for 24 > hours, why isn't Graylog able to find it, it's not like it is changing. > > > > On Tuesday, November 1, 2016 at 1:07:06 PM UTC-4, 123Dev wrote: >> >> I've been pulling my hair trying to figure this issue out. >> I've done countless number of "Recalculate Index Ranges" >> In fact maybe my problem is with "Recalculate Index Ranges" because just >> before my last try, I was getting messages, and now I don't. >> >> This is also playing havoc on many of the alerts that we have setup, >> because they're all being triggered as the system detecting inactivity. >> >> I'm beginning to think that this is a bug in Graylog, >> I copied the elasticsearch query >> ``` >> { >> "from": 0, >> "size": 150, >> "query": { >> "bool": { >> "must": { >> "match_all": {} >> }, >> "filter": { >> "bool": { >> "must": { >> "range": { >> "timestamp": { >> "from": "2016-11-01 16:15:32.456", >> "to": "2016-11-01 16:20:32.456", >> "include_lower": true, >> "include_upper": true >> } >> } >> } >> } >> } >> } >> }, >> "sort": [ >> { >> "timestamp": { >> "order": "desc" >> } >> } >> ], >> "highlight": { >> "require_field_match": false, >> "fields": { >> "*": { >> "fragment_size": 0, >> "number_of_fragments": 0 >> } >> } >> } >> } >> ``` >> >> and executed directly on the Elasticsearch cluster, and it returned the >> results, so why isn't Graylog returning any results? >> >> As I was posting this, Graylog had reported that: >> Range re-calculated 3 hours ago in 75ms. 176 segments, 1 open search >> contexts, 0 deleted messages >> >> Redid the "Recalculate Index Ranges" >> Range re-calculated a minute ago in 110ms. 182 segments, 0 open search >> contexts, 0 deleted messages >> >> And I started getting results again. >> Though couple of minutes later, NO more messages again. >> >> The problem is even worse, Statistics on non-numeric fields are >> non-existant. >> No Total, no cardinality, nada ... >> >> The only think I can think of that has recently changed is we added >> "Elasticsearch client node" for Kibana. >> Not sure if that has anything to do with it. >> >> >> >> >> On Tuesday, November 1, 2016 at 12:25:58 PM UTC-4, >> daniel...@hagan-consulting.com wrote: >>> >>> Surprisingly, I ran into this issue this morning as well. After some >>> poking around I found that my current index time ranges hadn't been updated >>> in 15 hours, which I assume was breaking the logic used to select indices >>> for searching. In System -> Indices, it said the index range hadn't been >>> updated in 15 hours. I went to System -> Indices and selected Maintenance >>> -> Recalculate index ranges, and that fixed it. >>> >>> On Monday, October 31, 2016 at 4:02:44 PM UTC-4, 123Dev wrote: >>>> >>>> Was this ever resolved. >>>> i never had this issue, and been running Graylog for a long time. >>>> >>>> Just today with the latest Graylog (2.1.1) >>>> Search * for Last, 5m 15m, 30m, 1hr don't return and results >>>> Search * in the last 2 hours returns messages just few seconds ago, >>>> This is across the board, in a clustered environment, with multiple >>>> sources. >>>> Impossible for all those sources to have the wrong dates. >>>> I checked all Graylog nodes ES Nodes and they all have the correct >>>> dates / timezones. >>>> >>>> What gives? >>>> Really puzzling. >>>> >>>> Thanks >>>> >>>> On Wednesday, April 15, 2015 at 4:22:18 PM UTC-4, Zi Dvbelju wrote: >>>>> >>>>> Hey Edmundo, >>>>> >>>>> Graylog user timezone and messages timezone are correct. Has been >>>>> setup that way for over a year, and I have confirmed they are still setup >>>>> correctly. >>>>> >>>>> I upgraded to the latest version hoping that would fix it, but still >>>>> running into the same issue. >>>>> >>>>> Any other suggestions? Error logs don't show any issues. >>>>> >>>>> Only relative searches return data; nothing from absolute. >>>>> >>>>> Thanks, >>>>> Z >>>>> >>>>> >>>>> On Tuesday, January 20, 2015 at 2:40:53 PM UTC-5, Edmundo Alvarez >>>>> wrote: >>>>>> >>>>>> Hello Zach, >>>>>> >>>>>> I would start taking a look at the time configuration, specially >>>>>> timezones. Could you verify the time settings in the machines sending >>>>>> logs, >>>>>> and that both your Graylog user's timezone and the messages' timezones >>>>>> are >>>>>> correct? >>>>>> >>>>>> I hope that helps. >>>>>> >>>>>> Regards, >>>>>> Edmundo >>>>>> >>>>>> -- >>>>>> Developer >>>>>> >>>>>> Tel.: +49 (0)40 609 452 077 >>>>>> Mobile: +49 (0)171 27 22 181 >>>>>> Mobile (US): +1 (713) 321 8126 >>>>>> Fax.: +49 (0)40 609 452 078 >>>>>> >>>>>> TORCH GmbH >>>>>> Steckelhörn 11 >>>>>> 20457 Hamburg >>>>>> Germany >>>>>> https://www.torch.sh/ >>>>>> >>>>>> Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 125175 >>>>>> Geschäftsführer: Lennart Koopmann (CEO) >>>>>> >>>>>> > On 20 Jan 2015, at 19:34, Zi Dvbelju <zidv...@gmail.com> wrote: >>>>>> > >>>>>> > Hey Team, >>>>>> > >>>>>> > Running into a strange issue with my Graylog2 setup. >>>>>> > >>>>>> > Search for anything with "last hour" returns no results. I can >>>>>> duplicate the same exact search with "last day" and see messages from >>>>>> within the last few seconds. >>>>>> > >>>>>> > Only searches with the "last hour" tag are returning nothing. >>>>>> > >>>>>> > v92.3 >>>>>> > >>>>>> > Thanks, >>>>>> > Zach >>>>>> > >>>>>> > -- >>>>>> > You received this message because you are subscribed to the Google >>>>>> Groups "graylog2" group. >>>>>> > To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to graylog2+u...@googlegroups.com. >>>>>> > For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d420fbf6-a8d4-4ed2-b578-e01ea57c6589%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
