Hi Jochen

There wasn't any error that pointed at this explicitly. This is
graylog-1.3.4 with ES-1.7 on CentOS7. There has been a general degradation
as my (single) test box grew to 5TB in size - I think I'm simply tickling
all the edge cases of performance issues and this is the consequence. I do
think ES could do with some major improvements - I continually saw graylog
reporting it couldn't push data into ES and yet ES reported no problems and
showed "green". And yet if I restarted ES (ie stop/start) it immediately
came up as "red" with 10,000+ UNASSIGNED shards - so there's a fairly
serious bug in there IMHO.

In any case I ended up deleting all the indexes that wouldn't move off
"UNASSIGNED" and basically ended up deleting most of my data, so I've
thrown the entire thing away (hey, it is a test box :-) and restarted with
the Beta graylog-2 with ES-2.3.1 and we'll see how that goes. This test box
is almost at an end - new hardware has finally arrived - so I think the
root cause of the issues will soon be resolved by actually having a
cluster/etc.

Thanks

Jason

On Wed, Apr 13, 2016 at 12:55 AM, Jochen Schalanda <[email protected]>
wrote:

> Hi Jason,
>
> what's your Graylog and Elasticsearch configuration? Are there any error
> messages in the logs of either Graylog or Elasticsearch?
>
> Cheers,
> Jochen
>
>
> On Monday, 11 April 2016 07:15:17 UTC+2, Jason Haar wrote:
>>
>> Hi there
>>
>> Over the weekend I installed the ES marvel diagnostics package and the
>> following day noticed that graylog was broken. Restarting ES showed 20,000
>> shards in an UNASSIGNED state. I disabled graylog-server (so there was no
>> new data flowing in) and watched over the next couple of hours as that
>> UNASSIGNED number dropped down to 0 (and GREEN). I then restarted ES and -
>> bam - back to 20,000 UNASSIGNED shards again
>>
>> I've now done three iterations of that - looks like it's completely
>> borked. There's over 5TB of data in there - how can I regain it?
>>
>> PS: I don't know if this has anything to do with marvel - it's just the
>> last change I made. The reason I installed it was because I have had ES
>> continually doing this kind of thing - but previously stopping graylog,
>> restarting ES and waiting would lead to a happy ES - but no longer.
>>
>> --
>> Cheers
>>
>> Jason Haar
>> Information Security Manager, Trimble Navigation Ltd.
>> Phone: +1 408 481 8171
>> PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
>>
> --
> You received this message because you are subscribed to the Google Groups
> "Graylog Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/graylog2/507dbfa3-0239-45ca-9799-2cae7db65418%40googlegroups.com
> <https://groups.google.com/d/msgid/graylog2/507dbfa3-0239-45ca-9799-2cae7db65418%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFChrg%2BL5zOhKu%2BHLJVtQjpZyjSQTe8HtsmS_%3DDJyBU%2B44UpTg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to