Re: Logstash failing to process messages

[Kernel Panic]

Just as an update, we upgraded the ElasticStack to 6.3.2 a couple of months
ago and Logstash has not crashed since, so whatever the problem was it
appears to have been fixed in the later release.

On Thu, 24 May 2018 at 13:47, Kernel Panic  wrote:

> Thanks for getting back to me, yes I suspect it has something to do with
> my filters though I've no idea which one it could be as I'm filtering on
> beats and syslog inputs. As a work around I've just added a cron command to
> restart Logstash every morning at 01:00, though obviously that means I'm
> losing non-beat events whilst it restarts. Please let me know if upgrading
> to the latest versions helps you, if it doesn't then perhaps a PR needs to
> be filed.
>
> On 24 May 2018 at 11:25, Benny Goemans  wrote:
>
>> I have seen the same issue. In my case however, I had about OOM caused by
>> parsing long grok patterns. I didn't have these in 5.3 either so I suspect
>> it's a memory leak somewhere.
>> I have since upgraded everything to 6.x and am waiting to see if the same
>> issue persists.
>>
>> Regards,
>> Benny Goemans
>>
>> On 23-05-2018 17:23, Kernel Panic wrote:
>>
>>> Hello, I'll just list the versions before I start:
>>>
>>> FreeBSD 11.1
>>>
>>> Logstash 6.23
>>> Elasticsearch 5.6.8
>>> Kibana 5.6.8
>>>
>>> The issue I'm having is that after a few days Logstash will stop
>>> processing
>>> any messages; I'm using the same config file that I used with Logstash
>>> 5.3.0 which worked without issue and was rock-solid. There's nothing in
>>> the
>>> Logstash log file apart from messages about a field in my Cisco logs
>>> being
>>> the wrong type and therefore failing to index, however this has always
>>> been
>>> the case. I have tried enabling the 'dead letter' feature in Logstash to
>>> process these Cisco logs but that just makes Logstash even more unstable.
>>>
>>> The Logstash service doesn't actually crash, it just stops processing
>>> messages and fails to respond to the restart command so I end up having
>>> to
>>> reboot the server. I should say though that Logstash continues to respond
>>> the the monitor API commands.
>>>
>>> I have tried updating all Logstash plugins however that has not fixed the
>>> issue.
>>>
>>> As I said, I never had any problems with Logstash 5.3.0 but the latest
>>> version (and version 5.6.8) just seem to become unstable after a few
>>> days.
>>>
>>> Any help is greatly appreciated.
>>> ___
>>> freebsd-ports@freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
>>> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
>>>
>>
>>
>>
>
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Logstash failing to process messages

[Kernel Panic]

Thanks for getting back to me, yes I suspect it has something to do with my
filters though I've no idea which one it could be as I'm filtering on beats
and syslog inputs. As a work around I've just added a cron command to
restart Logstash every morning at 01:00, though obviously that means I'm
losing non-beat events whilst it restarts. Please let me know if upgrading
to the latest versions helps you, if it doesn't then perhaps a PR needs to
be filed.

On 24 May 2018 at 11:25, Benny Goemans  wrote:

> I have seen the same issue. In my case however, I had about OOM caused by
> parsing long grok patterns. I didn't have these in 5.3 either so I suspect
> it's a memory leak somewhere.
> I have since upgraded everything to 6.x and am waiting to see if the same
> issue persists.
>
> Regards,
> Benny Goemans
>
> On 23-05-2018 17:23, Kernel Panic wrote:
>
>> Hello, I'll just list the versions before I start:
>>
>> FreeBSD 11.1
>>
>> Logstash 6.23
>> Elasticsearch 5.6.8
>> Kibana 5.6.8
>>
>> The issue I'm having is that after a few days Logstash will stop
>> processing
>> any messages; I'm using the same config file that I used with Logstash
>> 5.3.0 which worked without issue and was rock-solid. There's nothing in
>> the
>> Logstash log file apart from messages about a field in my Cisco logs being
>> the wrong type and therefore failing to index, however this has always
>> been
>> the case. I have tried enabling the 'dead letter' feature in Logstash to
>> process these Cisco logs but that just makes Logstash even more unstable.
>>
>> The Logstash service doesn't actually crash, it just stops processing
>> messages and fails to respond to the restart command so I end up having to
>> reboot the server. I should say though that Logstash continues to respond
>> the the monitor API commands.
>>
>> I have tried updating all Logstash plugins however that has not fixed the
>> issue.
>>
>> As I said, I never had any problems with Logstash 5.3.0 but the latest
>> version (and version 5.6.8) just seem to become unstable after a few days.
>>
>> Any help is greatly appreciated.
>> ___
>> freebsd-ports@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-ports
>> To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"
>>
>
>
>
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Re: Logstash failing to process messages

[Benny Goemans]

I have seen the same issue. In my case however, I had about OOM caused 
by parsing long grok patterns. I didn't have these in 5.3 either so I 
suspect it's a memory leak somewhere.
I have since upgraded everything to 6.x and am waiting to see if the 
same issue persists.


Regards,
Benny Goemans

On 23-05-2018 17:23, Kernel Panic wrote:

Hello, I'll just list the versions before I start:

FreeBSD 11.1

Logstash 6.23
Elasticsearch 5.6.8
Kibana 5.6.8

The issue I'm having is that after a few days Logstash will stop processing
any messages; I'm using the same config file that I used with Logstash
5.3.0 which worked without issue and was rock-solid. There's nothing in the
Logstash log file apart from messages about a field in my Cisco logs being
the wrong type and therefore failing to index, however this has always been
the case. I have tried enabling the 'dead letter' feature in Logstash to
process these Cisco logs but that just makes Logstash even more unstable.

The Logstash service doesn't actually crash, it just stops processing
messages and fails to respond to the restart command so I end up having to
reboot the server. I should say though that Logstash continues to respond
the the monitor API commands.

I have tried updating all Logstash plugins however that has not fixed the
issue.

As I said, I never had any problems with Logstash 5.3.0 but the latest
version (and version 5.6.8) just seem to become unstable after a few days.

Any help is greatly appreciated.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"



___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Logstash failing to process messages

[Kernel Panic]

Hello, I'll just list the versions before I start:

FreeBSD 11.1

Logstash 6.23
Elasticsearch 5.6.8
Kibana 5.6.8

The issue I'm having is that after a few days Logstash will stop processing
any messages; I'm using the same config file that I used with Logstash
5.3.0 which worked without issue and was rock-solid. There's nothing in the
Logstash log file apart from messages about a field in my Cisco logs being
the wrong type and therefore failing to index, however this has always been
the case. I have tried enabling the 'dead letter' feature in Logstash to
process these Cisco logs but that just makes Logstash even more unstable.

The Logstash service doesn't actually crash, it just stops processing
messages and fails to respond to the restart command so I end up having to
reboot the server. I should say though that Logstash continues to respond
the the monitor API commands.

I have tried updating all Logstash plugins however that has not fixed the
issue.

As I said, I never had any problems with Logstash 5.3.0 but the latest
version (and version 5.6.8) just seem to become unstable after a few days.

Any help is greatly appreciated.
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"