It seemed to hold up a little bit better but I ended up getting the same errors.
On Thu, Aug 24, 2017 at 11:35 AM, Nick Carenza < nick.care...@thecontrolgroup.com> wrote: > Okay, thank you Andre! > > - Nick > > On Wed, Aug 23, 2017 at 4:03 PM, Andre <andre-li...@fucs.org> wrote: > >> Nick, >> >> This is a bug (NIFI-3753). I am also able to confirm there is a >> workaround: just change the compression settings to a lower compression >> level (I believe anything between none and 3 should work) and/or reduce the >> number of messages batched on a single beats payload. >> >> Cheers >> >> On 23 Aug 2017 09:11, "Nick Carenza" <nick.care...@thecontrolgroup.com> >> wrote: >> >>> I am trying to cut Logstash out of my stack because i have moved all of >>> its functionality into Nifi but I haven't been able to get Filebeat to play >>> nicely with the ListenBeats processor so I have to keep Logstash around >>> just to forward messages to Nifi. >>> >>> I am using latest stable 5.5.2 but the ListenBeats processor throws >>> these errors and gets yielded for 30s in an endless loop. >>> >>> 2017-08-22 22:26:53,183 ERROR [pool-812781-thread-29] >>>> o.a.nifi.processors.beats.ListenBeats >>>> ListenBeats[id=9e14179f-101f-1159-4cf1-738e9a425a6e] >>>> Error reading Beats frames due to Error decoding Beats frame: Error >>>> decompressing frame: Unexpected end of ZLIB input stream: >>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error >>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB >>>> input stream >>>> >>>> org.apache.nifi.processors.beats.frame.BeatsFrameException: Error >>>> decoding Beats frame: Error decompressing frame: Unexpected end of ZLIB >>>> input stream >>>> >>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process( >>>> BeatsDecoder.java:123) >>>> >>>> at org.apache.nifi.processors.beats.handler.BeatsSocketChannelH >>>> andler.processBuffer(BeatsSocketChannelHandler.java:71) >>>> >>>> Caused by: org.apache.nifi.processors.beats.frame.BeatsFrameException: >>>> Error decompressing frame: Unexpected end of ZLIB input stream >>>> >>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP >>>> AYLOAD(BeatsDecoder.java:292) >>>> >>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.process( >>>> BeatsDecoder.java:103) >>>> >>>> at org.apache.nifi.processors.beats.frame.BeatsDecoder.processP >>>> AYLOAD(BeatsDecoder.java:277) >>>> >>> while filebeat is logging >>> >>> >>> >>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> sync.go:85: Failed to publish events caused by: read tcp >>>> localhost:45416->remotehost:5044: i/o timeout >>> >>> Aug 22 22:28:57 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> single.go:91: Error publishing events (retrying): read tcp >>>> localhost:45416-> remotehost:5044: i/o timeout >>> >>> Aug 22 22:29:26 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> metrics.go:39: Non-zero metrics in the last 30s: >>>> libbeat.logstash.call_count.PublishEvents=1 >>>> libbeat.logstash.publish.read_bytes=186 >>>> libbeat.logstash.publish.read_errors=1 >>>> libbeat.logstash.publish.write_bytes=31399 >>>> libbeat.logstash.published_and_acked_events=98 >>>> libbeat.logstash.published_but_not_acked_events=1950 >>> >>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> sync.go:85: Failed to publish events caused by: read tcp >>>> localhost:49790-> remotehost:5044: i/o timeout >>> >>> Aug 22 22:29:28 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> single.go:91: Error publishing events (retrying): read tcp >>>> localhost:49790-> remotehost:5044: i/o timeout >>> >>> Aug 22 22:29:56 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> metrics.go:39: Non-zero metrics in the last 30s: >>>> libbeat.logstash.call_count.PublishEvents=1 >>>> libbeat.logstash.publish.read_bytes=138 >>>> libbeat.logstash.publish.read_errors=1 >>>> libbeat.logstash.publish.write_bytes=27456 >>>> libbeat.logstash.published_and_acked_events=31 >>>> libbeat.logstash.published_but_not_acked_events=1919 >>> >>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> sync.go:85: Failed to publish events caused by: read tcp >>>> localhost:54060-> remotehost:5044: i/o timeout >>> >>> Aug 22 22:29:59 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> single.go:91: Error publishing events (retrying): read tcp >>>> localhost:54060-> remotehost:5044: i/o timeout >>> >>> Aug 22 22:30:26 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> metrics.go:39: Non-zero metrics in the last 30s: >>>> libbeat.logstash.call_count.PublishEvents=1 >>>> libbeat.logstash.publish.read_bytes=492 >>>> libbeat.logstash.publish.read_errors=1 >>>> libbeat.logstash.publish.write_bytes=63110 >>>> libbeat.logstash.published_and_acked_events=23 >>>> libbeat.logstash.published_but_not_acked_events=1896 >>> >>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> sync.go:85: Failed to publish events caused by: read tcp >>>> localhost:58359-> remotehost:5044: i/o timeout >>> >>> Aug 22 22:30:30 app-server /usr/share/filebeat/bin/filebeat[15883]: >>>> single.go:91: Error publishing events (retrying): read tcp >>>> localhost:58359-> remotehost:5044: i/o timeout >>> >>> >>> has anyone seen something similar or have any idea what might be going >>> on here? >>> >>> my processor params are: >>> >>> >>> Port >>> 5044 >>> >>> Receive Buffer Size >>> 10 MB >>> >>> Max Size of Message Queue >>> 10000 >>> >>> Max Size of Socket Buffer >>> 8 MB >>> >>> Character Set >>> UTF-8 >>> >>> Max Batch Size >>> 1 >>> >>> Batching Message Delimiter >>> \n >>> >>> Max Number of TCP Connections >>> 100 >>> >>> SSL Context Service >>> No value set >>> >>> Thanks, >>> Nick >>> >> >
