Hello list, I've installed OS updates on my Metron 0.4.2 yesterday, restarted all nodes and now my indexing topology keeps crashing.
This is what I see in the Storm UI for the indexing topology: Topology stats: 10m 0s 1304380 1953520 12499.833 1320 3h 0m 0s 1304380 1953520 12499.833 1320 1d 0h 0m 0s 1304380 1953520 12499.833 1320 All time 1304380 1953520 12499.833 1320 Spouts: kafkaSpout 1 1 1299940 1949080 12499.833 1320 0 metron3 6702 java.lang.OutOfMemoryError: GC overhead limit exceeded at java.lang.Long.valueOf(Long.java:840) at org.apache.storm.kafka.spout.KafkaSpoutRetryExponentialBackoff$RetryEntryTimeStampComparator.compar Bolts: hdfsIndexingBolt 1 1 1800 1800 0.278 7.022 1820 38.633 1800 0 metron3 6702 java.lang.NullPointerException at org.apache.metron.writer.hdfs.SourceHandler.handle(SourceHandler.java:80) at org.apache.metron.writer.hdfs.HdfsWriter.write(HdfsWriter.java:113) at org.apache.metr Thur, 13 Sep 2018 07:35:02 indexingBolt 1 1 1320 1320 0.217 7.662 1300 47.815 1300 0 metron3 6702 java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOfRange(Arrays.java:3664) at java.lang.String.<init>(String.java:207) at org.json.simple.parser.Yylex.yytext(Yylex.jav Thur, 13 Sep 2018 07:37:33 When I check the Kafka topic, I can see that there's at least 3 million messages in the kafka indexing topic... I _suspect_ that the indexing topology tries to write those but fails, restarts, tries to write, fails, etc... Metron is currently not ingesting any additional messages, but also can't seem to index the current ones... Any idea on how to proceed?
