Two questions: 1. How much memory are you giving the workers for the indexing topology? 2. how large are the messages you're sending through?
On Thu, Sep 13, 2018 at 2:00 PM Vets, Laurens <laur...@daemon.be> wrote: > Hello list, > > I've installed OS updates on my Metron 0.4.2 yesterday, restarted all > nodes and now my indexing topology keeps crashing. > > This is what I see in the Storm UI for the indexing topology: > > Topology stats: > 10m 0s 1304380 1953520 12499.833 1320 > 3h 0m 0s 1304380 1953520 12499.833 1320 > 1d 0h 0m 0s 1304380 1953520 12499.833 1320 > All time 1304380 1953520 12499.833 1320 > > Spouts: > kafkaSpout 1 1 1299940 1949080 12499.833 1320 0 > metron3 6702 java.lang.OutOfMemoryError: GC overhead limit > exceeded at java.lang.Long.valueOf(Long.java:840) at > > org.apache.storm.kafka.spout.KafkaSpoutRetryExponentialBackoff$RetryEntryTimeStampComparator.compar > > Bolts: > hdfsIndexingBolt 1 1 1800 1800 0.278 7.022 1820 > 38.633 1800 0 metron3 6702 java.lang.NullPointerException > at > org.apache.metron.writer.hdfs.SourceHandler.handle(SourceHandler.java:80) > at org.apache.metron.writer.hdfs.HdfsWriter.write(HdfsWriter.java:113) > at org.apache.metr Thur, 13 Sep 2018 07:35:02 > indexingBolt 1 1 1320 1320 0.217 7.662 1300 > 47.815 1300 0 metron3 6702 java.lang.OutOfMemoryError: GC > overhead limit exceeded at > java.util.Arrays.copyOfRange(Arrays.java:3664) at > java.lang.String.<init>(String.java:207) at > org.json.simple.parser.Yylex.yytext(Yylex.jav Thur, 13 Sep 2018 07:37:33 > > When I check the Kafka topic, I can see that there's at least 3 million > messages in the kafka indexing topic... I _suspect_ that the indexing > topology tries to write those but fails, restarts, tries to write, > fails, etc... Metron is currently not ingesting any additional messages, > but also can't seem to index the current ones... > > Any idea on how to proceed? > >
