Lauren,
The below guide might be useful, the setting for worker nodes heap size is worker.childopts -XmX https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_storm-component-guide/content/config-storm-settings.html Regards, Aaron From: Laurens Vets [mailto:laur...@daemon.be] Sent: 18 June 2017 13:47 To: user@metron.apache.org Cc: Nick Allen <n...@nickallen.org> Subject: Re: Storm indexing out of memory Where and how exactly? I don't seem to find that 64MB setting in Ambari... On 2017-06-14 22:22, Nick Allen wrote: Yes, allocate more memory to your Storm workers. On Wed, Jun 14, 2017 at 1:37 PM, Laurens Vets <laur...@daemon.be<mailto:laur...@daemon.be>> wrote: Hello list, One of the Storm workers dies with the following error message: 2017-06-14 11:17:32.503 o.a.s.util [ERROR] Async loop died! java.lang.OutOfMemoryError: Java heap space at org.apache.kafka.common.utils.Utils.toArray(Utils.java:272) ~[stormjar.jar:?] at org.apache.kafka.common.utils.Utils.toArray(Utils.java:265) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:626) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.Fetcher.parseFetchedData(Fetcher.java:548) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:354) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1000) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938) ~[stormjar.jar:?] at org.apache.storm.kafka.spout.KafkaSpout.pollKafkaBroker(KafkaSpout.java:286) ~[stormjar.jar:?] at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:224) ~[stormjar.jar:?] at org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651) ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] 2017-06-14 11:17:40.058 o.a.s.d.executor [ERROR] java.lang.OutOfMemoryError: Java heap space at org.apache.kafka.common.utils.Utils.toArray(Utils.java:272) ~[stormjar.jar:?] at org.apache.kafka.common.utils.Utils.toArray(Utils.java:265) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.Fetcher.parseRecord(Fetcher.java:626) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.Fetcher.parseFetchedData(Fetcher.java:548) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:354) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1000) ~[stormjar.jar:?] at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938) ~[stormjar.jar:?] at org.apache.storm.kafka.spout.KafkaSpout.pollKafkaBroker(KafkaSpout.java:286) ~[stormjar.jar:?] at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:224) ~[stormjar.jar:?] at org.apache.storm.daemon.executor$fn__6505$fn__6520$fn__6551.invoke(executor.clj:651) ~[storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.util$async_loop$fn__554.invoke(util.clj:484) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] 2017-06-14 11:18:08.898 o.a.s.util [ERROR] Halting process: ("Worker died") As far as I can see, it's automatically restarted, but then dies again. Might this be due to only having 64MB of memory available: 2017-06-14 11:06:35.034 o.a.s.d.supervisor [INFO] Launching worker with command: '/usr/jdk64/jdk1.8.0_77/bin/java' '-cp' '/usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/current/storm-supervisor/conf:/data1/hadoop/storm/supervisor /stormdist/indexing-5-1497457977/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf' '-Xmx64m' '-Dlogfile.name=worker.log' '-Dstorm.home=/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm' '-Dworkers.artifacts=/var/log/storm/workers-artifacts' '-Dstorm.id=indexing-5-1497457977' '-Dworker.id=98d9fa65-b025-4924-9d4a-8f77324f9cf2' '-Dworker.port=6705' '-Dstorm.log.dir=/var/log/storm' '-Dlog4j.configurationFile=/usr/hdp/2.5.3.0-37/storm/log4j2/worker.xml' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' 'org.apache.storm.LogWriter' '/usr/jdk64/jdk1.8.0_77/bin/java' '-server' '-Xmx1024m' '-javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6705_JVM' '-Djava.library.path=/data1/hadoop/storm/supervisor/stormdist/indexing-5-1497457977/resources/Linux-amd64:/data1/hadoop/storm/supervisor/stormdist/indexing-5-1497457977/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib' '-Dlogfile.name=worker.log' '-Dstorm.home=/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm' '-Dworkers.artifacts=/var/log/storm/workers-artifacts' '-Dstorm.conf.file=' '-Dstorm.options=' '-Dstorm.log.dir=/var/log/storm' '-Djava.io.tmpdir=/data1/hadoop/storm/workers/98d9fa65-b025-4924-9d4a-8f77324f9cf2/tmp' '-Dlogging.sensitivity=S3' '-Dlog4j.configurationFile=/usr/hdp/2.5.3.0-37/storm/log4j2/worker.xml' '-DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector' '-Dstorm.id=indexing-5-1497457977' '-Dworker.id=98d9fa65-b025-4924-9d4a-8f77324f9cf2' '-Dworker.port=6705' '-cp' '/usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/MailScanner has detected a possible fraud attempt from "2.5.3" claiming to be MailScanner warning: numerical links are often malicious: 2.5.3.<http://2.5.3.>0-37/storm/lib/ambari-metrics-storm-sink.jar:/usr/hdp/current/storm-supervisor/conf:/data1/hadoop/storm/supervisor /stormdist/indexing-5-1497457977/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf' 'org.apache.storm.daemon.worker' 'indexing-5-1497457977' '9c90251f-54cc-42d7-8c4e-6f3bb2528752' '6705' '98d9fa65-b025-4924-9d4a-8f77324f9cf2' Any ideas on what might be happening?
