Nick Allen created METRON-1551: ---------------------------------- Summary: Profiler Should Not Use Java Serialization Key: METRON-1551 URL: https://issues.apache.org/jira/browse/METRON-1551 Project: Metron Issue Type: Bug Reporter: Nick Allen
When running the Profiler in a topology where serialization occurs, the following error happens. This can occur when the number of workers is greater than 1. The topology should not be using Java serialization for serializing tuple values as this will negatively impact performance. {code} 2018-05-09 10:48:35.136 o.a.s.d.executor [ERROR] java.lang.RuntimeException: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$consume_loop_STAR_$fn__7183.invoke(disruptor.clj:83) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] Caused by: java.lang.RuntimeException: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:41) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] ... 6 more Caused by: java.io.NotSerializableException: org.apache.metron.common.configuration.profiler.ProfileResult at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) ~[?:1.8.0_162] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) ~[?:1.8.0_162] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) ~[?:1.8.0_162] at org.apache.storm.serialization.SerializableSerializer.write(SerializableSerializer.java:38) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:628) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:100) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.write(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:534) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.KryoValuesSerializer.serializeInto(KryoValuesSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.serialization.KryoTupleSerializer.serialize(KryoTupleSerializer.java:44) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.worker$mk_transfer_fn$transfer_fn__7805.invoke(worker.clj:193) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__7430.invoke(executor.clj:309) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.disruptor$clojure_handler$reify__7166.onEvent(disruptor.clj:40) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) ~[storm-core-1.1.0.2.6.4.0-91.jar:1.1.0.2.6.4.0-91] ... 6 more {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)