Hi All,
At some circumstance we have seen following exception while using geode-1.1.0
and trying to do Function Execution:
[severe 2018/05/22 01:32:31.964 EDT Node-7ccee253-7f00-4670-960a-3ba6e5a2365b
<Function Execution Processor22> tid=0x58] ack read exception
org.apache.geode.ToDataException: toData failed on DataSerializable class
org.apache.geode.internal.cache.UpdateOperation$UpdateMessage
at
org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2436)
at
org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1446)
at
org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:232)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:377)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:237)
at
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:603)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1684)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1875)
at
org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:82)
at
org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3416)
at
org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3453)
at
org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376)
at
org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:442)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at
org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:621)
at
org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1067)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: An Exception was thrown while
serializing.
at
org.apache.geode.internal.tcp.MsgStreamer.writeAsSerializedByteArray(MsgStreamer.java:930)
at
org.apache.geode.DataSerializer.writeObjectAsByteArray(DataSerializer.java:1305)
at
org.apache.geode.internal.cache.DistributedCacheOperation.writeValue(DistributedCacheOperation.java:124)
at
org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.toData(UpdateOperation.java:419)
at
org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2402)
... 61 more
Caused by: java.io.NotSerializableException: dataobject.PolicyInfo
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at
java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
At the same time, we have experienced OOM on our jvm and heapdump was
generated. Heapdump analysis showed that nearly half of the heap (16 GB of 32
GB) is consumed by SSLSocketImpl objects. Dominator Tree was topped by
'Function Execution Processor22' thread, which is mentioned above in the
stacktrace including serialization exception.
So the main question is whether having NotSerializableException exception in
sendMessage() flow could result to leak of not closed connections. Connection
closing calls are spread over the code and is hard to track them and I'm not
sure If org.apache.geode.internal.tcp.MsgStreamer#close is the one that is
responsible to close connections in this case. But if so, connections will not
be closed here as due to abovementioned NotSerializableException flushedBytes
will not be updated in org.apache.geode.internal.tcp.MsgStreamer#writeMessage.
Can someone assist us to understand if this exception can yield to the
SSLSocketImpl leak or we should look into another direction?
Thanks,
Vahram.
