Hi Bruce, Thank you for the feedback. Will keep an eye on this after Geode version update in our end.
Best Regards, Vahram. From: Bruce Schuchardt <[email protected]> Sent: Thursday, January 31, 2019 3:02 AM To: [email protected] Subject: Re: Numerous SSLSocketImpl objects after NotSerializableException in message sending flow Hi Vahram, The socket leak was fixed in v1.6 https://issues.apache.org/jira/browse/GEODE-3563<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FGEODE-3563&data=02%7C01%7Cvaharonyan%40vmware.com%7C04b42bd0388a4f8eb37a08d68706fd8a%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636844861427040537&sdata=S6LuHtzICKzkMns6KMkgVaSl8UQzGs730mtU8N2aZJA%3D&reserved=0> On 1/30/19 7:12 AM, Vahram Aharonyan wrote: Hi All, At some circumstance we have seen following exception while using geode-1.1.0 and trying to do Function Execution: [severe 2018/05/22 01:32:31.964 EDT Node-7ccee253-7f00-4670-960a-3ba6e5a2365b <Function Execution Processor22> tid=0x58] ack read exception org.apache.geode.ToDataException: toData failed on DataSerializable class org.apache.geode.internal.cache.UpdateOperation$UpdateMessage at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2436) at org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1446) at org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:232) at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:377) at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:237) at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:603) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1684) at org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1875) at org.apache.geode.distributed.internal.DistributionChannel.send(DistributionChannel.java:82) at org.apache.geode.distributed.internal.DistributionManager.sendOutgoing(DistributionManager.java:3416) at org.apache.geode.distributed.internal.DistributionManager.sendMessage(DistributionManager.java:3453) at org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:376) at org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:621) at org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1067) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException: An Exception was thrown while serializing. at org.apache.geode.internal.tcp.MsgStreamer.writeAsSerializedByteArray(MsgStreamer.java:930) at org.apache.geode.DataSerializer.writeObjectAsByteArray(DataSerializer.java:1305) at org.apache.geode.internal.cache.DistributedCacheOperation.writeValue(DistributedCacheOperation.java:124) at org.apache.geode.internal.cache.UpdateOperation$UpdateMessage.toData(UpdateOperation.java:419) at org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2402) ... 61 more Caused by: java.io.NotSerializableException: dataobject.PolicyInfo at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) At the same time, we have experienced OOM on our jvm and heapdump was generated. Heapdump analysis showed that nearly half of the heap (16 GB of 32 GB) is consumed by SSLSocketImpl objects. Dominator Tree was topped by 'Function Execution Processor22' thread, which is mentioned above in the stacktrace including serialization exception. So the main question is whether having NotSerializableException exception in sendMessage() flow could result to leak of not closed connections. Connection closing calls are spread over the code and is hard to track them and I'm not sure If org.apache.geode.internal.tcp.MsgStreamer#close is the one that is responsible to close connections in this case. But if so, connections will not be closed here as due to abovementioned NotSerializableException flushedBytes will not be updated in org.apache.geode.internal.tcp.MsgStreamer#writeMessage. Can someone assist us to understand if this exception can yield to the SSLSocketImpl leak or we should look into another direction? Thanks, Vahram.
