Hi John/Bruce,
Typo in email was by me but code does not have that.
Here is the detailed logs explaining what might have happened and looks like a
CRITICAL BUG only in both (1.6.0) and (1.8.0) as well.
[Attached email thread for GEODE 1.8.0 as well]
ISSUES:
Ø 1 common scenario which fails – Member is unable to join distributed system
back after FORCE DICONNECT
Ø Why does PDXSerializer play a role while boot strapping cache member? (Note
: No disk persistence is being used but only ReflectionBasedAutoSerializer)
Ø GMS is throwing NullPointer exception in this situation
Ø Same error messages and stack trace for GMS is being printed once with
fatal/error/warn log levels which does not look consistent.
[Pending] : Dharam to find out if that’s the same issue with
MappingPdxSerializer as suggested by John
Current logs both for GEODE – 1.6.0 & GEODE 1.8.0
[warn 2019-02-01 00:05:29.000 EST <StatSampler> statistics.HostStatSampler]
Statistics sampling thread detected a wakeup delay of 16617 ms, indicating a
possible resource issue. Check the GC, memory, and CPU statistics.
[fatal 2019-02-01 00:05:29.937 EST <unicast receiver,iaasn00005748-49024>
gms.Services] Membership service failure: Failed to acknowledge a new
membership view and then failed tcp/ip connection attempt
org.apache.geode.ForcedDisconnectException: Failed to acknowledge a new
membership view and then failed tcp/ip connection attempt
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2503)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:1049)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveRequest(GMSJoinLeave.java:654)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1810)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1301)
at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
at org.jgroups.JChannel.up(JChannel.java:741)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1077)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:792)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:433)
at
org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:73)
at
org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1876)
at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1789)
at org.jgroups.protocols.TP.receive(TP.java:1714)
at
org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:152)
at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
at java.lang.Thread.run(Thread.java:745)
[error 2019-02-01 00:06:35.651 EST <ReconnectThread> cache.GemFireCacheImpl]
org.apache.geode.cache.CacheClosedException: Could not PDX serialize because
the cache was closed
[warn 2019-02-01 00:06:35.652 EST <ReconnectThread>
internal.InternalDistributedSystem] Exception occurred while trying to create
the cache during reconnect
org.apache.geode.cache.CacheClosedException: Could not PDX serialize because
the cache was closed
at
org.apache.geode.pdx.internal.TypeRegistry.getPdxSerializer(TypeRegistry.java:317)
at
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1648)
at
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1548)
at
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2200)
at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2952)
at
org.apache.geode.internal.cache.MemberFunctionStreamingMessage.toData(MemberFunctionStreamingMessage.java:315)
at
org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2398)
at
org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1517)
at
org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:234)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:394)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:251)
at
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:616)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1686)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1864)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2860)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2780)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2819)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1523)
at
org.apache.geode.internal.cache.execute.StreamingFunctionOperation.getFunctionResultFrom(StreamingFunctionOperation.java:107)
at
org.apache.geode.internal.cache.execute.MemberFunctionExecutor.executeFunction(MemberFunctionExecutor.java:151)
at
org.apache.geode.internal.cache.execute.MemberFunctionExecutor.executeFunction(MemberFunctionExecutor.java:189)
at
org.apache.geode.internal.cache.execute.AbstractExecution.execute(AbstractExecution.java:392)
at
org.apache.geode.internal.cache.ClusterConfigurationLoader.requestConfigurationFromOneLocator(ClusterConfigurationLoader.java:312)
at
org.apache.geode.internal.cache.ClusterConfigurationLoader.requestConfigurationFromLocators(ClusterConfigurationLoader.java:282)
at
org.apache.geode.internal.cache.GemFireCacheImpl.requestSharedConfiguration(GemFireCacheImpl.java:1066)
at
org.apache.geode.internal.cache.GemFireCacheImpl.<init>(GemFireCacheImpl.java:857)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:794)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:773)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2765)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531)
at java.lang.Thread.run(Thread.java:745)
[fatal 2019-02-01 00:07:35.810 EST <ReconnectThread> gms.Services] Unexpected
exception while booting membership services
java.lang.NullPointerException
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:483)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:361)
at
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:146)
at
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:105)
at
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:771)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:889)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:533)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:769)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:362)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:348)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:342)
at
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:215)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2704)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531)
at java.lang.Thread.run(Thread.java:745)
[error 2019-02-01 00:07:35.810 EST <ReconnectThread> gms.Services] Unexpected
problem starting up membership services
java.lang.NullPointerException
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:483)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:361)
at
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:146)
at
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:105)
at
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:771)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:889)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:533)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:769)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:362)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:348)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:342)
at
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:215)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2704)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531)
at java.lang.Thread.run(Thread.java:745)
Thanks,
Dharam
From: John Blum [mailto:[email protected]]
Sent: Tuesday, January 29, 2019 3:49 AM
To: [email protected]
Subject: Re: NullPointerException with GMS services (Geode 1.6.0)
Although, there is an apparent problem with your Locator configuration
property, which can/will lead to an NPE thrown by Geode in this case (as I
reported in
https://issues.apache.org/jira/browse/GEODE-6153<https://secureweb.jpmchase.net/readonly/https:/issues.apache.org/jira/browse/GEODE-6153>,
which is not exactly the same as your problem), as Bruce points out, this is
occurring during reconnect.
If the default value of the Locator property had been used (which it apparently
wasn't in this case), then the node would have failed even on initial startup.
So, something else is going wrong here.
On Mon, Jan 28, 2019 at 10:32 AM Bruce Schuchardt
<[email protected]<mailto:[email protected]>> wrote:
These NPEs are probably not your main problem since their occurring in a
Reconnect Thread. Something bad happened when you tried to start these caches.
They joined the cluster and then were kicked out for some reason. The log
file for that process will likely tell you what the problem was. Post it or PM
it to me if you want help.
On 1/27/19 10:56 PM, Thacker, Dharam wrote:
java.lang.NullPointerException
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:468)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:355)
at
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:157)
at
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:106)
at
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:1027)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:1061)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:554)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:763)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:355)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:341)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:335)
at
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:211)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2736)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2560)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1041)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:4033)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1554)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$1(GMSMembershipManager.java:2561)
at java.lang.Thread.run(Thread.java:745)
--
-John
john.blum10101 (skype)
This message is confidential and subject to terms at:
https://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal
privilege, viruses and monitoring of electronic messages. If you are not the
intended recipient, please delete this message and notify the sender
immediately. Any unauthorized use is strictly prohibited.
--- Begin Message ---
Although, honestly, I would not use the ReflectionBasedAutoSerializer. In a
Spring context, I would use SDG's MappingPdxSerializer since it automatically
excludes all java.*, com.gemstone.gemfire.*, org.apache.geode.* and
org.springframework.* classes. It can do so easily because it has a much more
robust exclude/include filtering mechanism, by using the Java
java.util.function.Predicate interface. And, it is simple to enable, simply by
doing the following...
@SpringBootApplication
@EnablePdx
...
class MySpringGemFireApplication { ... }
You can read full details about the MappingPdxSerializer here...
https://docs.spring.io/spring-data/gemfire/docs/current/reference/html/#mapping.pdx-serializer<https://secureweb.jpmchase.net/readonly/https://docs.spring.io/spring-data/gemfire/docs/current/reference/html/#mapping.pdx-serializer>
And, specifically about Class Type Filtering here...
https://docs.spring.io/spring-data/gemfire/docs/current/reference/html/#mapping.pdx-serializer.type-filtering<https://secureweb.jpmchase.net/readonly/https://docs.spring.io/spring-data/gemfire/docs/current/reference/html/#mapping.pdx-serializer.type-filtering>
Regards,
John
On Wed, Jan 30, 2019 at 9:17 AM John Blum
<[email protected]<mailto:[email protected]>> wrote:
It's no different than configuring the ReflectionBasedAutoSerializer without
Spring.
On Wed, Jan 30, 2019 at 9:12 AM Bruce Schuchardt
<[email protected]<mailto:[email protected]>> wrote:
I do wonder, though, if your ReflectionBasedAutoSerializer is being
configured with a string that is including Geode-internal classes. You have
"x.y.z.*" in your example but you probably had something different in your
actual test run I think. If that's the case you ought to restrict the pattern
a bit more or perhaps John has some advice concerning how to configure
serialization when using Spring.
On 1/28/19 11:49 PM, Thacker, Dharam wrote:
Hi Bruce/John,
I am getting below exceptions with GEODE 1.8.0. I remember this issue
was raised by me in past with GEODE 1.1.1 and it was fixed but looks like it’s
throwing different exceptions now.
I had reported enterprise gemfire case as well In past where ever we
use gemfire for critical apps,
https://community.pivotal.io/s/case/5000e00001TPiZwAAL/authenticationrequiredexception-on-force-disconnection<https://secureweb.jpmchase.net/readonly/https://community.pivotal.io/s/case/5000e00001TPiZwAAL/authenticationrequiredexception-on-force-disconnection>
Config:
spring.data.gemfire.pdx.read-serialized = true
spring.data.gemfire.pdx.serializer-bean-name = pdxSerializer
@Bean
public ReflectionBasedAutoSerializer pdxSerializer() {
return new ReflectionBasedAutoSerializer("x.y.z.*");
}
Steps to reproduce:
1. Start locators
2. Start server1 [pid1]
3. Start server2 [pid2]
4. kill –STOP pid2 (Wait until server2 is thrown out of
distributed system – ForceDisconnect)
5. kill –CONT pid2 (Let the server2 bootstrap and re-join
system) – But it fails
Exception Stack Trace:
[fatal 2019-01-29 02:33:09.106 EST <unicast
receiver,iaasn00001746-19619> gms.Services] Membership service failure: Member
isn't responding to heartbeat requests
org.apache.geode.ForcedDisconnectException: Member isn't responding to
heartbeat requests
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2503)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:1049)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveRequest(GMSJoinLeave.java:654)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1810)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1301)
at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
at org.jgroups.JChannel.up(JChannel.java:741)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at
org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1077)
at
org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:792)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:433)
at
org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:73)
at
org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
at
org.jgroups.protocols.TP<https://secureweb.jpmchase.net/readonly/http://org.jgroups.protocols.TP>.passMessageUp(TP.java:1658)
at
org.jgroups.protocols.TP<https://secureweb.jpmchase.net/readonly/http://org.jgroups.protocols.TP>$SingleMessageHandler.run(TP.java:1876)
at
org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
at
org.jgroups.protocols.TP<https://secureweb.jpmchase.net/readonly/http://org.jgroups.protocols.TP>.handleSingleMessage(TP.java:1789)
at
org.jgroups.protocols.TP<https://secureweb.jpmchase.net/readonly/http://org.jgroups.protocols.TP>.receive(TP.java:1714)
at
org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:152)
at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
at java.lang.Thread.run(Thread.java:745)
[error 2019-01-29 02:34:09.715 EST <ReconnectThread>
cache.GemFireCacheImpl] org.apache.geode.cache.CacheClosedException: Could not
PDX serialize because the cache was closed
[warn 2019-01-29 02:34:09.715 EST <ReconnectThread>
internal.InternalDistributedSystem] Exception occurred while trying to create
the cache during reconnect
org.apache.geode.cache.CacheClosedException: Could not PDX serialize
because the cache was closed
at
org.apache.geode.pdx.internal.TypeRegistry.getPdxSerializer(TypeRegistry.java:317)
at
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1648)
at
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1548)
at
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2200)
at
org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2952)
at
org.apache.geode.internal.cache.MemberFunctionStreamingMessage.toData(MemberFunctionStreamingMessage.java:315)
at
org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2398)
at
org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1517)
at
org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:234)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:394)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:251)
at
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:616)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1686)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1864)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2860)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2780)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2819)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1523)
at
org.apache.geode.internal.cache.execute.StreamingFunctionOperation.getFunctionResultFrom(StreamingFunctionOperation.java:107)
at
org.apache.geode.internal.cache.execute.MemberFunctionExecutor.executeFunction(MemberFunctionExecutor.java:151)
at
org.apache.geode.internal.cache.execute.MemberFunctionExecutor.executeFunction(MemberFunctionExecutor.java:189)
at
org.apache.geode.internal.cache.execute.AbstractExecution.execute(AbstractExecution.java:392)
at
org.apache.geode.internal.cache.ClusterConfigurationLoader.requestConfigurationFromOneLocator(ClusterConfigurationLoader.java:312)
at
org.apache.geode.internal.cache.ClusterConfigurationLoader.requestConfigurationFromLocators(ClusterConfigurationLoader.java:282)
at
org.apache.geode.internal.cache.GemFireCacheImpl.requestSharedConfiguration(GemFireCacheImpl.java:1066)
at
org.apache.geode.internal.cache.GemFireCacheImpl.<init>(GemFireCacheImpl.java:857)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:794)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:773)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2765)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531)
at java.lang.Thread.run(Thread.java:745)
Thanks,
Dharam
This message is confidential and subject to terms at:
https://www.jpmorgan.com/emaildisclaimer<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.jpmorgan.com_emaildisclaimer&d=DwMFAg&c=lnl9vOaLMzsy2niBC8-h_K-7QJuNJEsFrzdndhuJ3Sw&r=JEKigqAv3f2lWHmA02pq9MDT5naXLkEStB4d4n0NQmk&m=Fj5zKecHrfN1_Vk8of9RJQ7f3xQADDRQoQwya9PZ5z4&s=BwHIHPOpkkC7f6dgcLl2hLEl5ojYxNdSoGSYWg-dBhk&e=>
including on confidentiality, legal privilege, viruses and monitoring of
electronic messages. If you are not the intended recipient, please delete this
message and notify the sender immediately. Any unauthorized use is strictly
prohibited.
--
-John
john.blum10101 (skype)
--
-John
john.blum10101 (skype)
--- End Message ---
