John/Bruce,
It’s the same behavior even with “MappingPdxSerializer” for “Force
Reconnect” L
I would really appreciate if we can address into GEODE-1.9.0 release J
It tries for 6 attempts to reconnect distributed system. In first
attempt it still complains for “pdxSerializer” issue and in later
attempts it always gives,
[info 2019-02-01 18:01:33.878 IST <ReconnectThread>
internal.ClusterDistributionManager tid=142] Serial Queue info :
THROTTLE_PERCENT: 0.75 SERIAL_QUEUE_BYTE_LIMIT :41943040
SERIAL_QUEUE_THROTTLE :31457280 TOTAL_SERIAL_QUEUE_BYTE_LIMIT
:83886080 TOTAL_SERIAL_QUEUE_THROTTLE :31457280
SERIAL_QUEUE_SIZE_LIMIT :20000 SERIAL_QUEUE_SIZE_THROTTLE :15000
[info 2019-02-01 18:01:33.883 IST <ReconnectThread> gms.Services
tid=142] Starting membership services
[fatal 2019-02-01 18:01:33.883 IST <ReconnectThread> gms.Services
tid=142] Unexpected exception while booting membership services
_java.lang.NullPointerException_
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(_JGroupsMessenger.java:483_)
…
@EnablePdx(readSerialized = *true*, serializerBeanName =
"mappingPdxSerializer")
@Bean
public MappingPdxSerializer mappingPdxSerializer() {
MappingPdxSerializer mappingPdxSerializer = new MappingPdxSerializer();
Map<Class<?>, EntityInstantiator> customInstantiators = new HashMap<>();
EntityInstantiator instantiator = ReflectionEntityInstantiator.INSTANCE;
customInstantiators.put(A1.class, instantiator);
customInstantiators.put(A1Audit.class, instantiator);
customInstantiators.put(B1.class, instantiator);
customInstantiators.put(B1Audit.class, instantiator);
return mappingPdxSerializer;
}
1 more issue with MappingPdxSerializer,
Class A1 àregion A1
a1Reposiroty.findAll() àThis should give me List<A1> as a standard
contact.
ØWorks with ReflectionBasedAutoSerializer
ØFails with “MappingPdxSerializer” (It gives List<PdxInstanceImp> and
internally you have to call pdxInstance.getObject() to get real
instance of A1)
Thanks,
Dharam
*From:*Thacker, Dharam
*Sent:* Friday, February 01, 2019 12:58 PM
*To:* '[email protected] <mailto:[email protected]>'
<[email protected] <mailto:[email protected]>>
*Subject:* RE: NullPointerException with GMS services (Geode 1.6.0)
Hi John/Bruce,
Typo in email was by me but code does not have that.
Here is the detailed logs explaining what might have happened and
looks like a CRITICAL BUG only in both (1.6.0) and (1.8.0) as well.
[Attached email thread for GEODE 1.8.0 as well]
*_ISSUES:_*
Ø1 common scenario which fails – Member is unable to join distributed
system back after FORCE DICONNECT
ØWhy does PDXSerializer play a role while boot strapping cache member?
(Note : No disk persistence is being used but only
ReflectionBasedAutoSerializer)
ØGMS is throwing NullPointer exception in this situation
ØSame error messages and stack trace for GMS is being printed once
with fatal/error/warn log levels which does not look consistent.
[Pending] : Dharam to find out if that’s the same issue with
MappingPdxSerializer as suggested by John
Current logs both for GEODE – 1.6.0 & GEODE 1.8.0
[warn 2019-02-01 00:05:29.000 EST <StatSampler>
statistics.HostStatSampler] Statistics sampling thread detected a
wakeup delay of 16617 ms, indicating a possible resource issue. Check
the GC, memory, and CPU statistics.
[fatal 2019-02-01 00:05:29.937 EST <unicast
receiver,iaasn00005748-49024> gms.Services] Membership service
failure: Failed to acknowledge a new membership view and then failed
tcp/ip connection attempt
org.apache.geode.ForcedDisconnectException: Failed to acknowledge a
new membership view and then failed tcp/ip connection attempt
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.forceDisconnect(GMSMembershipManager.java:2503)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.forceDisconnect(GMSJoinLeave.java:1049)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processRemoveRequest(GMSJoinLeave.java:654)
at
org.apache.geode.distributed.internal.membership.gms.membership.GMSJoinLeave.processMessage(GMSJoinLeave.java:1810)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger$JGroupsReceiver.receive(JGroupsMessenger.java:1301)
at org.jgroups.JChannel.invokeCallback(JChannel.java:816)
at org.jgroups.JChannel.up(JChannel.java:741)
at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1030)
at org.jgroups.protocols.FRAG2.up(FRAG2.java:165)
at org.jgroups.protocols.FlowControl.up(FlowControl.java:390)
at org.jgroups.protocols.UNICAST3.deliverMessage(UNICAST3.java:1077)
at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:792)
at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:433)
at
org.apache.geode.distributed.internal.membership.gms.messenger.StatRecorder.up(StatRecorder.java:73)
at
org.apache.geode.distributed.internal.membership.gms.messenger.AddressManager.up(AddressManager.java:72)
at org.jgroups.protocols.TP.passMessageUp(TP.java:1658)
at org.jgroups.protocols.TP$SingleMessageHandler.run(TP.java:1876)
at org.jgroups.util.DirectExecutor.execute(DirectExecutor.java:10)
at org.jgroups.protocols.TP.handleSingleMessage(TP.java:1789)
at org.jgroups.protocols.TP.receive(TP.java:1714)
at
org.apache.geode.distributed.internal.membership.gms.messenger.Transport.receive(Transport.java:152)
at org.jgroups.protocols.UDP$PacketReceiver.run(UDP.java:701)
at java.lang.Thread.run(Thread.java:745)
[error 2019-02-01 00:06:35.651 EST <ReconnectThread>
cache.GemFireCacheImpl] org.apache.geode.cache.CacheClosedException:
Could not PDX serialize because the cache was closed
[warn 2019-02-01 00:06:35.652 EST <ReconnectThread>
internal.InternalDistributedSystem] Exception occurred while trying to
create the cache during reconnect
org.apache.geode.cache.CacheClosedException: Could not PDX serialize
because the cache was closed
at
org.apache.geode.pdx.internal.TypeRegistry.getPdxSerializer(TypeRegistry.java:317)
at
org.apache.geode.internal.InternalDataSerializer.writeUserObject(InternalDataSerializer.java:1648)
at
org.apache.geode.internal.InternalDataSerializer.writeWellKnownObject(InternalDataSerializer.java:1548)
at
org.apache.geode.internal.InternalDataSerializer.basicWriteObject(InternalDataSerializer.java:2200)
at org.apache.geode.DataSerializer.writeObject(DataSerializer.java:2952)
at
org.apache.geode.internal.cache.MemberFunctionStreamingMessage.toData(MemberFunctionStreamingMessage.java:315)
at
org.apache.geode.internal.InternalDataSerializer.invokeToData(InternalDataSerializer.java:2398)
at
org.apache.geode.internal.InternalDataSerializer.writeDSFID(InternalDataSerializer.java:1517)
at
org.apache.geode.internal.tcp.MsgStreamer.writeMessage(MsgStreamer.java:234)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:394)
at
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:251)
at
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:616)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.directChannelSend(GMSMembershipManager.java:1686)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.send(GMSMembershipManager.java:1864)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendViaMembershipManager(ClusterDistributionManager.java:2860)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendOutgoing(ClusterDistributionManager.java:2780)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.sendMessage(ClusterDistributionManager.java:2819)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.putOutgoing(ClusterDistributionManager.java:1523)
at
org.apache.geode.internal.cache.execute.StreamingFunctionOperation.getFunctionResultFrom(StreamingFunctionOperation.java:107)
at
org.apache.geode.internal.cache.execute.MemberFunctionExecutor.executeFunction(MemberFunctionExecutor.java:151)
at
org.apache.geode.internal.cache.execute.MemberFunctionExecutor.executeFunction(MemberFunctionExecutor.java:189)
at
org.apache.geode.internal.cache.execute.AbstractExecution.execute(AbstractExecution.java:392)
at
org.apache.geode.internal.cache.ClusterConfigurationLoader.requestConfigurationFromOneLocator(ClusterConfigurationLoader.java:312)
at
org.apache.geode.internal.cache.ClusterConfigurationLoader.requestConfigurationFromLocators(ClusterConfigurationLoader.java:282)
at
org.apache.geode.internal.cache.GemFireCacheImpl.requestSharedConfiguration(GemFireCacheImpl.java:1066)
at
org.apache.geode.internal.cache.GemFireCacheImpl.<init>(GemFireCacheImpl.java:857)
at
org.apache.geode.internal.cache.GemFireCacheImpl.basicCreate(GemFireCacheImpl.java:794)
at
org.apache.geode.internal.cache.GemFireCacheImpl.create(GemFireCacheImpl.java:773)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2765)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531)
at java.lang.Thread.run(Thread.java:745)
[fatal 2019-02-01 00:07:35.810 EST <ReconnectThread> gms.Services]
Unexpected exception while booting membership services
java.lang.NullPointerException
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:483)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:361)
at
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:146)
at
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:105)
at
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:771)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:889)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:533)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:769)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:362)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:348)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:342)
at
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:215)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2704)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531)
at java.lang.Thread.run(Thread.java:745)
[error 2019-02-01 00:07:35.810 EST <ReconnectThread> gms.Services]
Unexpected problem starting up membership services
java.lang.NullPointerException
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:483)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:361)
at
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:146)
at
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:105)
at
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:771)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:889)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:533)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:769)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:362)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:348)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:342)
at
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:215)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2704)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2530)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1044)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:3406)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1534)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$3(GMSMembershipManager.java:2531)
at java.lang.Thread.run(Thread.java:745)
Thanks,
Dharam
*From:*John Blum [mailto:[email protected]]
*Sent:* Tuesday, January 29, 2019 3:49 AM
*To:* [email protected] <mailto:[email protected]>
*Subject:* Re: NullPointerException with GMS services (Geode 1.6.0)
Although, there is an apparent problem with your Locator configuration
property, which can/will lead to an NPE thrown by Geode in this case
(as I reported in https://issues.apache.org/jira/browse/GEODE-6153
<https://issues.apache.org/jira/browse/GEODE-6153>, which is not
exactly the same as your problem), as Bruce points out, this is
occurring during reconnect.
If the default value of the Locator property had been used (which it
apparently wasn't in this case), then the node would have failed even
on initial startup. So, something else is going wrong here.
On Mon, Jan 28, 2019 at 10:32 AM Bruce Schuchardt
<[email protected] <mailto:[email protected]>> wrote:
These NPEs are probably not your main problem since their
occurring in a Reconnect Thread. Something bad happened when you
tried to start these caches. They joined the cluster and then
were kicked out for some reason. The log file for that process
will likely tell you what the problem was. Post it or PM it to me
if you want help.
On 1/27/19 10:56 PM, Thacker, Dharam wrote:
java.lang.NullPointerException
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.establishLocalAddress(JGroupsMessenger.java:468)
at
org.apache.geode.distributed.internal.membership.gms.messenger.JGroupsMessenger.start(JGroupsMessenger.java:355)
at
org.apache.geode.distributed.internal.membership.gms.Services.start(Services.java:157)
at
org.apache.geode.distributed.internal.membership.gms.GMSMemberFactory.newMembershipManager(GMSMemberFactory.java:106)
at
org.apache.geode.distributed.internal.membership.MemberFactory.newMembershipManager(MemberFactory.java:90)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:1027)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.<init>(ClusterDistributionManager.java:1061)
at
org.apache.geode.distributed.internal.ClusterDistributionManager.create(ClusterDistributionManager.java:554)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.initialize(InternalDistributedSystem.java:763)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:355)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:341)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.newInstance(InternalDistributedSystem.java:335)
at
org.apache.geode.distributed.DistributedSystem.connect(DistributedSystem.java:211)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.reconnect(InternalDistributedSystem.java:2736)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.tryReconnect(InternalDistributedSystem.java:2560)
at
org.apache.geode.distributed.internal.InternalDistributedSystem.disconnect(InternalDistributedSystem.java:1041)
at
org.apache.geode.distributed.internal.ClusterDistributionManager$DMListener.membershipFailure(ClusterDistributionManager.java:4033)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.uncleanShutdown(GMSMembershipManager.java:1554)
at
org.apache.geode.distributed.internal.membership.gms.mgr.GMSMembershipManager.lambda$forceDisconnect$1(GMSMembershipManager.java:2561)
at java.lang.Thread.run(Thread.java:745)
--
-John
john.blum10101 (skype)
This message is confidential and subject to terms at:
https://www.jpmorgan.com/emaildisclaimer
<https://www.jpmorgan.com/emaildisclaimer> including on
confidentiality, legal privilege, viruses and monitoring of electronic
messages. If you are not the intended recipient, please delete this
message and notify the sender immediately. Any unauthorized use is
strictly prohibited.
