Hi, It looks bit different to me. I can provide some more information.
Host reboot information: user@member002:/ $ last reboot | head -1 reboot system boot 2.6.32-642.11.1. Sun Jun 4 22:44 - 12:20 (13:36) user@member001:/ $ last reboot | head -1 reboot system boot 2.6.32-573.35.2. Mon Mar 6 16:56 - 12:21 (90+18:24) Above information proves that member001 was never rebooted which I can see from timestamp as well. If I follow error logs, Exception: ConflictingPersistentDataException Conditions: On member001 : ConfigDiskDir_Locator1 created at timestamp 1496241336712 version 0 diskStoreId 31efa18230134865-b4fd0fcbde63ade6 On member002 : ConfigDiskDir_Locator2 created at timestamp 1496241344046 version 0 diskStoreId df94511d0f3d4295-91ec9286a18aaa75 Later on, 1. member001 never went offline actually but member002 was rebooted [As explained above] 2. Locator and server already running in member001 3. Locator and server were requested to start on member002 4. member002 refused as explained in "Conditions" Thanks & Regards, Dharam From: Jinmei Liao [mailto:[email protected]] Sent: Monday, June 05, 2017 8:38 PM To: [email protected] Subject: Re: How to deal with cluster configuration service failure Is this related to https://issues.apache.org/jira/browse/GEODE-3003? On Sun, Jun 4, 2017 at 11:39 PM, Thacker, Dharam <[email protected]<mailto:[email protected]>> wrote: Hi Team, Could someone help to understand how to deal with below scenario where cluster configuration service fails to start in another locator? Which supportive action should we take to rectify this? Note: member001.IP.MAKSED – IP address of member001 member002.IP.MASKED – IP address of member002 Locator logs on member002: [info 2017/06/05 02:07:11.941 EDT Locator2 <Pooled Message Processor 1> tid=0x3d] Initializing region _ConfigurationRegion [warning 2017/06/05 02:07:11.951 EDT Locator2 <Pooled Message Processor 1> tid=0x3d] Initialization failed for Region /_ConfigurationRegion org.apache.geode.cache.persistence.ConflictingPersistentDataException: Region /_ConfigurationRegion refusing to initialize from member member001(Locator1:5160:locator)<ec><v0>:1024 with persistent data /member001.IP.MASKED:/local/apps/shared/geode/members/Locator1/work/ConfigDiskDir_Locator1 created at timestamp 1496241336712 version 0 diskStoreId 31efa18230134865-b4fd0fcbde63ade6 name Locator1 which was offline when the local data from /member002.IP.MASKED:/local/apps/shared/geode/members/Locator2/work/ConfigDiskDir_Locator2 created at timestamp 1496241344046 version 0 diskStoreId df94511d0f3d4295-91ec9286a18aaa75 name Locator2 was last online at org.apache.geode.internal.cache.persistence.PersistenceAdvisorImpl.checkMyStateOnMembers(PersistenceAdvisorImpl.java:751) at org.apache.geode.internal.cache.persistence.PersistenceAdvisorImpl.getInitialImageAdvice(PersistenceAdvisorImpl.java:812) at org.apache.geode.internal.cache.persistence.CreatePersistentRegionProcessor.getInitialImageAdvice(CreatePersistentRegionProcessor.java:52) at org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1267) at org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1101) at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3308) at org.apache.geode.distributed.internal.ClusterConfigurationService.getConfigurationRegion(ClusterConfigurationService.java:709) at org.apache.geode.distributed.internal.ClusterConfigurationService.initSharedConfiguration(ClusterConfigurationService.java:426) at org.apache.geode.distributed.internal.InternalLocator$SharedConfigurationRunnable.run(InternalLocator.java:649) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:621) at org.apache.geode.distributed.internal.DistributionManager$4$1.run(DistributionManager.java:878) at java.lang.Thread.run(Thread.java:745) [error 2017/06/05 02:07:11.959 EDT Locator2 <Pooled Message Processor 1> tid=0x3d] Error occurred while initializing cluster configuration java.lang.RuntimeException: Error occurred while initializing cluster configuration at org.apache.geode.distributed.internal.ClusterConfigurationService.getConfigurationRegion(ClusterConfigurationService.java:722) at org.apache.geode.distributed.internal.ClusterConfigurationService.initSharedConfiguration(ClusterConfigurationService.java:426) at org.apache.geode.distributed.internal.InternalLocator$SharedConfigurationRunnable.run(InternalLocator.java:649) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:621) at org.apache.geode.distributed.internal.DistributionManager$4$1.run(DistributionManager.java:878) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.geode.cache.persistence.ConflictingPersistentDataException: Region /_ConfigurationRegion refusing to initialize from member member001(Locator1:5160:locator)<ec><v0>:1024 with persistent data /member001.IP.MASKED:/local/apps/shared/geode/members/Locator1/work/ConfigDiskDir_RavenLocator1 created at timestamp 1496241336712 version 0 diskStoreId 31efa18230134865-b4fd0fcbde63ade6 name RavenLocator1 which was offline when the local data from /member002.IP.MASKED:/local/apps/shared/geode/members/Locator2/work/ConfigDiskDir_Locator2 created at timestamp 1496241344046 version 0 diskStoreId df94511d0f3d4295-91ec9286a18aaa75 name Locator2 was last online at org.apache.geode.internal.cache.persistence.PersistenceAdvisorImpl.checkMyStateOnMembers(PersistenceAdvisorImpl.java:751) at org.apache.geode.internal.cache.persistence.PersistenceAdvisorImpl.getInitialImageAdvice(PersistenceAdvisorImpl.java:812) at org.apache.geode.internal.cache.persistence.CreatePersistentRegionProcessor.getInitialImageAdvice(CreatePersistentRegionProcessor.java:52) at org.apache.geode.internal.cache.DistributedRegion.getInitialImageAndRecovery(DistributedRegion.java:1267) at org.apache.geode.internal.cache.DistributedRegion.initialize(DistributedRegion.java:1101) at org.apache.geode.internal.cache.GemFireCacheImpl.createVMRegion(GemFireCacheImpl.java:3308) at org.apache.geode.distributed.internal.ClusterConfigurationService.getConfigurationRegion(ClusterConfigurationService.java:709) ... 7 more Thanks & Regards, Dharam This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer<http://www.jpmorgan.com/emaildisclaimer> including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited. -- Cheers Jinmei This message is confidential and subject to terms at: http://www.jpmorgan.com/emaildisclaimer including on confidentiality, legal privilege, viruses and monitoring of electronic messages. If you are not the intended recipient, please delete this message and notify the sender immediately. Any unauthorized use is strictly prohibited.
