Re: Client (weblogic) attempting to rejoin cluster causes shutdown.
Ok, there is a lot to dig through here but let me try with establishing simple things first. 1. If two nodes (client or server) have the same cache specified in the configuration, the configs must be identical. 2. If one node has a cache configuration, it will be shared between all nodes automatically. 3. Client doesn't store data (except for LOCAL caches or near caches) Try only specifying caches and data regions on your server. Does it help? Stan On Tue, Dec 10, 2019 at 7:58 PM Ilya Kasnacheev wrote: > Hello! > > Unfortunately it's hard to say without looking at complete logs from your > nodes. There's too many questions and not enough clues. > > Regards, > -- > Ilya Kasnacheev > > > пт, 6 дек. 2019 г. в 03:56, Steven.Gerdes : > >> The production instance has issues with ignite heaping out, the solution >> we >> attempted to implement was to set the default data region to have swap >> enabled and also set a eviction policy on the server with a maxMemorySize >> such that it was much less then the Xmx jvm memory size. >> >> Testing locally with a dev version of our server (weblogic acting as >> ignite >> with client mode enabled) and the docker instance of ignite 2.7.6 it >> appears >> as though using this configuration does not solve ignites instability >> issues. >> >> Many different configurations were attempted (for the full config see >> bottom >> of post). The desired configuration would be one which the client has no >> cache and the server does all the caching. That was done with attempting >> the >> below on the server: >> >> >> >> >> >> > class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory"> >> >> >> >> >> >> >> With the above server configuration 3 attempts were made to the client >> configuration: >> >> 1. Mirrored configuration on the client >> 2. Similar configuration with maxSize set to 0 (an attempt at ensuring the >> client didn't try to cache) >> 3. enabling IGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK and not having any >> eviction policy for the client >> >> All three of these configurations resulted in the client weblogic to >> disconnect from the cluster and finally to die (while attempting to >> reconnect to ignite) >> >> Error from client before death: >> | Servlet failed with an Exception >> | java.lang.IllegalStateException: Grid is in invalid state to perform >> this operation. It either not started yet or has already being or have >> stopped [igniteInstanceName=null, state=STOPPING] >> | at >> >> org.apache.ignite.internal.GridKernalGatewayImpl.illegalState(GridKernalGatewayImpl.java:201) >> | at >> >> org.apache.ignite.internal.GridKernalGatewayImpl.readLock(GridKernalGatewayImpl.java:95) >> | at >> org.apache.ignite.internal.IgniteKernal.guard(IgniteKernal.java:3886) >> | at >> >> org.apache.ignite.internal.IgniteKernal.transactions(IgniteKernal.java:2862) >> | at >> >> org.apache.ignite.cache.websession.CustomWebSessionFilter.init(CustomWebSessionFilter.java:273) >> | Truncated. see log file for complete stacktrace >> >> Error in ignitevisorcmd.sh: >> SEVERE: Blocked system-critical thread has been detected. This can lead to >> cluster-wide undefined behaviour [threadName=tcp-disco-msg-worker, >> blockedFor=17s] >> Dec 06, 2019 12:17:27 AM java.util.logging.LogManager$RootLogger log >> SEVERE: Critical system error detected. Will be handled accordingly to >> configured handler [hnd=StopNodeFailureHandler >> [super=AbstractFailureHandler >> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SEGMENTATION]]], >> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class >> o.a.i.IgniteException: GridWorker [name=tcp-disco-msg-worker, >> igniteInstanceName=null, finished=false, heartbeatTs=1575591430824]]] >> class org.apache.ignite.IgniteException: GridWorker >> [name=tcp-disco-msg-worker, igniteInstanceName=null, finished=false, >> heartbeatTs=1575591430824] >> at >> >> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) >> at >> >> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) >> at >> >> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) >> at >> >> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) >> at >> >> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221) >> at >> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) >> at java.lang.Thread.run(Thread.java:748) >> >> Sometimes in testing it was possible for the client to successfully >> reconnect. But I could not see why it was inconsistent with this behavior. >> >> A separate test was conducted in which there was no eviction policy or >> on-heap enabled on either the client or server. This seems to
Re: Client (weblogic) attempting to rejoin cluster causes shutdown.
Hello! Unfortunately it's hard to say without looking at complete logs from your nodes. There's too many questions and not enough clues. Regards, -- Ilya Kasnacheev пт, 6 дек. 2019 г. в 03:56, Steven.Gerdes : > The production instance has issues with ignite heaping out, the solution we > attempted to implement was to set the default data region to have swap > enabled and also set a eviction policy on the server with a maxMemorySize > such that it was much less then the Xmx jvm memory size. > > Testing locally with a dev version of our server (weblogic acting as ignite > with client mode enabled) and the docker instance of ignite 2.7.6 it > appears > as though using this configuration does not solve ignites instability > issues. > > Many different configurations were attempted (for the full config see > bottom > of post). The desired configuration would be one which the client has no > cache and the server does all the caching. That was done with attempting > the > below on the server: > > > > > > class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory"> > > > > > > > With the above server configuration 3 attempts were made to the client > configuration: > > 1. Mirrored configuration on the client > 2. Similar configuration with maxSize set to 0 (an attempt at ensuring the > client didn't try to cache) > 3. enabling IGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK and not having any > eviction policy for the client > > All three of these configurations resulted in the client weblogic to > disconnect from the cluster and finally to die (while attempting to > reconnect to ignite) > > Error from client before death: > | Servlet failed with an Exception > | java.lang.IllegalStateException: Grid is in invalid state to perform > this operation. It either not started yet or has already being or have > stopped [igniteInstanceName=null, state=STOPPING] > | at > > org.apache.ignite.internal.GridKernalGatewayImpl.illegalState(GridKernalGatewayImpl.java:201) > | at > > org.apache.ignite.internal.GridKernalGatewayImpl.readLock(GridKernalGatewayImpl.java:95) > | at > org.apache.ignite.internal.IgniteKernal.guard(IgniteKernal.java:3886) > | at > > org.apache.ignite.internal.IgniteKernal.transactions(IgniteKernal.java:2862) > | at > > org.apache.ignite.cache.websession.CustomWebSessionFilter.init(CustomWebSessionFilter.java:273) > | Truncated. see log file for complete stacktrace > > Error in ignitevisorcmd.sh: > SEVERE: Blocked system-critical thread has been detected. This can lead to > cluster-wide undefined behaviour [threadName=tcp-disco-msg-worker, > blockedFor=17s] > Dec 06, 2019 12:17:27 AM java.util.logging.LogManager$RootLogger log > SEVERE: Critical system error detected. Will be handled accordingly to > configured handler [hnd=StopNodeFailureHandler > [super=AbstractFailureHandler > [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SEGMENTATION]]], > failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class > o.a.i.IgniteException: GridWorker [name=tcp-disco-msg-worker, > igniteInstanceName=null, finished=false, heartbeatTs=1575591430824]]] > class org.apache.ignite.IgniteException: GridWorker > [name=tcp-disco-msg-worker, igniteInstanceName=null, finished=false, > heartbeatTs=1575591430824] > at > > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) > at > > org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) > at > > org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) > at > > org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) > at > > org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > > Sometimes in testing it was possible for the client to successfully > reconnect. But I could not see why it was inconsistent with this behavior. > > A separate test was conducted in which there was no eviction policy or > on-heap enabled on either the client or server. This seems to be more > stable. > > Is there something incorrect with the configuration? Is there something > missing that would allow us to use on-heap memory without it causing issue > with our client? > > > > > Apendix: > > Client Configuration: > > > http://www.springframework.org/schema/beans"; > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" > http://www.springframework.org/schema/beans > http://www.springframework.org/schema/beans/spring-beans.xsd";> >class="org.apache.ignite.configuration.IgniteConfiguration"> > > > > > > > > class="org.apache.ignite.spi.
Client (weblogic) attempting to rejoin cluster causes shutdown.
The production instance has issues with ignite heaping out, the solution we attempted to implement was to set the default data region to have swap enabled and also set a eviction policy on the server with a maxMemorySize such that it was much less then the Xmx jvm memory size. Testing locally with a dev version of our server (weblogic acting as ignite with client mode enabled) and the docker instance of ignite 2.7.6 it appears as though using this configuration does not solve ignites instability issues. Many different configurations were attempted (for the full config see bottom of post). The desired configuration would be one which the client has no cache and the server does all the caching. That was done with attempting the below on the server: With the above server configuration 3 attempts were made to the client configuration: 1. Mirrored configuration on the client 2. Similar configuration with maxSize set to 0 (an attempt at ensuring the client didn't try to cache) 3. enabling IGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK and not having any eviction policy for the client All three of these configurations resulted in the client weblogic to disconnect from the cluster and finally to die (while attempting to reconnect to ignite) Error from client before death: | Servlet failed with an Exception | java.lang.IllegalStateException: Grid is in invalid state to perform this operation. It either not started yet or has already being or have stopped [igniteInstanceName=null, state=STOPPING] | at org.apache.ignite.internal.GridKernalGatewayImpl.illegalState(GridKernalGatewayImpl.java:201) | at org.apache.ignite.internal.GridKernalGatewayImpl.readLock(GridKernalGatewayImpl.java:95) | at org.apache.ignite.internal.IgniteKernal.guard(IgniteKernal.java:3886) | at org.apache.ignite.internal.IgniteKernal.transactions(IgniteKernal.java:2862) | at org.apache.ignite.cache.websession.CustomWebSessionFilter.init(CustomWebSessionFilter.java:273) | Truncated. see log file for complete stacktrace Error in ignitevisorcmd.sh: SEVERE: Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=tcp-disco-msg-worker, blockedFor=17s] Dec 06, 2019 12:17:27 AM java.util.logging.LogManager$RootLogger log SEVERE: Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SEGMENTATION]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=tcp-disco-msg-worker, igniteInstanceName=null, finished=false, heartbeatTs=1575591430824]]] class org.apache.ignite.IgniteException: GridWorker [name=tcp-disco-msg-worker, igniteInstanceName=null, finished=false, heartbeatTs=1575591430824] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) at org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Sometimes in testing it was possible for the client to successfully reconnect. But I could not see why it was inconsistent with this behavior. A separate test was conducted in which there was no eviction policy or on-heap enabled on either the client or server. This seems to be more stable. Is there something incorrect with the configuration? Is there something missing that would allow us to use on-heap memory without it causing issue with our client? Apendix: Client Configuration: http://www.springframework.org/schema/beans"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd";> 127.0.0.1:47500..47509 SYSTEM_WORKER_BLOCKED SEGMENTATION Server Configuration (used with docker image apache-ignite 2.7.6): http://www.springframework.org/schema/beans"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation=" http://www.springframework.org/sche
