Hello!
Unfortunately it's hard to say without looking at complete logs from your
nodes. There's too many questions and not enough clues.
Regards,
--
Ilya Kasnacheev
пт, 6 дек. 2019 г. в 03:56, Steven.Gerdes <steven.ger...@shareworks.com>:
> The production instance has issues with ignite heaping out, the solution we
> attempted to implement was to set the default data region to have swap
> enabled and also set a eviction policy on the server with a maxMemorySize
> such that it was much less then the Xmx jvm memory size.
>
> Testing locally with a dev version of our server (weblogic acting as ignite
> with client mode enabled) and the docker instance of ignite 2.7.6 it
> appears
> as though using this configuration does not solve ignites instability
> issues.
>
> Many different configurations were attempted (for the full config see
> bottom
> of post). The desired configuration would be one which the client has no
> cache and the server does all the caching. That was done with attempting
> the
> below on the server:
>
> <property name="onheapCacheEnabled" value="true"/>
>
> <property name="evictionPolicyFactory">
>
> <bean
> class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicyFactory">
> <property name="maxMemorySize" value="#{50L * 1024 * 1024}"/>
>
> <property name="maxSize" value="1000000000"/>
> </bean>
> </property>
>
> With the above server configuration 3 attempts were made to the client
> configuration:
>
> 1. Mirrored configuration on the client
> 2. Similar configuration with maxSize set to 0 (an attempt at ensuring the
> client didn't try to cache)
> 3. enabling IGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK and not having any
> eviction policy for the client
>
> All three of these configurations resulted in the client weblogic to
> disconnect from the cluster and finally to die (while attempting to
> reconnect to ignite)
>
> Error from client before death:
> | Servlet failed with an Exception
> | java.lang.IllegalStateException: Grid is in invalid state to perform
> this operation. It either not started yet or has already being or have
> stopped [igniteInstanceName=null, state=STOPPING]
> | at
>
> org.apache.ignite.internal.GridKernalGatewayImpl.illegalState(GridKernalGatewayImpl.java:201)
> | at
>
> org.apache.ignite.internal.GridKernalGatewayImpl.readLock(GridKernalGatewayImpl.java:95)
> | at
> org.apache.ignite.internal.IgniteKernal.guard(IgniteKernal.java:3886)
> | at
>
> org.apache.ignite.internal.IgniteKernal.transactions(IgniteKernal.java:2862)
> | at
>
> org.apache.ignite.cache.websession.CustomWebSessionFilter.init(CustomWebSessionFilter.java:273)
> | Truncated. see log file for complete stacktrace
>
> Error in ignitevisorcmd.sh:
> SEVERE: Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=tcp-disco-msg-worker,
> blockedFor=17s]
> Dec 06, 2019 12:17:27 AM java.util.logging.LogManager$RootLogger log
> SEVERE: Critical system error detected. Will be handled accordingly to
> configured handler [hnd=StopNodeFailureHandler
> [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SEGMENTATION]]],
> failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class
> o.a.i.IgniteException: GridWorker [name=tcp-disco-msg-worker,
> igniteInstanceName=null, finished=false, heartbeatTs=1575591430824]]]
> class org.apache.ignite.IgniteException: GridWorker
> [name=tcp-disco-msg-worker, igniteInstanceName=null, finished=false,
> heartbeatTs=1575591430824]
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
>
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
>
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
>
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
>
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:221)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
>
> Sometimes in testing it was possible for the client to successfully
> reconnect. But I could not see why it was inconsistent with this behavior.
>
> A separate test was conducted in which there was no eviction policy or
> on-heap enabled on either the client or server. This seems to be more
> stable.
>
> Is there something incorrect with the configuration? Is there something
> missing that would allow us to use on-heap memory without it causing issue
> with our client?
>
>
>
>
> Apendix:
>
> Client Configuration:
> <?xml version="1.0" encoding="UTF-8"?>
>
> <beans xmlns="http://www.springframework.org/schema/beans";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="
> http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd";>
> <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="clientMode" value="true"/>
> <property name="peerClassLoadingEnabled" value="true"/>
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="forceServerMode" value="true"/>
> <property name="ipFinder">
> <bean
>
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <value>127.0.0.1:47500..47509</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
> <property name="cacheConfiguration">
> <list>
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="WebCache"/>
> <property name="cacheMode" value="REPLICATED"/>
> </bean>
> </list>
> </property>
> <property name="failureHandler">
> <bean class="org.apache.ignite.failure.StopNodeFailureHandler">
> <property name="ignoredFailureTypes">
> <list>
> <value>SYSTEM_WORKER_BLOCKED</value>
> <value>SEGMENTATION</value>
> </list>
> </property>
> </bean>
> </property>
> <property name="lifecycleBeans">
> <list>
> <bean class="com.company.common.ignite.CustomIgniteLifecycleBean"/>
> </list>
> </property>
> </bean>
> </beans>
>
>
> Server Configuration (used with docker image apache-ignite 2.7.6):
> <?xml version="1.0" encoding="UTF-8"?>
>
> <beans xmlns="http://www.springframework.org/schema/beans";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation=" http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd";>
> <bean id="grid.cfg"
> class="org.apache.ignite.configuration.IgniteConfiguration">
> <property name="peerClassLoadingEnabled" value="true"/>
> <property name="discoverySpi">
> <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
> <property name="ipFinder">
> <bean
>
> class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
> <property name="addresses">
> <list>
> <value>127.0.0.1:47500..47509</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </property>
> <property name="dataStorageConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataStorageConfiguration">
> <property name="defaultDataRegionConfiguration">
> <bean
> class="org.apache.ignite.configuration.DataRegionConfiguration">
> <property name="initialSize" value="#{10L * 1024 * 1024}"/>
> <property name="maxSize" value="#{20L * 1024 * 1024 * 1024}"/>
> <property name="swapPath" value="/opt/ignite/"/>
> </bean>
> </property>
> </bean>
> </property>
> <property name="cacheConfiguration">
> <list>
> <bean class="org.apache.ignite.configuration.CacheConfiguration">
> <property name="name" value="WebCache"/>
> <property name="cacheMode" value="REPLICATED"/>
> </bean>
> </list>
> </property>
> <property name="failureHandler">
> <bean class="org.apache.ignite.failure.StopNodeFailureHandler">
> <property name="ignoredFailureTypes">
> <list>
> <value>SYSTEM_WORKER_BLOCKED</value>
> <value>SEGMENTATION</value>
> </list>
> </property>
> </bean>
> </property>
> </bean>
> </beans>
>
>
>
>
> --
> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
