Hi All,
we notice more than one time the ignite node and cache just stop itself quietly
. this is weird, no specific ERROR log print.
we allocated enough memory for those cache. From the GC log we can not find
any obvious abnormal as the main process still running .
<property name="cacheMode" value="REPLICATED"/>
<property name="atomicityMode" value="ATOMIC"/>
<property name="readFromBackup" value="true"/>
<property name="readThrough" value="true"/>
<property name="writeThrough" value="true"/>
<property name="writeBehindEnabled" value="true"/>
<property name="writeBehindBatchSize" value="8192"/>
<property name="writeBehindFlushThreadCount" value="5"/>
<property name="writeBehindFlushFrequency" value="10000"/>
<property name="storeKeepBinary" value="false"/>
<property name="onheapCacheEnabled" value="true"/>
<property name="copyOnRead" value="false"/>
<property name="memoryPolicyName" value="Trans_Region"/>
<property name="evictionPolicy">
<bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
<property name="maxSize" value="#{80000}"/>
</bean>
</property>
we have enough memory for this cache:
<bean class="org.apache.ignite.configuration.MemoryPolicyConfiguration">
<property name="name" value="Trans_Region"/>
<!-- 8~12g -->
<property name="initialSize" value="#{8L *1024L * 1024L * 1024L}"/>
<property name="maxSize" value="#{12L *1024L * 1024L * 1024L}"/>
<property name="pageEvictionMode" value="RANDOM_2_LRU"/>
</bean>
the server have 64G memory totally
But if we keep run for more than 5 hours then we keep getting:
javax.cache.CacheException: class
org.apache.ignite.internal.NodeStoppingException: Operation has been cancelled
(node is stopping).
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1312)
~[ignite-core-2.1.0.jar!/:2.1.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2630)
~[ignite-core-2.1.0.jar!/:2.1.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1509)
~[ignite-core-2.1.0.jar!/:2.1.0]
at
com.tophold.trade.booker.TransactionCommandGateWay.handle(TransactionCommandGateWay.java:343)
~[classes!/:?]
at sun.reflect.GeneratedMethodAccessor73.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_131]
Eventually:
java.lang.IllegalStateException: class
org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed to
perform cache operation (cache is stopped): TransactionEntry
at
org.apache.ignite.internal.processors.cache.GridCacheGateway.checkState(GridCacheGateway.java:88)
~[ignite-core-2.1.0.jar!/:2.1.0]
at
org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:179)
~[ignite-core-2.1.0.jar!/:2.1.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.onEnter(IgniteCacheProxy.java:2722)
~[ignite-core-2.1.0.jar!/:2.1.0]
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.get(IgniteCacheProxy.java:1172)
~[ignite-core-2.1.0.jar!/:2.1.0]
our JVM arguments:
-Xms10g
-Xmx10g
-server
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=D:\data\memory
-XX:NewRatio=3
-XX:SurvivorRatio=8
-XX:+UseTLAB
-XX:GCTimeRatio=4
-XX:+ScavengeBeforeFullGC
-XX:G1HeapRegionSize=8M
-XX:ConcGCThreads=8
-XX:G1HeapWastePercent=10
-XX:+AggressiveOpts
-XX:MaxMetaspaceSize=256m
-XX:+UseG1GC
-XX:MaxGCPauseMillis=1000
-XX:InitiatingHeapOccupancyPercent=35
-XX:+DisableExplicitGC
-Djava.awt.headless=true
Need we detect the Node stopping itself events? then re-bring up the Ignite
instance and cache? very appreciate your time!
Regards
aa...@tophold.com
