I am testing a very large durable cache that mostly cannot fit into memory.
I start loading in a lot of data via a data streamer. At some point the
data becomes too large to fit into memory so ignite starts writing a lot to
disk during checkpoints. But after some point after that, the datastreamer
stops streaming. It's stuck. Doing a jstack on ignite i see the
datastreamer threads are all stuck in a parked state. Any thoughts?
"data-streamer-stripe-4-#37" #62 prio=5 os_prio=0 tid=0x00007fe8c212d000
nid=0x1410 waiting on condition [0x00007fe73d1d6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
at
org.apache.ignite.internal.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
at java.lang.Thread.run(Thread.java:745)
The checkpoint thread seems to be parked too
"checkpoint-runner-#354" #397 prio=5 os_prio=0 tid=0x00007fe49c04f000
nid=0x177c waiting on condition [0x00007fdc54a1c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000002c08a6ef8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- None
My configuration is as follows:
<bean id="ignite.cfg"
class="org.apache.ignite.configuration.IgniteConfiguration">
<property name="publicThreadPoolSize" value="128"/>
<property name="systemThreadPoolSize" value="128"/>
<property name="dataStreamerThreadPoolSize" value="128"/>
<property name="deploymentMode" value="SHARED"/>
<property name="peerClassLoadingEnabled" value="true"/>
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="storagePath" value="/per/persistence"/>
<property name="walPath" value="/wal"/>
<property name="walMode" value="BACKGROUND"/>
<property name="walFlushFrequency" value="10000"/>
<property name="checkpointThreads" value="128"/>
<property name="writeThrottlingEnabled" value="true"/>
<property name="walArchivePath" value="/wal/archive"/>
<property name="defaultDataRegionConfiguration">
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="Default_Region"/>
<property name="maxSize" value="#{100L * 1024 *
1024}"/>
</bean>
</property>
<property name="dataRegionConfigurations">
<list>
<bean
class="org.apache.ignite.configuration.DataRegionConfiguration">
<property name="name" value="20G_Region_Eviction_P"/>
<property name="initialSize" value="#{10L * 1024 * 1024}"/>
<property name="maxSize" value="#{20 * 1024L * 1024 *
1024}"/>
<property name="persistenceEnabled" value="true"/>
<property name="pageEvictionMode" value="RANDOM_2_LRU"/>
<property name="checkpointPageBufferSize" value="#{2 * 1024L
* 1024 * 1024}"/>
</bean>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
