Hi! I'm using the cassandra as persistence store for my caches and have one issue by handling a huge data (via IgniteDataStreamer from kafka). Ignite Configuration: final IgniteConfiguration igniteConfiguration = new IgniteConfiguration(); igniteConfiguration.setIgniteInstanceName("test"); igniteConfiguration.setClientMode(true); igniteConfiguration.setGridLogger(new Slf4jLogger()); igniteConfiguration.setMetricsLogFrequency(0); igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi()); final BinaryConfiguration binaryConfiguration = new BinaryConfiguration(); binaryConfiguration.setCompactFooter(false); igniteConfiguration.setBinaryConfiguration(binaryConfiguration); igniteConfiguration.setPeerClassLoadingEnabled(true); final MemoryPolicyConfiguration memoryPolicyConfiguration = new MemoryPolicyConfiguration(); memoryPolicyConfiguration.setName("3Gb_Region_Eviction"); memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L); memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L); memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU); final MemoryConfiguration memoryConfiguration = new MemoryConfiguration(); memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration); igniteConfiguration.setMemoryConfiguration(memoryConfiguration);
Cache configuration: final CacheConfiguration<String, BinaryObject> cacheConfiguration = new CacheConfiguration<>(); cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC); cacheConfiguration.setStoreKeepBinary(true); cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); cacheConfiguration.setBackups(0); cacheConfiguration.setStatisticsEnabled(false); cacheConfiguration.setName("TestCache"); cacheConfiguration.setReadThrough(true); cacheConfiguration.setWriteThrough(true); cacheConfiguration.setWriteBehindEnabled(true); cacheConfiguration.setWriteBehindFlushFrequency(10000); cacheConfiguration.setWriteBehindFlushSize(0); cacheConfiguration.setWriteBehindFlushThreadCount(2); cacheConfiguration.setWriteBehindBatchSize(10000); final CassandraCacheStoreFactory<String, BinaryObject> cacheStoreFactory = new CassandraCacheStoreFactory<>(); final DataSource dataSource = new DataSource(); dataSource.setContactPoints(contactPoints); dataSource.setReadConsistency("ONE"); dataSource.setWriteConsistency("ONE"); dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new RoundRobinPolicy())); cacheStoreFactory.setDataSource(dataSource); final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\" table=\"%s\">" + "<keyPersistence class=\"%s\" strategy=\"%s\"/>" + "<valuePersistence class=\"%s\" strategy=\"%s\"/>" + "</persistence>"; final KeyValuePersistenceSettings settings = new KeyValuePersistenceSettings( String.format(CASSANDRA_PERSISTENCE, "test", "test_table", "java.lang.String", "PRIMITIVE", "org.apache.ignite.binary.BinaryObject", "BLOB")); cacheStoreFactory.setPersistenceSettings(settings); cacheConfiguration.setCacheStoreFactory(cacheStoreFactory); When application works some time (hour or more, may be less - from time to time) I see this exceptions on the ignite nodes: Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072] ^-- H/N/C [hosts=3, nodes=3, CPUs=96] ^-- CPU [cur=0%, avg=1.86%, GC=0%] ^-- PageMemory [pages=118064] ^-- Heap [used=4800MB, free=53.12%, comm=10240MB] ^-- Non heap [used=78MB, free=-1%, comm=80MB] ^-- Public thread pool [active=0, idle=32, qSize=0] ^-- System thread pool [active=0, idle=32, qSize=0] ^-- Outbound messages queue [size=0] [15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList [name=null, buckets=256, dataPages=102080, reusePages=0] [15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared statement cluster error detected, refreshing Cassandra session com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra session refreshed [15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms [15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms [15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared statement cluster error detected, refreshing Cassandra session com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed [15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed [15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed [15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed [15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed [15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed [15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed [15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed [15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed [15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra session refreshed [15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms [15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) After this I have to restart the cluster and application or have "transactions deadlock" message. And I checked: network is fine, no OOME in the ignite and cassandra (cassandra logs don't say anything about it). I understand that this is issue of persistence into cassandra but don't know how can I fix it. And after some attempts ignite node is stopped. I tried to change write behind parameters and disable it at all - but have the same issue. May be I have to see in the other place? May be this is important - i use IgniteDataStreamer for reading values from kafka (earlier I used the simple cache operations and didn't have this problem at all). final IgniteDataStreamer<String, BinaryObject> streamer = ignite.dataStreamer(callCache().getName()); streamer.autoFlushFrequency(5000); streamer.keepBinary(true); streamer.perNodeBufferSize(5120); And one more question about persistence settings. If I'm working with BinaryObjects and ignite doesn't know about my class types, should I use valuePersistence class = "org.apache.ignite.binary.BinaryObject"? I did it because peer class loading doesn't work for my own classes and I don't want to add new jars into all ignite nodes and all my apps. Thanks for help! -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Ignite-cassandra-module-issue-tp13808.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.