Hi Andrey et al I believe I now know what the problem is, the Cassandra session is refreshed, but before it is a prepared statement is created/used and there, and so using a new session with an old prepared statement is not working.
The way to reproduce is 1) Start Ignite Server Node 2) Start client which inserts a batch of 100 elements 3) End client 4) Now Ignite Server Node returns the Cassandra Session to the pool 5) Wait 5+ minutes 6) Now Ignite Server Node has does a clean-up of the “unused” Cassandra sessions 7) Start client which inserts a batch of 100 elements 8) Boom ! The exception starts to happen Reason is 1) Execute is called for a BATCH 2) Prepared-statement is reused since there is a cache of those 3) It is about to do session().execute( batch ) 4) BUT the call to session() results in refreshing the session, and this is where the prepared statements as the old session new them are cleaned up 5) Now it is looping over 100 times with a NEW session but with an OLD prepared statement This is a bug, -Tobias From: Andrey Mashenkov <andrey.mashen...@gmail.com> Reply-To: "user@ignite.apache.org" <user@ignite.apache.org> Date: Wednesday, 25 October 2017 at 14:12 To: "user@ignite.apache.org" <user@ignite.apache.org> Subject: Re: Ignite-cassandra module issue Hi Tobias, What ignite version do you use? May be this was already fixed in latest one? I see related fix inclueded in upcoming 2.3 version. See IGNITE-5897 [1] issue. It is unobvious, but this fix session init\end logic, so session should be closed in proper way. [1] https://issues.apache.org/jira/browse/IGNITE-5897 On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson <tobias.eriks...@qvantel.com<mailto:tobias.eriks...@qvantel.com>> wrote: Hi Sorry did not include the context when I replied Has anyone been able to resolve this problem, cause I have it too on and off In fact it sometimes happens just like that, e.g. I have been running my Ignite client and then stop it, and then it takes a while and run it again, and all by a sudden this error shows up. An that is the first thing that happens, and there is NOT a massive amount of load on Cassandra at that time. But I have also seen it when I hammer Ignite/Cassandra with updates/inserts. This is a deal-breaker for me, I need to understand how to fix this, cause having this in production is not an option. -Tobias Hi! I'm using the cassandra as persistence store for my caches and have one issue by handling a huge data (via IgniteDataStreamer from kafka). Ignite Configuration: final IgniteConfiguration igniteConfiguration = new IgniteConfiguration(); igniteConfiguration.setIgniteInstanceName("test"); igniteConfiguration.setClientMode(true); igniteConfiguration.setGridLogger(new Slf4jLogger()); igniteConfiguration.setMetricsLogFrequency(0); igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi()); final BinaryConfiguration binaryConfiguration = new BinaryConfiguration(); binaryConfiguration.setCompactFooter(false); igniteConfiguration.setBinaryConfiguration(binaryConfiguration); igniteConfiguration.setPeerClassLoadingEnabled(true); final MemoryPolicyConfiguration memoryPolicyConfiguration = new MemoryPolicyConfiguration(); memoryPolicyConfiguration.setName("3Gb_Region_Eviction"); memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L); memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L); memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU); final MemoryConfiguration memoryConfiguration = new MemoryConfiguration(); memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration); igniteConfiguration.setMemoryConfiguration(memoryConfiguration); Cache configuration: final CacheConfiguration<String, BinaryObject> cacheConfiguration = new CacheConfiguration<>(); cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC); cacheConfiguration.setStoreKeepBinary(true); cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); cacheConfiguration.setBackups(0); cacheConfiguration.setStatisticsEnabled(false); cacheConfiguration.setName("TestCache"); cacheConfiguration.setReadThrough(true); cacheConfiguration.setWriteThrough(true); cacheConfiguration.setWriteBehindEnabled(true); cacheConfiguration.setWriteBehindFlushFrequency(10000); cacheConfiguration.setWriteBehindFlushSize(0); cacheConfiguration.setWriteBehindFlushThreadCount(2); cacheConfiguration.setWriteBehindBatchSize(10000); final CassandraCacheStoreFactory<String, BinaryObject> cacheStoreFactory = new CassandraCacheStoreFactory<>(); final DataSource dataSource = new DataSource(); dataSource.setContactPoints(contactPoints); dataSource.setReadConsistency("ONE"); dataSource.setWriteConsistency("ONE"); dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new RoundRobinPolicy())); cacheStoreFactory.setDataSource(dataSource); final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\" table=\"%s\">" + "<keyPersistence class=\"%s\" strategy=\"%s\"/>" + "<valuePersistence class=\"%s\" strategy=\"%s\"/>" + "</persistence>"; final KeyValuePersistenceSettings settings = new KeyValuePersistenceSettings( String.format(CASSANDRA_PERSISTENCE, "test", "test_table", "java.lang.String", "PRIMITIVE", "org.apache.ignite.binary.BinaryObject", "BLOB")); cacheStoreFactory.setPersistenceSettings(settings); cacheConfiguration.setCacheStoreFactory(cacheStoreFactory); When application works some time (hour or more, may be less - from time to time) I see this exceptions on the ignite nodes: Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072] ^-- H/N/C [hosts=3, nodes=3, CPUs=96] ^-- CPU [cur=0%, avg=1.86%, GC=0%] ^-- PageMemory [pages=118064] ^-- Heap [used=4800MB, free=53.12%, comm=10240MB] ^-- Non heap [used=78MB, free=-1%, comm=80MB] ^-- Public thread pool [active=0, idle=32, qSize=0] ^-- System thread pool [active=0, idle=32, qSize=0] ^-- Outbound messages queue [size=0] [15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList [name=null, buckets=256, dataPages=102080, reusePages=0] [15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared statement cluster error detected, refreshing Cassandra session com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra session refreshed [15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms [15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms [15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms [15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared statement cluster error detected, refreshing Cassandra session com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed [15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed [15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed [15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed [15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed [15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed [15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed [15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed [15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed [15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2 attempt to execute Cassandra CQL statement: select "value" from "triggers"."callback" where "key"=?; [15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra session refreshed [15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms [15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared statement cluster error detected, another thread already fixed the problem com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have used a PreparedStatement that was created with another Cluster instance. at com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568) at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63) at org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144) at org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168) at org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500) at org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416) at org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059) at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057) at org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406) at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621) at org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) After this I have to restart the cluster and application or have "transactions deadlock" message. And I checked: network is fine, no OOME in the ignite and cassandra (cassandra logs don't say anything about it). I understand that this is issue of persistence into cassandra but don't know how can I fix it. And after some attempts ignite node is stopped. I tried to change write behind parameters and disable it at all - but have the same issue. May be I have to see in the other place? May be this is important - i use IgniteDataStreamer for reading values from kafka (earlier I used the simple cache operations and didn't have this problem at all). final IgniteDataStreamer<String, BinaryObject> streamer = ignite.dataStreamer(callCache().getName()); streamer.autoFlushFrequency(5000); streamer.keepBinary(true); streamer.perNodeBufferSize(5120); And one more question about persistence settings. If I'm working with BinaryObjects and ignite doesn't know about my class types, should I use valuePersistence class = "org.apache.ignite.binary.BinaryObject"? I did it because peer class loading doesn't work for my own classes and I don't want to add new jars into all ignite nodes and all my apps. Thanks for help! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/ -- Best regards, Andrey V. Mashenkov
