Hi Dmitriy, I created a ticket for this: https://issues.apache.org/jira/browse/IGNITE-6853 it will be fixed in 2.4.
Thanks, Mike. 2017-11-09 2:35 GMT+03:00 Dmitriy Setrakyan <[email protected]>: > Hi Michael, do you have any update for the issue? > > On Thu, Nov 2, 2017 at 5:14 PM, Michael Cherkasov < > [email protected]> wrote: > >> Hi Tobias, >> >> Thank you for explaining how to reproduce it, I'll try your instruction. >> I spend several days trying to reproduce the issue, >> but I thought that the reason of this is too high load and I didn't stop >> client during testing. >> I'll check your instruction and try to fix the issue. >> >> Thanks, >> Mike. >> >> 2017-10-25 16:23 GMT+03:00 Tobias Eriksson <[email protected]>: >> >>> Hi Andrey et al >>> >>> I believe I now know what the problem is, the Cassandra session is >>> refreshed, but before it is a prepared statement is created/used and there, >>> and so using a new session with an old prepared statement is not working. >>> >>> >>> >>> The way to reproduce is >>> >>> 1) Start Ignite Server Node >>> >>> 2) Start client which inserts a batch of 100 elements >>> >>> 3) End client >>> >>> 4) Now Ignite Server Node returns the Cassandra Session to the >>> pool >>> >>> 5) Wait 5+ minutes >>> >>> 6) Now Ignite Server Node has does a clean-up of the “unused” >>> Cassandra sessions >>> >>> 7) Start client which inserts a batch of 100 elements >>> >>> 8) Boom ! The exception starts to happen >>> >>> >>> >>> Reason is >>> >>> 1) Execute is called for a BATCH >>> >>> 2) Prepared-statement is reused since there is a cache of those >>> >>> 3) It is about to do session().execute( batch ) >>> >>> 4) BUT the call to session() results in refreshing the session, >>> and this is where the prepared statements as the old session new them are >>> cleaned up >>> >>> 5) Now it is looping over 100 times with a NEW session but with >>> an OLD prepared statement >>> >>> >>> >>> This is a bug, >>> >>> >>> >>> -Tobias >>> >>> >>> >>> >>> >>> *From: *Andrey Mashenkov <[email protected]> >>> *Reply-To: *"[email protected]" <[email protected]> >>> *Date: *Wednesday, 25 October 2017 at 14:12 >>> *To: *"[email protected]" <[email protected]> >>> *Subject: *Re: Ignite-cassandra module issue >>> >>> >>> >>> Hi Tobias, >>> >>> >>> >>> What ignite version do you use? May be this was already fixed in latest >>> one? >>> >>> I see related fix inclueded in upcoming 2.3 version. >>> >>> >>> >>> See IGNITE-5897 [1] issue. It is unobvious, but this fix session >>> init\end logic, so session should be closed in proper way. >>> >>> >>> >>> [1] https://issues.apache.org/jira/browse/IGNITE-5897 >>> >>> >>> >>> >>> >>> On Wed, Oct 25, 2017 at 11:13 AM, Tobias Eriksson < >>> [email protected]> wrote: >>> >>> Hi >>> Sorry did not include the context when I replied >>> Has anyone been able to resolve this problem, cause I have it too on and >>> off >>> In fact it sometimes happens just like that, e.g. I have been running my >>> Ignite client and then stop it, and then it takes a while and run it >>> again, >>> and all by a sudden this error shows up. An that is the first thing that >>> happens, and there is NOT a massive amount of load on Cassandra at that >>> time. But I have also seen it when I hammer Ignite/Cassandra with >>> updates/inserts. >>> >>> This is a deal-breaker for me, I need to understand how to fix this, >>> cause >>> having this in production is not an option. >>> >>> -Tobias >>> >>> >>> Hi! >>> I'm using the cassandra as persistence store for my caches and have one >>> issue by handling a huge data (via IgniteDataStreamer from kafka). >>> Ignite Configuration: >>> final IgniteConfiguration igniteConfiguration = new >>> IgniteConfiguration(); >>> igniteConfiguration.setIgniteInstanceName("test"); >>> igniteConfiguration.setClientMode(true); >>> igniteConfiguration.setGridLogger(new Slf4jLogger()); >>> igniteConfiguration.setMetricsLogFrequency(0); >>> igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi()); >>> final BinaryConfiguration binaryConfiguration = new >>> BinaryConfiguration(); >>> binaryConfiguration.setCompactFooter(false); >>> igniteConfiguration.setBinaryConfiguration(binaryConfiguration); >>> igniteConfiguration.setPeerClassLoadingEnabled(true); >>> final MemoryPolicyConfiguration memoryPolicyConfiguration = new >>> MemoryPolicyConfiguration(); >>> memoryPolicyConfiguration.setName("3Gb_Region_Eviction"); >>> memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L); >>> memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L); >>> >>> memoryPolicyConfiguration.setPageEvictionMode(DataPageEvicti >>> onMode.RANDOM_2_LRU); >>> final MemoryConfiguration memoryConfiguration = new >>> MemoryConfiguration(); >>> memoryConfiguration.setMemoryPolicies(memoryPolicyConfigurat >>> ion); >>> igniteConfiguration.setMemoryConfiguration(memoryConfiguration); >>> >>> Cache configuration: >>> final CacheConfiguration<String, BinaryObject> cacheConfiguration = new >>> CacheConfiguration<>(); >>> cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC); >>> cacheConfiguration.setStoreKeepBinary(true); >>> cacheConfiguration.setCacheMode(CacheMode.PARTITIONED); >>> cacheConfiguration.setBackups(0); >>> cacheConfiguration.setStatisticsEnabled(false); >>> cacheConfiguration.setName("TestCache"); >>> >>> cacheConfiguration.setReadThrough(true); >>> cacheConfiguration.setWriteThrough(true); >>> >>> cacheConfiguration.setWriteBehindEnabled(true); >>> cacheConfiguration.setWriteBehindFlushFrequency(10000); >>> cacheConfiguration.setWriteBehindFlushSize(0); >>> cacheConfiguration.setWriteBehindFlushThreadCount(2); >>> cacheConfiguration.setWriteBehindBatchSize(10000); >>> >>> >>> final CassandraCacheStoreFactory<String, BinaryObject> >>> cacheStoreFactory = new CassandraCacheStoreFactory<>(); >>> final DataSource dataSource = new DataSource(); >>> dataSource.setContactPoints(contactPoints); >>> dataSource.setReadConsistency("ONE"); >>> dataSource.setWriteConsistency("ONE"); >>> dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new >>> RoundRobinPolicy())); >>> cacheStoreFactory.setDataSource(dataSource); >>> >>> final String CASSANDRA_PERSISTENCE = "<persistence >>> keyspace=\"%s\" >>> table=\"%s\">" + >>> "<keyPersistence class=\"%s\" strategy=\"%s\"/>" + >>> "<valuePersistence class=\"%s\" strategy=\"%s\"/>" + >>> "</persistence>"; >>> final KeyValuePersistenceSettings settings = new >>> KeyValuePersistenceSettings( >>> String.format(CASSANDRA_PERSISTENCE, "test", >>> "test_table", >>> "java.lang.String", "PRIMITIVE", >>> "org.apache.ignite.binary.BinaryObject", >>> "BLOB")); >>> cacheStoreFactory.setPersistenceSettings(settings); >>> cacheConfiguration.setCacheStoreFactory(cacheStoreFactory); >>> >>> When application works some time (hour or more, may be less - from time >>> to >>> time) I see this exceptions on the ignite nodes: >>> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >>> ^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072] >>> ^-- H/N/C [hosts=3, nodes=3, CPUs=96] >>> ^-- CPU [cur=0%, avg=1.86%, GC=0%] >>> ^-- PageMemory [pages=118064] >>> ^-- Heap [used=4800MB, free=53.12%, comm=10240MB] >>> ^-- Non heap [used=78MB, free=-1%, comm=80MB] >>> ^-- Public thread pool [active=0, idle=32, qSize=0] >>> ^-- System thread pool [active=0, idle=32, qSize=0] >>> ^-- Outbound messages queue [size=0] >>> [15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] >>> FreeList >>> [name=null, buckets=256, dataPages=102080, reusePages=0] >>> [15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, refreshing Cassandra session >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra >>> session refreshed >>> [15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for >>> 332ms >>> [15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for >>> 332ms >>> [15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for >>> 229ms >>> [15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for >>> 229ms >>> [15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for >>> 229ms >>> [15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for >>> 229ms >>> [15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for >>> 229ms >>> [15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for >>> 229ms >>> [15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for >>> 229ms >>> [15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, refreshing Cassandra session >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> [15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed >>> [15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep >>> completed >>> [15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep >>> completed >>> [15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep >>> completed >>> [15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep >>> completed >>> [15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep >>> completed >>> [15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2 >>> attempt to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep >>> completed >>> [15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2 >>> attempt to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2 >>> attempt to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2 >>> attempt to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2 >>> attempt to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 >>> attempt >>> to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2 >>> attempt to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep >>> completed >>> [15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep >>> completed >>> [15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2 >>> attempt to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2 >>> attempt to execute Cassandra CQL statement: select "value" from >>> "triggers"."callback" where "key"=?; >>> [15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra >>> session refreshed >>> [15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for >>> 229ms >>> [15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared >>> statement cluster error detected, another thread already fixed the >>> problem >>> com.datastax.driver.core.exceptions.InvalidQueryException: Tried to >>> execute >>> unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may >>> have >>> used a PreparedStatement that was created with another Cluster instance. >>> at >>> com.datastax.driver.core.SessionManager.makeRequestMessage(S >>> essionManager.java:568) >>> at >>> com.datastax.driver.core.SessionManager.executeAsync(Session >>> Manager.java:131) >>> at >>> com.datastax.driver.core.AbstractSession.execute(AbstractSes >>> sion.java:63) >>> at >>> org.apache.ignite.cache.store.cassandra.session.CassandraSes >>> sionImpl.execute(CassandraSessionImpl.java:144) >>> at >>> org.apache.ignite.cache.store.cassandra.CassandraCacheStore. >>> load(CassandraCacheStore.java:168) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheW >>> riteBehindStore.load(GridCacheWriteBehindStore.java:500) >>> at >>> org.apache.ignite.internal.processors.cache.CacheStoreBalanc >>> ingWrapper.load(CacheStoreBalancingWrapper.java:98) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAda >>> pter.java:416) >>> at >>> org.apache.ignite.internal.processors.cache.store.GridCacheS >>> toreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2059) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> $16.call(GridCacheAdapter.java:2057) >>> at >>> org.apache.ignite.internal.processors.cache.GridCacheContext >>> $3.call(GridCacheContext.java:1406) >>> at >>> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader >>> (IgniteUtils.java:6621) >>> at >>> org.apache.ignite.internal.processors.closure.GridClosurePro >>> cessor$2.body(GridClosureProcessor.java:954) >>> at >>> org.apache.ignite.internal.util.worker.GridWorker.run(GridWo >>> rker.java:110) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool >>> Executor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo >>> lExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> After this I have to restart the cluster and application or have >>> "transactions deadlock" message. >>> >>> And I checked: network is fine, no OOME in the ignite and cassandra >>> (cassandra logs don't say anything about it). I understand that this is >>> issue of persistence into cassandra but don't know how can I fix it. And >>> after some attempts ignite node is stopped. I tried to change write >>> behind >>> parameters and disable it at all - but have the same issue. May be I >>> have to >>> see in the other place? May be this is important - i use >>> IgniteDataStreamer >>> for reading values from kafka (earlier I used the simple cache operations >>> and didn't have this problem at all). >>> >>> final IgniteDataStreamer<String, BinaryObject> streamer = >>> ignite.dataStreamer(callCache().getName()); >>> streamer.autoFlushFrequency(5000); >>> streamer.keepBinary(true); >>> streamer.perNodeBufferSize(5120); >>> >>> >>> And one more question about persistence settings. If I'm working with >>> BinaryObjects and ignite doesn't know about my class types, should I use >>> valuePersistence class = "org.apache.ignite.binary.BinaryObject"? I >>> did it >>> because peer class loading doesn't work for my own classes and I don't >>> want >>> to add new jars into all ignite nodes and all my apps. >>> >>> Thanks for help! >>> >>> >>> -- >>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>> >>> >>> >>> >>> -- >>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >>> >>> >>> >>> >>> >>> -- >>> >>> Best regards, >>> Andrey V. Mashenkov >>> >> >> >
