Hi
Sorry did not include the context when I replied
Has anyone been able to resolve this problem, cause I have it too on and
off
In fact it sometimes happens just like that, e.g. I have been running my
Ignite client and then stop it, and then it takes a while and run it again,
and all by a sudden this error shows up. An that is the first thing that
happens, and there is NOT a massive amount of load on Cassandra at that
time. But I have also seen it when I hammer Ignite/Cassandra with
updates/inserts.
This is a deal-breaker for me, I need to understand how to fix this, cause
having this in production is not an option.
-Tobias
Hi!
I'm using the cassandra as persistence store for my caches and have one
issue by handling a huge data (via IgniteDataStreamer from kafka).
Ignite Configuration:
final IgniteConfiguration igniteConfiguration = new IgniteConfiguration();
igniteConfiguration.setIgniteInstanceName("test");
igniteConfiguration.setClientMode(true);
igniteConfiguration.setGridLogger(new Slf4jLogger());
igniteConfiguration.setMetricsLogFrequency(0);
igniteConfiguration.setDiscoverySpi(configureTcpDiscoverySpi());
final BinaryConfiguration binaryConfiguration = new BinaryConfiguration();
binaryConfiguration.setCompactFooter(false);
igniteConfiguration.setBinaryConfiguration(binaryConfiguration);
igniteConfiguration.setPeerClassLoadingEnabled(true);
final MemoryPolicyConfiguration memoryPolicyConfiguration = new
MemoryPolicyConfiguration();
memoryPolicyConfiguration.setName("3Gb_Region_Eviction");
memoryPolicyConfiguration.setInitialSize(1024L * 1024L * 1024L);
memoryPolicyConfiguration.setMaxSize(3072L * 1024L * 1024L);
memoryPolicyConfiguration.setPageEvictionMode(DataPageEvictionMode.RANDOM_2_LRU);
final MemoryConfiguration memoryConfiguration = new MemoryConfiguration();
memoryConfiguration.setMemoryPolicies(memoryPolicyConfiguration);
igniteConfiguration.setMemoryConfiguration(memoryConfiguration);
Cache configuration:
final CacheConfiguration<String, BinaryObject> cacheConfiguration = new
CacheConfiguration<>();
cacheConfiguration.setAtomicityMode(CacheAtomicityMode.ATOMIC);
cacheConfiguration.setStoreKeepBinary(true);
cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setBackups(0);
cacheConfiguration.setStatisticsEnabled(false);
cacheConfiguration.setName("TestCache");
cacheConfiguration.setReadThrough(true);
cacheConfiguration.setWriteThrough(true);
cacheConfiguration.setWriteBehindEnabled(true);
cacheConfiguration.setWriteBehindFlushFrequency(10000);
cacheConfiguration.setWriteBehindFlushSize(0);
cacheConfiguration.setWriteBehindFlushThreadCount(2);
cacheConfiguration.setWriteBehindBatchSize(10000);
final CassandraCacheStoreFactory<String, BinaryObject>
cacheStoreFactory = new CassandraCacheStoreFactory<>();
final DataSource dataSource = new DataSource();
dataSource.setContactPoints(contactPoints);
dataSource.setReadConsistency("ONE");
dataSource.setWriteConsistency("ONE");
dataSource.setLoadBalancingPolicy(new TokenAwarePolicy(new
RoundRobinPolicy()));
cacheStoreFactory.setDataSource(dataSource);
final String CASSANDRA_PERSISTENCE = "<persistence keyspace=\"%s\"
table=\"%s\">" +
"<keyPersistence class=\"%s\" strategy=\"%s\"/>" +
"<valuePersistence class=\"%s\" strategy=\"%s\"/>" +
"</persistence>";
final KeyValuePersistenceSettings settings = new
KeyValuePersistenceSettings(
String.format(CASSANDRA_PERSISTENCE, "test", "test_table",
"java.lang.String", "PRIMITIVE",
"org.apache.ignite.binary.BinaryObject", "BLOB"));
cacheStoreFactory.setPersistenceSettings(settings);
cacheConfiguration.setCacheStoreFactory(cacheStoreFactory);
When application works some time (hour or more, may be less - from time to
time) I see this exceptions on the ignite nodes:
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=4f43d78b, name=null, uptime=00:12:00:072]
^-- H/N/C [hosts=3, nodes=3, CPUs=96]
^-- CPU [cur=0%, avg=1.86%, GC=0%]
^-- PageMemory [pages=118064]
^-- Heap [used=4800MB, free=53.12%, comm=10240MB]
^-- Non heap [used=78MB, free=-1%, comm=80MB]
^-- Public thread pool [active=0, idle=32, qSize=0]
^-- System thread pool [active=0, idle=32, qSize=0]
^-- Outbound messages queue [size=0]
[15:28:28,626][INFO][grid-timeout-worker-#39%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=102080, reusePages=0]
[15:29:02,317][WARNING][sys-#106%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][WARNING][sys-#106%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:04,599][WARNING][sys-#139%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,599][INFO][sys-#106%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][INFO][sys-#139%null%][CassandraCacheStore] Sleeping for 332ms
[15:29:04,600][WARNING][sys-#84%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][WARNING][sys-#120%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#84%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,600][WARNING][sys-#131%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,600][INFO][sys-#120%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][INFO][sys-#131%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#107%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,605][INFO][sys-#107%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,605][WARNING][sys-#140%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#140%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#125%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,606][INFO][sys-#125%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,606][WARNING][sys-#115%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,607][INFO][sys-#115%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:04,607][WARNING][sys-#144%null%][CassandraCacheStore] Prepared
statement cluster error detected, refreshing Cassandra session
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[15:29:04,830][INFO][sys-#84%null%][CassandraCacheStore] Sleep completed
[15:29:04,837][INFO][sys-#125%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#120%null%][CassandraCacheStore] Sleep completed
[15:29:04,841][INFO][sys-#115%null%][CassandraCacheStore] Sleep completed
[15:29:04,836][INFO][sys-#131%null%][CassandraCacheStore] Sleep completed
[15:29:04,835][INFO][sys-#107%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#107%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,835][INFO][sys-#140%null%][CassandraCacheStore] Sleep completed
[15:29:04,847][WARNING][sys-#140%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,846][WARNING][sys-#131%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#115%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#120%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#84%null%][CassandraCacheStore] Trying 2 attempt
to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,843][WARNING][sys-#125%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][INFO][sys-#106%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][INFO][sys-#139%null%][CassandraCacheStore] Sleep completed
[15:29:04,932][WARNING][sys-#106%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:04,932][WARNING][sys-#139%null%][CassandraCacheStore] Trying 2
attempt to execute Cassandra CQL statement: select "value" from
"triggers"."callback" where "key"=?;
[15:29:06,903][WARNING][sys-#144%null%][CassandraCacheStore] Cassandra
session refreshed
[15:29:06,903][INFO][sys-#144%null%][CassandraCacheStore] Sleeping for 229ms
[15:29:06,903][WARNING][sys-#141%null%][CassandraCacheStore] Prepared
statement cluster error detected, another thread already fixed the problem
com.datastax.driver.core.exceptions.InvalidQueryException: Tried to execute
unknown prepared query : 0xcad5832309a512feeb602eec67408130. You may have
used a PreparedStatement that was created with another Cluster instance.
at
com.datastax.driver.core.SessionManager.makeRequestMessage(SessionManager.java:568)
at
com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:131)
at
com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:63)
at
org.apache.ignite.cache.store.cassandra.session.CassandraSessionImpl.execute(CassandraSessionImpl.java:144)
at
org.apache.ignite.cache.store.cassandra.CassandraCacheStore.load(CassandraCacheStore.java:168)
at
org.apache.ignite.internal.processors.cache.store.GridCacheWriteBehindStore.load(GridCacheWriteBehindStore.java:500)
at
org.apache.ignite.internal.processors.cache.CacheStoreBalancingWrapper.load(CacheStoreBalancingWrapper.java:98)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadFromStore(GridCacheStoreManagerAdapter.java:316)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.load(GridCacheStoreManagerAdapter.java:282)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAllFromStore(GridCacheStoreManagerAdapter.java:416)
at
org.apache.ignite.internal.processors.cache.store.GridCacheStoreManagerAdapter.loadAll(GridCacheStoreManagerAdapter.java:382)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2059)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.call(GridCacheAdapter.java:2057)
at
org.apache.ignite.internal.processors.cache.GridCacheContext$3.call(GridCacheContext.java:1406)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6621)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:954)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
After this I have to restart the cluster and application or have
"transactions deadlock" message.
And I checked: network is fine, no OOME in the ignite and cassandra
(cassandra logs don't say anything about it). I understand that this is
issue of persistence into cassandra but don't know how can I fix it. And
after some attempts ignite node is stopped. I tried to change write behind
parameters and disable it at all - but have the same issue. May be I have to
see in the other place? May be this is important - i use IgniteDataStreamer
for reading values from kafka (earlier I used the simple cache operations
and didn't have this problem at all).
final IgniteDataStreamer<String, BinaryObject> streamer =
ignite.dataStreamer(callCache().getName());
streamer.autoFlushFrequency(5000);
streamer.keepBinary(true);
streamer.perNodeBufferSize(5120);
And one more question about persistence settings. If I'm working with
BinaryObjects and ignite doesn't know about my class types, should I use
valuePersistence class = "org.apache.ignite.binary.BinaryObject"? I did it
because peer class loading doesn't work for my own classes and I don't want
to add new jars into all ignite nodes and all my apps.
Thanks for help!
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
