Re: Re[2]: Failed to perform cache operation (cache is stopped)
This is the codebase of AgmsCacheJdbcStoreSessionListner.java
This null pointer occurs due to a datasource bean not found.
The cluster was working fine but what could be the reason for
unavailability of datasource bean in between running cluster.
public class AgmsCacheJdbcStoreSessionListener extends
CacheJdbcStoreSessionListener {
@SpringApplicationContextResource
public void setupDataSourceFromSpringContext(Object appCtx) {
ApplicationContext appContext = (ApplicationContext) appCtx;
setDataSource((DataSource) appContext.getBean("dataSource"));
}
}
I can see one log line that tells us about a problem on the network
side. Is this the possible reason?
2021-10-07 16:28:22,889 197776202 [tcp-disco-msg-worker-[fb67a5fd
XX.XX.XX.XX:47500 crd]-#2%springDataNode%-#69%springDataNode%] WARN
o.a.i.s.d.tcp.TcpDiscoverySpi - Node is out of topology (probably, due
to short-time network problems).
On Mon, Oct 11, 2021 at 7:15 PM stanilovsky evgeny <
estanilovs...@gridgain.com> wrote:
> may be this ?
>
> Caused by: java.lang.NullPointerException: null
> at
> com.xyz.agms.grid.cache.loader.AgmsCacheJdbcStoreSessionListener.setupDataSourceFromSpringContext(AgmsCacheJdbcStoreSessionListener.java:14)
> ... 23 common frames omitted
>
>
> Hi Zhenya,
> CacheStoppedException occurred again on our ignite cluster. I have
> captured logs with IGNITE_QUIET = false.
> There are four core nodes in the cluster and two nodes gone down. I am
> attaching the logs for two failed nodes.
> Please let me know if you need any further details.
>
> Thanks,
> Akash
>
> On Tue, Sep 7, 2021 at 12:19 PM Zhenya Stanilovsky
> wrote:
>
>> plz share somehow these logs, if you have no ideas how to share, you can
>> send it directly to arzamas...@mail.ru
>>
>>
>> Meanwhile I grep the logs with the next occurrence of cache stopped
>> exception,can someone highlight if there is any known bug related to this?
>> I want to check the possible reason for this cache stop exception.
>>
>> On Mon, Sep 6, 2021 at 6:27 PM Akash Shinde > > wrote:
>>
>> Hi Zhenya,
>> Thanks for the quick response.
>> I believe you are talking about ignite instances. There is
>> single ignite using in application.
>> I also want to point out that I am not using destroyCache() method
>> anywhere in application.
>>
>> I will set IGNITE_QUIET = false and try to grep the required logs.
>> This issue occurs by random and there is no way reproduce it.
>>
>> Thanks,
>> Akash
>>
>>
>>
>> On Mon, Sep 6, 2021 at 5:33 PM Zhenya Stanilovsky > > wrote:
>>
>> Hi, Akash
>> You can obtain such a case, for example when you have several instances
>> and :
>> inst1:
>> cache = inst1.getOrCreateCache("cache1");
>>
>> after inst2 destroy calling:
>>
>> cache._some_method_call_
>>
>> inst2:
>> inst2.destroyCache("cache1");
>>
>> or shorter: you still use instance that already destroyed, you can simple
>> grep your logs and found the time when cache has been stopped.
>> probably you need to set IGNITE_QUIET = false.
>> [1] https://ignite.apache.org/docs/latest/logging
>>
>>
>>
>>
>>
>>
>> Hi,
>> I have four server nodes and six client nodes on ignite cluster. I am
>> using ignite 2.10 version.
>> Some operations are failing due to the CacheStoppedException exception on
>> the server nodes. This has become a blocker issue.
>> Could someone please help me to resolve this issue.
>>
>> *Cache Configuration*
>>
>> CacheConfiguration subscriptionCacheCfg = new
>> CacheConfiguration<>(CacheName.SUBSCRIPTION_CACHE.name());
>> subscriptionCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>> subscriptionCacheCfg.setWriteThrough(false);
>> subscriptionCacheCfg.setReadThrough(true);
>> subscriptionCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>> subscriptionCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>> subscriptionCacheCfg.setBackups(*2*);
>> Factory storeFactory =
>> FactoryBuilder.factoryOf(SubscriptionDataLoader.class);
>> subscriptionCacheCfg.setCacheStoreFactory(storeFactory);
>> subscriptionCacheCfg.setIndexedTypes(DefaultDataKey.class,
>> SubscriptionData.class);
>> subscriptionCacheCfg.setSqlIndexMaxInlineSize(47);
>> RendezvousAffinityFunction affinityFunction = new
>> RendezvousAffinityFunction();
>> affinityFunction.setExcludeNeighbors(true);
>> subscriptionCacheCfg.setAffinity(affinityFunction);
>> subscriptionCacheCfg.setStatisticsEnabled(true);
>> subscriptionCacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>>
>>
>> *Exception stack trace*
>>
>> ERROR c.q.dgms.kafka.TaskRequestListener - Error occurred while consuming
>> the object
>> com.baidu.unbiz.fluentvalidator.exception.RuntimeValidateException:
>> java.lang.IllegalStateException: class
>> org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed
>> to perform cache operation (cache is stopped): SUBSCRIPTION_CACHE
>> at
>> com.baidu.unbiz.fluentvalidator.FluentValidator.doValidate(FluentValidator.java:506)
>> at
>> com.baidu.un
Re: Re[2]: Failed to perform cache operation (cache is stopped)
may be this ?
Caused by: java.lang.NullPointerException: null
at
com.xyz.agms.grid.cache.loader.AgmsCacheJdbcStoreSessionListener.setupDataSourceFromSpringContext(AgmsCacheJdbcStoreSessionListener.java:14)
... 23 common frames omitted
Hi Zhenya,
CacheStoppedException occurred again on our ignite cluster. I have
captured logs with IGNITE_QUIET = false.
There are four core nodes in the cluster and two nodes gone down. I am
attaching the logs for two failed nodes.
Please let me know if you need any further details.
Thanks,
Akash
On Tue, Sep 7, 2021 at 12:19 PM Zhenya Stanilovsky
wrote:
plz share somehow these logs, if you have no ideas how to share, you
can send it directly to arzamas...@mail.ru
Meanwhile I grep the logs with the next occurrence of cache stopped
exception,can someone highlight if there >>>is any known bug related
to this?
I want to check the possible reason for this cache stop exception.
On Mon, Sep 6, 2021 at 6:27 PM Akash Shinde
wrote:
Hi Zhenya,
Thanks for the quick response.
I believe you are talking about ignite instances. There is single
ignite using in application.
I also want to point out that I am not using destroyCache() method
anywhere in application.
I will set IGNITE_QUIET = false and try to grep the required logs.
This issue occurs by random and there is no way reproduce it.
Thanks,
Akash
On Mon, Sep 6, 2021 at 5:33 PM Zhenya Stanilovsky
wrote:
Hi, Akash
You can obtain such a case, for example when you have several
instances and :
inst1:
cache = inst1.getOrCreateCache("cache1");
after inst2 destroy calling:
cache._some_method_call_
inst2:
inst2.destroyCache("cache1");
or shorter: you still use instance that already destroyed, you can
simple grep your logs and found the >time when cache has been
stopped.
probably you need to set IGNITE_QUIET = false.
[1] https://ignite.apache.org/docs/latest/logging
Hi,
I have four server nodes and six client nodes on ignite cluster.
I am using ignite 2.10 version.
Some operations are failing due to the CacheStoppedException
exception on the server nodes. This has become a blocker
issue.Could someone please help me to resolve this issue.
Cache Configuration
CacheConfiguration subscriptionCacheCfg = new
CacheConfiguration<>(CacheName.SUBSCRIPTION_CACHE.name());
subscriptionCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
subscriptionCacheCfg.setWriteThrough(false);
subscriptionCacheCfg.setReadThrough(true);
subscriptionCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
subscriptionCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
subscriptionCacheCfg.setBackups(2);
Factory storeFactory =
FactoryBuilder.factoryOf(SubscriptionDataLoader.class);
subscriptionCacheCfg.setCacheStoreFactory(storeFactory);
subscriptionCacheCfg.setIndexedTypes(DefaultDataKey.class,
SubscriptionData.class);
subscriptionCacheCfg.setSqlIndexMaxInlineSize(47);
RendezvousAffinityFunction affinityFunction = new
RendezvousAffinityFunction();
affinityFunction.setExcludeNeighbors(true);
subscriptionCacheCfg.setAffinity(affinityFunction);
subscriptionCacheCfg.setStatisticsEnabled(true);
subscriptionCacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
Exception stack trace
ERROR c.q.dgms.kafka.TaskRequestListener - Error occurred while
consuming the object
com.baidu.unbiz.fluentvalidator.exception.RuntimeValidateException:
java.lang.IllegalStateException: class
org.apache.ignite.internal.processors.cache.CacheStoppedException:
Failed to perform cache operation (cache is stopped):
SUBSCRIPTION_CACHE
at
com.baidu.unbiz.fluentvalidator.FluentValidator.doValidate(FluentValidator.java:506)
at
com.baidu.unbiz.fluentvalidator.FluentValidator.doValidate(FluentValidator.java:461)
at
com.xyz.dgms.service.UserManagementServiceImpl.deleteUser(UserManagementServiceImpl.java:710)
at
com.xyz.dgms.kafka.TaskRequestListener.processRequest(TaskRequestListener.java:190)
at
com.xyz.dgms.kafka.TaskRequestListener.process(TaskRequestListener.java:89)
at
com.xyz.libraries.mom.kafka.consumer.TopicConsumer.lambda$run$3(TopicConsumer.java:162)
at net.jodah.failsafe.Functions$12.call(Functions.java:274)
at net.jodah.failsafe.SyncFailsafe.call(SyncFailsafe.java:145)
at net.jodah.failsafe.SyncFailsafe.run(SyncFailsafe.java:93)
at
com.xyz.libraries.mom.kafka.consumer.TopicConsumer.run(TopicConsumer.java:159)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: class
org.apache.ignite.internal.processors.cache.CacheStoppedException:
Failed to perform cache operation (cache is stopped):
SUBSCRIPTION_CACHE
at
org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(Gr
Re[2]: Failed to perform cache operation (cache is stopped)
plz share somehow these logs, if you have no ideas how to share, you can send
it directly to arzamas...@mail.ru
>Meanwhile I grep the logs with the next occurrence of cache stopped
>exception,can someone highlight if there is any known bug related to this?
>I want to check the possible reason for this cache stop exception.
>On Mon, Sep 6, 2021 at 6:27 PM Akash Shinde < akashshi...@gmail.com > wrote:
>>Hi Zhenya,
>>Thanks for the quick response.
>>I believe you are talking about ignite instances. There is single ignite
>>using in application.
>>I also want to point out that I am not using destroyCache() method anywhere
>>in application.
>>
>>I will set IGNITE_QUIET = false and try to grep the required logs.
>>This issue occurs by random and there is no way reproduce it.
>>
>>Thanks,
>>Akash
>>
>>
>>On Mon, Sep 6, 2021 at 5:33 PM Zhenya Stanilovsky < arzamas...@mail.ru >
>>wrote:
>>>Hi, Akash
>>>You can obtain such a case, for example when you have several instances and :
>>>inst1:
>>>cache = inst1.getOrCreateCache("cache1");
>>>
>>>after inst2 destroy calling:
>>>
>>>cache._some_method_call_
>>>
>>>inst2:
>>> inst2.destroyCache("cache1");
>>>
>>>or shorter: you still use instance that already destroyed, you can simple
>>>grep your logs and found the time when cache has been stopped.
>>>probably you need to set IGNITE_QUIET = false.
>>>[1] https://ignite.apache.org/docs/latest/logging
>>>
>
>>Hi,
>>I have four server nodes and six client nodes on ignite cluster. I am
>>using ignite 2.10 version.
>>Some operations are failing due to the CacheStoppedException exception on
>>the server nodes. This has become a blocker issue.
>>Could someone please help me to resolve this issue.
>>
>>Cache Configuration
>>CacheConfiguration subscriptionCacheCfg = new
>>CacheConfiguration<>(CacheName.SUBSCRIPTION_CACHE.name());
>>subscriptionCacheCfg.setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL);
>>subscriptionCacheCfg.setWriteThrough(false);
>>subscriptionCacheCfg.setReadThrough(true);
>>subscriptionCacheCfg.setRebalanceMode(CacheRebalanceMode.ASYNC);
>>subscriptionCacheCfg.setWriteSynchronizationMode(CacheWriteSynchronizationMode.FULL_SYNC);
>>subscriptionCacheCfg.setBackups(2);
>>Factory storeFactory =
>>FactoryBuilder.factoryOf(SubscriptionDataLoader.class);
>>subscriptionCacheCfg.setCacheStoreFactory(storeFactory);
>>subscriptionCacheCfg.setIndexedTypes(DefaultDataKey.class,
>>SubscriptionData.class);
>>subscriptionCacheCfg.setSqlIndexMaxInlineSize(47);
>>RendezvousAffinityFunction affinityFunction = new
>>RendezvousAffinityFunction();
>>affinityFunction.setExcludeNeighbors(true);
>>subscriptionCacheCfg.setAffinity(affinityFunction);
>>subscriptionCacheCfg.setStatisticsEnabled(true);
>>subscriptionCacheCfg.setPartitionLossPolicy(PartitionLossPolicy.READ_WRITE_SAFE);
>>
>>Exception stack trace
>>
>>ERROR c.q.dgms.kafka.TaskRequestListener - Error occurred while consuming
>>the object
>>com.baidu.unbiz.fluentvalidator.exception.RuntimeValidateException:
>>java.lang.IllegalStateException: class
>>org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed
>>to perform cache operation (cache is stopped): SUBSCRIPTION_CACHE
>>at
>>com.baidu.unbiz.fluentvalidator.FluentValidator.doValidate(FluentValidator.java:506)
>>at
>>com.baidu.unbiz.fluentvalidator.FluentValidator.doValidate(FluentValidator.java:461)
>>at
>>com.xyz.dgms.service.UserManagementServiceImpl.deleteUser(UserManagementServiceImpl.java:710)
>>at
>>com.xyz.dgms.kafka.TaskRequestListener.processRequest(TaskRequestListener.java:190)
>>at
>>com.xyz.dgms.kafka.TaskRequestListener.process(TaskRequestListener.java:89)
>>at
>>com.xyz.libraries.mom.kafka.consumer.TopicConsumer.lambda$run$3(TopicConsumer.java:162)
>>at net.jodah.failsafe.Functions$12.call(Functions.java:274)
>>at net.jodah.failsafe.SyncFailsafe.call(SyncFailsafe.java:145)
>>at net.jodah.failsafe.SyncFailsafe.run(SyncFailsafe.java:93)
>>at
>>com.xyz.libraries.mom.kafka.consumer.TopicConsumer.run(TopicConsumer.java:159)
>>at
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>at
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>at java.lang.Thread.run(Thread.java:748)
>>Caused by: java.lang.IllegalStateException: class
>>org.apache.ignite.internal.processors.cache.CacheStoppedException: Failed
>>to perform cache operation (cache is stopped): SUBSCRIPTION_CACHE
>>at
>>org.apache.ignite.internal.processors.cache.GridCacheGateway.enter(GridCacheGateway.java:166)
>>at
>>org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.onEnter(GatewayProtectedCacheProxy.java:1625)
>>at
>>org.apac
