Igor, Do you have any glues/ideas how to fix it? Is the provided information enough for you?
-- Denis On Mon, Jun 11, 2018 at 11:45 PM Igor Rudyak <irud...@gmail.com> wrote: > Hi Kotamrajuyashasvi, > > Could you please create a ticket for this in Ignite JIRA? That's the > standard process to make improvements/fixes to Ignite. > > Thanks, > Igor Rudyak > > On Mon, Jun 11, 2018 at 11:36 PM, kotamrajuyashasvi < > kotamrajuyasha...@gmail.com> wrote: > > > Hi > > > > We are working on an Ignite project with Cassandra as persistent storage. > > During our tests we faced the continuous cassandra session refresh issue. > > https://issues.apache.org/jira/browse/IGNITE-8354 > > > > When we observed the above issue we also ran into OutOfMemory Exception. > > Though the above issue is solved we ran through the source code to find > out > > the root cause > > of OOM. We found one potential cause. > > > > In org.apache.ignite.cache.store.cassandra.session. > > CassandraSessionImpl.java > > when refresh() method is invoked to handle Exceptions, new Cluster is > build > > with same LoadBalancingPolicy Object. We are using RoundRobinPolicy so > same > > RoundRobinPolicy object would be used while building Cluster when > refresh() > > is invoked. In RoundRobinPolicy there is a CopyOnWriteArrayList<Host> > > liveHosts. When ever init(Cluster cluster, Collection<Host> hosts) is > > called > > on RoundRobinPolicy it calls liveHosts.addAll(hosts) adding all the Host > > Object Collection to liveHosts. > > When ever Cluster is build during refresh() the Host Collection are added > > again to the liveHosts of the same RoundRobinPolicy that is used. Thus > same > > Hosts are added again to liveHosts for every refresh() and the size would > > grow indefinitely after many refresh() calls causing OOM. Even in the > heap > > dump post OOM we found huge number of Objects in liveHosts of > > RoundRobinPolicy Object. > > > > IGNITE-8354 has fixed the OOM by preventing unnecessary refresh() but > still > > does not fix the actual Memory leak caused due to RoundRobinPolicy . In a > > long run we can have many Cassandra refresh due to some genuine reasons > and > > then we end up with many Hosts in liveHosts of the RoundRobinPolicy > Object. > > Some possible solutions would be > > 1. To use new LoadBalancingPolicy object while building new Cluster > during > > refresh(). > > 2. Somehow clear Objects in liveHosts during refresh(). > > > > Also there's a work around to use DCAwareRoundRobinPolicy as it uses adds > > hosts dc wise and adds only if absent. But we are using single datacenter > > and its not recommended to use DCAwareRoundRobinPolicy when we have > single > > datacenter. > > > > I would like to request some one from ignite cassandra module development > > look into this issue. > > > > > > > > -- > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > >
