Denis, I don't have ideas right now. First need to create a test to reproduce this case. Then I'll have some ideas :-)
Igor On Tue, Jun 12, 2018 at 11:26 AM, Denis Magda <dma...@apache.org> wrote: > Igor, > > Do you have any glues/ideas how to fix it? Is the provided information > enough for you? > > -- > Denis > > On Mon, Jun 11, 2018 at 11:45 PM Igor Rudyak <irud...@gmail.com> wrote: > > > Hi Kotamrajuyashasvi, > > > > Could you please create a ticket for this in Ignite JIRA? That's the > > standard process to make improvements/fixes to Ignite. > > > > Thanks, > > Igor Rudyak > > > > On Mon, Jun 11, 2018 at 11:36 PM, kotamrajuyashasvi < > > kotamrajuyasha...@gmail.com> wrote: > > > > > Hi > > > > > > We are working on an Ignite project with Cassandra as persistent > storage. > > > During our tests we faced the continuous cassandra session refresh > issue. > > > https://issues.apache.org/jira/browse/IGNITE-8354 > > > > > > When we observed the above issue we also ran into OutOfMemory > Exception. > > > Though the above issue is solved we ran through the source code to find > > out > > > the root cause > > > of OOM. We found one potential cause. > > > > > > In org.apache.ignite.cache.store.cassandra.session. > > > CassandraSessionImpl.java > > > when refresh() method is invoked to handle Exceptions, new Cluster is > > build > > > with same LoadBalancingPolicy Object. We are using RoundRobinPolicy so > > same > > > RoundRobinPolicy object would be used while building Cluster when > > refresh() > > > is invoked. In RoundRobinPolicy there is a CopyOnWriteArrayList<Host> > > > liveHosts. When ever init(Cluster cluster, Collection<Host> hosts) is > > > called > > > on RoundRobinPolicy it calls liveHosts.addAll(hosts) adding all the > Host > > > Object Collection to liveHosts. > > > When ever Cluster is build during refresh() the Host Collection are > added > > > again to the liveHosts of the same RoundRobinPolicy that is used. Thus > > same > > > Hosts are added again to liveHosts for every refresh() and the size > would > > > grow indefinitely after many refresh() calls causing OOM. Even in the > > heap > > > dump post OOM we found huge number of Objects in liveHosts of > > > RoundRobinPolicy Object. > > > > > > IGNITE-8354 has fixed the OOM by preventing unnecessary refresh() but > > still > > > does not fix the actual Memory leak caused due to RoundRobinPolicy . > In a > > > long run we can have many Cassandra refresh due to some genuine reasons > > and > > > then we end up with many Hosts in liveHosts of the RoundRobinPolicy > > Object. > > > Some possible solutions would be > > > 1. To use new LoadBalancingPolicy object while building new Cluster > > during > > > refresh(). > > > 2. Somehow clear Objects in liveHosts during refresh(). > > > > > > Also there's a work around to use DCAwareRoundRobinPolicy as it uses > adds > > > hosts dc wise and adds only if absent. But we are using single > datacenter > > > and its not recommended to use DCAwareRoundRobinPolicy when we have > > single > > > datacenter. > > > > > > I would like to request some one from ignite cassandra module > development > > > look into this issue. > > > > > > > > > > > > -- > > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > > > >
