I will be also good to know which version of Cassandra driver was used to run into OOM exception.
Igor On Tue, Jun 12, 2018 at 11:39 AM, Igor Rudyak <irud...@gmail.com> wrote: > Denis, > > I don't have ideas right now. First need to create a test to reproduce > this case. Then I'll have some ideas :-) > > Igor > > On Tue, Jun 12, 2018 at 11:26 AM, Denis Magda <dma...@apache.org> wrote: > >> Igor, >> >> Do you have any glues/ideas how to fix it? Is the provided information >> enough for you? >> >> -- >> Denis >> >> On Mon, Jun 11, 2018 at 11:45 PM Igor Rudyak <irud...@gmail.com> wrote: >> >> > Hi Kotamrajuyashasvi, >> > >> > Could you please create a ticket for this in Ignite JIRA? That's the >> > standard process to make improvements/fixes to Ignite. >> > >> > Thanks, >> > Igor Rudyak >> > >> > On Mon, Jun 11, 2018 at 11:36 PM, kotamrajuyashasvi < >> > kotamrajuyasha...@gmail.com> wrote: >> > >> > > Hi >> > > >> > > We are working on an Ignite project with Cassandra as persistent >> storage. >> > > During our tests we faced the continuous cassandra session refresh >> issue. >> > > https://issues.apache.org/jira/browse/IGNITE-8354 >> > > >> > > When we observed the above issue we also ran into OutOfMemory >> Exception. >> > > Though the above issue is solved we ran through the source code to >> find >> > out >> > > the root cause >> > > of OOM. We found one potential cause. >> > > >> > > In org.apache.ignite.cache.store.cassandra.session. >> > > CassandraSessionImpl.java >> > > when refresh() method is invoked to handle Exceptions, new Cluster is >> > build >> > > with same LoadBalancingPolicy Object. We are using RoundRobinPolicy so >> > same >> > > RoundRobinPolicy object would be used while building Cluster when >> > refresh() >> > > is invoked. In RoundRobinPolicy there is a CopyOnWriteArrayList<Host> >> > > liveHosts. When ever init(Cluster cluster, Collection<Host> hosts) is >> > > called >> > > on RoundRobinPolicy it calls liveHosts.addAll(hosts) adding all the >> Host >> > > Object Collection to liveHosts. >> > > When ever Cluster is build during refresh() the Host Collection are >> added >> > > again to the liveHosts of the same RoundRobinPolicy that is used. Thus >> > same >> > > Hosts are added again to liveHosts for every refresh() and the size >> would >> > > grow indefinitely after many refresh() calls causing OOM. Even in the >> > heap >> > > dump post OOM we found huge number of Objects in liveHosts of >> > > RoundRobinPolicy Object. >> > > >> > > IGNITE-8354 has fixed the OOM by preventing unnecessary refresh() but >> > still >> > > does not fix the actual Memory leak caused due to RoundRobinPolicy . >> In a >> > > long run we can have many Cassandra refresh due to some genuine >> reasons >> > and >> > > then we end up with many Hosts in liveHosts of the RoundRobinPolicy >> > Object. >> > > Some possible solutions would be >> > > 1. To use new LoadBalancingPolicy object while building new Cluster >> > during >> > > refresh(). >> > > 2. Somehow clear Objects in liveHosts during refresh(). >> > > >> > > Also there's a work around to use DCAwareRoundRobinPolicy as it uses >> adds >> > > hosts dc wise and adds only if absent. But we are using single >> datacenter >> > > and its not recommended to use DCAwareRoundRobinPolicy when we have >> > single >> > > datacenter. >> > > >> > > I would like to request some one from ignite cassandra module >> development >> > > look into this issue. >> > > >> > > >> > > >> > > -- >> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ >> > > >> > >> > >
