Hi Shawn, The OOM error occurs on remote server node, there were not sufficient memory to process request, but other threads were not affected by this. Looks like, Ignite was able to recover from the error as it was suppressed and reply to client has been sended.
On Mon, Apr 2, 2018 at 8:22 AM, shawn.du <shawn...@neulion.com.cn> wrote: > Hi Andrey, > > Thanks for your replay. It still confused me for: > 1) for storm worker process, If it is because of OOM and crashed. it > should dump the heap. for I set -XX:+HeapDumpOnOutOfMemoryError > but it didn't. For storm worker, it behaves like a normal fatal error > which make storm worker restart. > 2) It did make ignite server heap dump. for I analyzed the dump, it is > ignite server's. (seeing from PID and java command etc.) > > anyone can explain? Thanks. > > Thanks > Shawn > > On 3/31/2018 01:36，Andrey Mashenkov<andrey.mashen...@gmail.com> > <andrey.mashen...@gmail.com> wrote： > > Hi Shawn, > > 1. Ignite use off heap to store cache entries. Client store no cache data. > Cache in LOCAL mode can be used on client side, and it should use offheap > of course. > > All data client retreive from server will be in offheap > > 2. It is not IgniteOutOfMemory error, but JVM OOM. > So, try to investigate if there is a memory leak in your code. > > On Fri, Mar 30, 2018 at 6:36 AM, shawn.du <shawn...@neulion.com.cn> wrote: > >> Hi, >> >> My Ignite client heap OOM yesterday. This is the first time we encounter >> this issue. >> >> My ignite client colocates within Storm worker process. this issue cause >> storm worker restart. >> I have several questions about it: our ignite version is 2.3.0 >> 1) if ignite in client mode, it use offheap? how to set the max >> onheap/offheap memory to use. >> 2) our storm worker have 8G memory, ignite client print OOM, it doesn't >> trigger storm worker to dump the heap. >> but we get a ignite server's heap dump. ignite server didn't die. >> The ignite server's heap dump is very small. only have 200M. >> which process is OOM? worker or ignite server? >> >> This is logs: Thanks in advance. >> >> Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update >> keys on primary node. >> at org.apache.ignite.internal.pro >> cessors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.distributed.dht.atomic.GridNearAtomicUpdateRes >> ponse.addFailedKeys(GridNearAtomicUpdateResponse.java:342) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache.upda >> teAllAsyncInternal0(GridDhtAtomicCache.java:1784) ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache.upda >> teAllAsyncInternal(GridDhtAtomicCache.java:1627) ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache.proc >> essNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache.acce >> ss$400(GridDhtAtomicCache.java:129) ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache$5. >> apply(GridDhtAtomicCache.java:265) ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache$5. >> apply(GridDhtAtomicCache.java:260) ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.man >> agers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.man >> agers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.man >> agers.communication.GridIoManager.access$4200(GridIoManager.java:126) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.man >> agers.communication.GridIoManager$9.run(GridIoManager.java:1090) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.uti >> l.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[stormjar.jar:?] >> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151] >> Suppressed: java.lang.OutOfMemoryError: Java heap space >> at org.apache.ignite.internal.pro >> cessors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cacheobject.IgniteCacheObjectProcessorImpl. >> toCacheObject(IgniteCacheObjectProcessorImpl.java:191) ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.CacheDataRowAdapter.readIncomplete >> Value(CacheDataRowAdapter.java:404) ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:248) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:174) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:114) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:36) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.database.H2Tree.getRow(H2Tree.java:123) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.database.H2Tree.getRow(H2Tree.java:40) ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.database.H2Tree.compare(H2Tree.java:200) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.database.H2Tree.compare(H2Tree.java:40) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.DataStructure.read(DataStructure.java:319) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1823) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1842) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1752) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.persistence.tree.BPlusTree.remove(BPlusTree.java:1578) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.database.H2TreeIndex.remove(H2TreeIndex.java:216) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:504) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.opt.GridH2Table.update(GridH2Table.java:423) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:580) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2334) >> ~[stormjar.jar:?] >> at org.apache.ignite.internal.pro >> cessors.cache.query.GridCacheQueryManager.remove(GridCacheQu >> eryManager.java:461) >> >> >> >> >> >> Thanks >> Shawn >> >> > > > -- > Best regards, > Andrey V. Mashenkov > > -- Best regards, Andrey V. Mashenkov
