Re: Out of memory streaming to Ignite Apache 2.0

[ilya.kasnacheev]

You may be hitting this scenario from my experience:

As you have three nodes you begin to get deadlocks between load tasks.

These deadlocks cause tasks to be postponed, but real troubles happen when
they survive past 30 seconds and are dumped to logs. There's massive amount
of data in your tasks and they are pretty-printed for logging, leading to
massive spikes in RAM usage and eventually OOM.

The solution here to use smaller batches. How much data do you pass with one
task currently?

Regards,



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Out of memory streaming to Ignite Apache 2.0

[AndreK]

I'm running compute tasks to load csv files into an a cache. Program runs
fine on one node but I get out of memory errors (Java heap) when running on
3 vms. It's 2.2 and I've tried various config options. If I comment out the
stream use then Ignite still seems to stop after several 1000 tasks which
may be a secondary issue. I've tried various config options (setting max
message queue size and FULL_SYNC on the cache).

SEVERE: Failed to map task jobs to nodes due to undeclared user exception
[cause=Java heap space, ses=GridTaskSessionImpl
[taskName=com.geospock.ignited.Server$start$$inlined$mapIndexed$lambda$1,
dep=LocalDeployment [super=GridDeployment [ts=1509377011339, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader@33909752,
clsLdrId=b2d61ed6f51-04a0268a-8450-4c1f-8f26-d1fa1c5893a0, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.geospock.ignited.Server$start$$inlined$mapIndexed$lambda$1,
sesId=b0591ed6f51-04a0268a-8450-4c1f-8f26-d1fa1c5893a0,
startTime=1509377606896, endTime=9223372036854775807,
taskNodeId=04a0268a-8450-4c1f-8f26-d1fa1c5893a0,
clsLdr=sun.misc.Launcher$AppClassLoader@33909752, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
subjId=04a0268a-8450-4c1f-8f26-d1fa1c5893a0, mapFut=IgniteFuture
[orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null,
hash=1266438780]], execName=null]]
java.lang.OutOfMemoryError: Java heap space
at
org.apache.ignite.internal.binary.streams.BinaryHeapOutputStream.arrayCopy(BinaryHeapOutputStream.java:76)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.array(BinaryWriterExImpl.java:213)
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.marshal(GridBinaryMarshaller.java:250)
at
org.apache.ignite.internal.binary.BinaryMarshaller.marshal0(BinaryMarshaller.java:82)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:58)
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:9815)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.sendRequest(GridTaskWorker.java:1353)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.processMappedJobs(GridTaskWorker.java:640)
at
org.apache.ignite.internal.processors.task.GridTaskWorker.body(GridTaskWorker.java:532)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor.startTask(GridTaskProcessor.java:749)
at
org.apache.ignite.internal.processors.task.GridTaskProcessor.execute(GridTaskProcessor.java:448)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor.callAsync(GridClosureProcessor.java:673)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor.callAsync(GridClosureProcessor.java:478)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor.callAsync(GridClosureProcessor.java:464)
at
org.apache.ignite.internal.executor.GridExecutorService.submit(GridExecutorService.java:263)
at com.geospock.ignited.Server.start(Server.kt:210)
at com.geospock.ignited.App.main(App.kt:11)

Oct 30, 2017 3:33:27 PM org.apache.ignite.logger.java.JavaLogger error
SEVERE: Runtime error caught during grid runnable execution: GridTaskWorker
[internal=false, marsh=o.a.i.i.binary.BinaryMarshaller@63372b46,
ses=GridTaskSessionImpl
[taskName=com.geospock.ignited.Server$start$$inlined$mapIndexed$lambda$1,
dep=LocalDeployment [super=GridDeployment [ts=1509377011339, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader@33909752,
clsLdrId=b2d61ed6f51-04a0268a-8450-4c1f-8f26-d1fa1c5893a0, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.geospock.ignited.Server$start$$inlined$mapIndexed$lambda$1,
sesId=b0591ed6f51-04a0268a-8450-4c1f-8f26-d1fa1c5893a0,
startTime=1509377606896, endTime=9223372036854775807,
taskNodeId=04a0268a-8450-4c1f-8f26-d1fa1c5893a0,
clsLdr=sun.misc.Launcher$AppClassLoader@33909752, closed=false, cpSpi=null,
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false,
subjId=04a0268a-8450-4c1f-8f26-d1fa1c5893a0, mapFut=IgniteFuture
[orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null,
hash=1266438780]], execName=null], fut=ComputeTaskInternalFuture
[ses=GridTaskSessionImpl
[taskName=com.geospock.ignited.Server$start$$inlined$mapIndexed$lambda$1,
dep=LocalDeployment [super=GridDeployment [ts=1509377011339, depMode=SHARED,
clsLdr=sun.misc.Launcher$AppClassLoader@33909752,
clsLdrId=b2d61ed6f51-04a0268a-8450-4c1f-8f26-d1fa1c5893a0, userVer=0,
loc=true, sampleClsName=java.lang.String, pendingUndeploy=false,
undeployed=false, usage=0]],
taskClsName=com.geospock.ignited.Server$start$$inlined$mapIndexed$lambda$1,
sesId=b0591ed6f51-04a0268a-8450-4c1f-8f26-d1fa1c5893a0,
startTime=1509377606896,