The delta iterations are currently still memory sensitive (the last component in flink that is).
We are working on providing better memory usage for them, but the only short term solution is to increase the memory size of the JVMs that run the code, or to use more machines. Stephan On Tue, Jun 24, 2014 at 4:59 PM, Janani Chakkaradhari < janani.cs...@gmail.com> wrote: > Hi, > > I am facing the following exception while using Delta iteration. The > dataset size is 27GB. Any suggestions to overcome this issue? > > 15:14:30,431 ERROR > eu.stratosphere.pact.runtime.task.RegularPactTask - Error in > task code: SolutionSet Delta (142/284) > java.lang.RuntimeException: Memory ran out. Compaction failed. > numPartitions: 32 minPartition: 23 maxPartition: 40 number of overflow > segments: 15 bucketSize: 242 Message: Index: 59, Size: 24 > at > > eu.stratosphere.pact.runtime.hash.CompactingHashTable.insertOrReplaceRecord(CompactingHashTable.java:526) > at > > eu.stratosphere.pact.runtime.iterative.io.SolutionSetUpdateOutputCollector.collect(SolutionSetUpdateOutputCollector.java:55) > at > eu.stratosphere.pact.runtime.task.NoOpDriver.run(NoOpDriver.java:64) > at > > eu.stratosphere.pact.runtime.task.RegularPactTask.run(RegularPactTask.java:505) > at > > eu.stratosphere.pact.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:132) > at > > eu.stratosphere.pact.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:100) > at > > eu.stratosphere.pact.runtime.task.RegularPactTask.invoke(RegularPactTask.java:370) > at > > eu.stratosphere.nephele.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:284) > at java.lang.Thread.run(Thread.java:853) > > Thanks, > Janani >
