Hi Dennis, can you give us some details about your setup? e.g. where you are running your job, your input size, the configured memory, etc. It would also be helpful if you could share your code. Getting an out of memory error with just 100 nodes seems weird.
Best, -Vasia. On 6 October 2016 at 13:29, <d...@web.de> wrote: > > Dear ladies and gentlemen, > > I got a problem using Gelly in Flink. Currently I am loading a Virtuoso > Graph into > Flink's Gelly and I want to analyze it for the different paths one can > take to link > the different nodes. Therefore I am using the ScatterGatherIteration. > However, my code just works with about ten to twenty nodes. When I try to > upload > a hundred nodes, the following error occurs: > > Exception in thread "main" org.apache.flink.runtime. > client.JobExecutionException: Job execution failed. > at org.apache.flink.runtime.jobmanager.JobManager$$ > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply$ > mcV$sp(JobManager.scala:822) > at org.apache.flink.runtime.jobmanager.JobManager$$ > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:768) > at org.apache.flink.runtime.jobmanager.JobManager$$ > anonfun$handleMessage$1$$anonfun$applyOrElse$8.apply(JobManager.scala:768) > at scala.concurrent.impl.Future$PromiseCompletingRunnable. > liftedTree1$1(Future.scala:24) > at scala.concurrent.impl.Future$PromiseCompletingRunnable.run( > Future.scala:24) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) > at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec( > AbstractDispatcher.scala:401) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue. > runTask(ForkJoinPool.java:1339) > at scala.concurrent.forkjoin.ForkJoinPool.runWorker( > ForkJoinPool.java:1979) > at scala.concurrent.forkjoin.ForkJoinWorkerThread.run( > ForkJoinWorkerThread.java:107) > Caused by: java.lang.RuntimeException: Memory ran out. Compaction failed. > numPartitions: 32 minPartition: 1 maxPartition: 431 number of overflow > segments: 0 bucketSize: 251 Overall memory: 45613056 Partition memory: > 33685504 Message: null > at org.apache.flink.runtime.operators.hash.CompactingHashTable. > insertRecordIntoPartition(CompactingHashTable.java:457) > at org.apache.flink.runtime.operators.hash.CompactingHashTable. > insertOrReplaceRecord(CompactingHashTable.java:392) > at org.apache.flink.runtime.iterative.io.SolutionSetUpdateOutputCollect > or.collect(SolutionSetUpdateOutputCollector.java:54) > at org.apache.flink.graph.spargel.GatherFunction.setNewVertexValue( > GatherFunction.java:123) > at org.apache.flink.quickstart.PathRank$PathUpdateFunction. > updateVertex(PathRank.java:357) > at org.apache.flink.graph.spargel.ScatterGatherIteration$ > GatherUdfSimpleVV.coGroup(ScatterGatherIteration.java:389) > at org.apache.flink.runtime.operators.CoGroupWithSolutionSetSecondDr > iver.run(CoGroupWithSolutionSetSecondDriver.java:218) > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:486) > at org.apache.flink.runtime.iterative.task.AbstractIterativeTask.run( > AbstractIterativeTask.java:146) > at org.apache.flink.runtime.iterative.task.IterationTailTask.run( > IterationTailTask.java:107) > at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:351) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:584) > at java.lang.Thread.run(Thread.java:745) > > > I tried to google it a bit, and this problems seems to occur often when > using Gelly. I hope you have any ideas or approaches how I can handle this > error. > > Thank you in advance! > All the best, > Dennis >
