[
https://issues.apache.org/jira/browse/FLINK-945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063449#comment-14063449
]
Robert Metzger commented on FLINK-945:
--------------------------------------
I came across this issue while testing the 0.5.2 bugfix release with a large
KMeans job (27 gb of data). I'm currently trying out if I can reproduce the
error.
{code}
12:21:19,844 ERROR eu.stratosphere.nephele.taskmanager.Task
- java.lang.NullPointerException: in
eu.stratosphere.example.java.clustering.KMeans
$.Centroid null of eu.stratosphere.example.java.clustering.KMeans$.Centroid
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:145)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:58)
at
eu.stratosphere.api.java.typeutils.runtime.AvroSerializer.serialize(AvroSerializer.java:104)
at
eu.stratosphere.pact.runtime.iterative.io.WorksetUpdateOutputCollector.collect(WorksetUpdateOutputCollector.java:52)
at
eu.stratosphere.pact.runtime.task.chaining.ChainedMapDriver.collect(ChainedMapDriver.java:71)
at
eu.stratosphere.pact.runtime.task.ReduceDriver.run(ReduceDriver.java:121)
at
eu.stratosphere.pact.runtime.task.RegularPactTask.run(RegularPactTask.java:505)
at
eu.stratosphere.pact.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:132)
at
eu.stratosphere.pact.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:100)
at
eu.stratosphere.pact.runtime.task.RegularPactTask.invoke(RegularPactTask.java:370)
at
eu.stratosphere.nephele.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:284)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at
eu.stratosphere.core.memory.MemorySegment.putInt(MemorySegment.java:432)
at
eu.stratosphere.core.memory.MemorySegment.putIntBigEndian(MemorySegment.java:480)
at
eu.stratosphere.nephele.services.memorymanager.AbstractPagedOutputView.writeInt(AbstractPagedOutputView.java:257)
at
eu.stratosphere.api.java.typeutils.runtime.DataOutputEncoder.writeInt(DataOutputEncoder.java:55)
at
org.apache.avro.reflect.FieldAccessUnsafe$UnsafeIntField.write(FieldAccessUnsafe.java:124)
at
org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:161)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:104)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:66)
at
org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:143)
... 11 more
{code}
> Iterations sometimes release memory prematurely
> -----------------------------------------------
>
> Key: FLINK-945
> URL: https://issues.apache.org/jira/browse/FLINK-945
> Project: Flink
> Issue Type: Bug
> Components: Iterations, Local Runtime
> Affects Versions: 0.6-incubating, pre-apache-0.5.2
> Reporter: Stephan Ewen
>
> It seems that the iteration tail is sometimes releasing the memory
> prematurely, causing still working operations to fail.
> {code}
> java.lang.NullPointerException
> at eu.stratosphere.core.memory.MemorySegment.put(MemorySegment.java:186)
> at
> eu.stratosphere.nephele.services.memorymanager.AbstractPagedOutputView.writeByte(AbstractPagedOutputView.java:214)
> at
> eu.stratosphere.nephele.services.memorymanager.AbstractPagedOutputView.write(AbstractPagedOutputView.java:167)
> at eu.stratosphere.types.Record.serialize(Record.java:1208)
> at
> eu.stratosphere.api.java.typeutils.runtime.record.RecordSerializer.serialize(RecordSerializer.java:79)
> at
> eu.stratosphere.api.java.typeutils.runtime.record.RecordSerializer.serialize(RecordSerializer.java:27)
> at
> eu.stratosphere.pact.runtime.iterative.io.WorksetUpdateOutputCollector.collect(WorksetUpdateOutputCollector.java:52)
> at
> eu.stratosphere.test.recordJobs.kmeans.KMeansBroadcast$RecomputeClusterCenter.reduce(KMeansBroadcast.java:255)
> at
> eu.stratosphere.pact.runtime.task.GroupReduceDriver.run(GroupReduceDriver.java:103)
> at
> eu.stratosphere.pact.runtime.task.RegularPactTask.run(RegularPactTask.java:505)
> at
> eu.stratosphere.pact.runtime.iterative.task.AbstractIterativePactTask.run(AbstractIterativePactTask.java:132)
> at
> eu.stratosphere.pact.runtime.iterative.task.IterationTailPactTask.run(IterationTailPactTask.java:100)
> at
> eu.stratosphere.pact.runtime.task.RegularPactTask.invoke(RegularPactTask.java:370)
> at
> eu.stratosphere.nephele.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:284)
> at java.lang.Thread.run(Thread.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
