[
https://issues.apache.org/jira/browse/FLINK-3805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302025#comment-15302025
]
ASF GitHub Bot commented on FLINK-3805:
---------------------------------------
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/1999#issuecomment-221861420
Thanks for looking into this.
Unfortunately, this change needs some thorough unit tests before it can be
merged.
It looks like a very delicate change. The Bloom Filters are not initialized
under certain conditions. If the hash table still expects them to be there,
this would lead to wrong results. The tests would need to make sure that this
situation does not happen.
> BloomFilter initialized with no memory available
> ------------------------------------------------
>
> Key: FLINK-3805
> URL: https://issues.apache.org/jira/browse/FLINK-3805
> Project: Flink
> Issue Type: Bug
> Components: Distributed Runtime
> Affects Versions: 1.1.0, 1.0.1
> Reporter: Greg Hogan
> Assignee: Rekha Joshi
> Priority: Critical
>
> I flagged this as 1.1.0 and 1.0.1 without checking the latter.
> Link to build, command, and stacktrace follow.
> {{MutableHashTable.initTable}} is calling {{initBloomFilter}} when
> {{this.availableMemory.size()==0}}.
> https://s3.amazonaws.com/apache-flink/flink_bloomfilter_crash.tar.bz2
> ./bin/flink run -class org.apache.flink.graph.examples.TriangleListing
> ~/flink-gelly-examples_2.10-1.1-SNAPSHOT.jar --clip_and_flip false --output
> print --output print --scale 14 --count
> {code}
> org.apache.flink.client.program.ProgramInvocationException: The program
> execution failed: Job execution failed.
> at org.apache.flink.client.program.Client.runBlocking(Client.java:381)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:355)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:315)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:61)
> at
> org.apache.flink.api.java.ExecutionEnvironment.execute(ExecutionEnvironment.java:898)
> at org.apache.flink.api.java.DataSet.collect(DataSet.java:410)
> at org.apache.flink.api.java.DataSet.print(DataSet.java:1605)
> at
> org.apache.flink.graph.examples.TriangleListing.main(TriangleListing.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403)
> at org.apache.flink.client.program.Client.runBlocking(Client.java:248)
> at
> org.apache.flink.client.CliFrontend.executeProgramBlocking(CliFrontend.java:860)
> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:327)
> at
> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1187)
> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1238)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply$mcV$sp(JobManager.scala:805)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:751)
> at
> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$7.apply(JobManager.scala:751)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
> at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.pollAndExecAll(ForkJoinPool.java:1253)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1346)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> Caused by: java.lang.IllegalArgumentException: expectedEntries should be > 0
> at
> org.apache.flink.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
> at
> org.apache.flink.runtime.operators.util.BloomFilter.<init>(BloomFilter.java:53)
> at
> org.apache.flink.runtime.operators.hash.MutableHashTable.initBloomFilter(MutableHashTable.java:823)
> at
> org.apache.flink.runtime.operators.hash.MutableHashTable.initTable(MutableHashTable.java:1183)
> at
> org.apache.flink.runtime.operators.hash.MutableHashTable.buildTableFromSpilledPartition(MutableHashTable.java:887)
> at
> org.apache.flink.runtime.operators.hash.MutableHashTable.prepareNextPartition(MutableHashTable.java:631)
> at
> org.apache.flink.runtime.operators.hash.MutableHashTable.nextRecord(MutableHashTable.java:666)
> at
> org.apache.flink.runtime.operators.hash.NonReusingBuildFirstHashJoinIterator.callWithNextKey(NonReusingBuildFirstHashJoinIterator.java:116)
> at
> org.apache.flink.runtime.operators.JoinDriver.run(JoinDriver.java:216)
> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
> at
> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:345)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:579)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
