[
https://issues.apache.org/jira/browse/FLINK-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550892#comment-14550892
]
Stephan Ewen commented on FLINK-1952:
-------------------------------------
Next step in the diagnosis: There is apparently something wrong in the
hierarchy of shared slots. The root slot (that is actually allocated from the
TaskManager) is released, while the slot associated with the CoLocation group
of the tasks is still alive.
> Cannot run ConnectedComponents example: Could not allocate a slot on instance
> -----------------------------------------------------------------------------
>
> Key: FLINK-1952
> URL: https://issues.apache.org/jira/browse/FLINK-1952
> Project: Flink
> Issue Type: Bug
> Components: Scheduler
> Affects Versions: 0.9
> Reporter: Robert Metzger
> Priority: Blocker
>
> Steps to reproduce
> {code}
> ./bin/yarn-session.sh -n 350
> {code}
> ... wait until they are connected ...
> {code}
> Number of connected TaskManagers changed to 266. Slots available: 266
> Number of connected TaskManagers changed to 323. Slots available: 323
> Number of connected TaskManagers changed to 334. Slots available: 334
> Number of connected TaskManagers changed to 343. Slots available: 343
> Number of connected TaskManagers changed to 350. Slots available: 350
> {code}
> Start CC
> {code}
> ./bin/flink run -p 350
> ./examples/flink-java-examples-0.9-SNAPSHOT-ConnectedComponents.jar
> {code}
> ---> it runs
> Run KMeans, let it fail with
> {code}
> Failed to deploy the task Map (Map at main(KMeans.java:100)) (1/350) -
> execution #0 to slot SimpleSlot (2)(2)(0) - 182b7661ca9547a84591de940c47a200
> - ALLOCATED/ALIVE: java.io.IOException: Insufficient number of network
> buffers: required 350, but only 254 available. The total number of network
> buffers is currently set to 2048. You can increase this number by setting the
> configuration key 'taskmanager.network.numberOfBuffers'.
> {code}
> ... as expected.
> (I've waited for 10 minutes between the two submissions)
> Starting CC now will fail:
> {code}
> ./bin/flink run -p 350
> ./examples/flink-java-examples-0.9-SNAPSHOT-ConnectedComponents.jar
> {code}
> Error message(s):
> {code}
> Caused by: java.lang.IllegalStateException: Could not schedule consumer
> vertex IterationHead(WorksetIteration (Unnamed Delta Iteration)) (19/350)
> at
> org.apache.flink.runtime.executiongraph.Execution$3.call(Execution.java:479)
> at
> org.apache.flink.runtime.executiongraph.Execution$3.call(Execution.java:469)
> at akka.dispatch.Futures$$anonfun$future$1.apply(Future.scala:94)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
> at
> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
> at
> scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
> ... 4 more
> Caused by:
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> Could not allocate a slot on instance 4a6d761cb084c32310ece1f849556faf @
> cloud-19 - 1 slots - URL:
> akka.tcp://flink@130.149.21.23:51400/user/taskmanager, as required by the
> co-location constraint.
> at
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleTask(Scheduler.java:247)
> at
> org.apache.flink.runtime.jobmanager.scheduler.Scheduler.scheduleImmediately(Scheduler.java:110)
> at
> org.apache.flink.runtime.executiongraph.Execution.scheduleForExecution(Execution.java:262)
> at
> org.apache.flink.runtime.executiongraph.ExecutionVertex.scheduleForExecution(ExecutionVertex.java:436)
> at
> org.apache.flink.runtime.executiongraph.Execution$3.call(Execution.java:475)
> ... 9 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
