the job manager log probably is more interesting: 2017-09-15 12:47:45,420 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2017-09-15 12:47:45,650 INFO org.apache.flink.runtime.jobmanager.JobManager - -------------------------------------------------------------------------------- 2017-09-15 12:47:45,650 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager (Version: 1.3.2, Rev:0399bee, Date:03.08.2017 @ 10:23:11 UTC) 2017-09-15 12:47:45,650 INFO org.apache.flink.runtime.jobmanager.JobManager - Current user: giordano 2017-09-15 12:47:45,651 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM: Java HotSpot(TM) 64-Bit Server VM - Oracle Corporation - 1.8/25.131-b11 2017-09-15 12:47:45,652 INFO org.apache.flink.runtime.jobmanager.JobManager - Maximum heap size: 491 MiBytes 2017-09-15 12:47:45,652 INFO org.apache.flink.runtime.jobmanager.JobManager - JAVA_HOME: /usr/lib/jvm/java-8-oracle 2017-09-15 12:47:45,658 INFO org.apache.flink.runtime.jobmanager.JobManager - Hadoop version: 2.7.2 2017-09-15 12:47:45,658 INFO org.apache.flink.runtime.jobmanager.JobManager - JVM Options: 2017-09-15 12:47:45,658 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xms512m 2017-09-15 12:47:45,658 INFO org.apache.flink.runtime.jobmanager.JobManager - -Xmx512m 2017-09-15 12:47:45,658 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog.file=/home/giordano/flink-1.3.2/log/flink-giordano-jobmanager-0-giordano-2-2-100-1.log 2017-09-15 12:47:45,658 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlog4j.configuration=file:/home/giordano/flink-1.3.2/conf/log4j.properties 2017-09-15 12:47:45,658 INFO org.apache.flink.runtime.jobmanager.JobManager - -Dlogback.configurationFile=file:/home/giordano/flink-1.3.2/conf/logback.xml 2017-09-15 12:47:45,658 INFO org.apache.flink.runtime.jobmanager.JobManager - Program Arguments: 2017-09-15 12:47:45,659 INFO org.apache.flink.runtime.jobmanager.JobManager - --configDir 2017-09-15 12:47:45,659 INFO org.apache.flink.runtime.jobmanager.JobManager - /home/giordano/flink-1.3.2/conf 2017-09-15 12:47:45,659 INFO org.apache.flink.runtime.jobmanager.JobManager - --executionMode 2017-09-15 12:47:45,659 INFO org.apache.flink.runtime.jobmanager.JobManager - cluster 2017-09-15 12:47:45,659 INFO org.apache.flink.runtime.jobmanager.JobManager - Classpath: /home/giordano/flink-1.3.2/lib/flink-python_2.11-1.3.2.jar:/home/giordano/flink-1.3.2/lib/flin$ 2017-09-15 12:47:45,659 INFO org.apache.flink.runtime.jobmanager.JobManager - -------------------------------------------------------------------------------- 2017-09-15 12:47:45,661 INFO org.apache.flink.runtime.jobmanager.JobManager - Registered UNIX signal handlers for [TERM, HUP, INT] 2017-09-15 12:47:45,947 INFO org.apache.flink.runtime.jobmanager.JobManager - Loading configuration from /home/giordano/flink-1.3.2/conf 2017-09-15 12:47:45,953 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.home, /usr/lib/jvm/java-8-oracle 2017-09-15 12:47:45,953 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost 2017-09-15 12:47:45,953 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2017-09-15 12:47:45,954 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 512 2017-09-15 12:47:45,954 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2017-09-15 12:47:45,954 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false 2017-09-15 12:47:45,955 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081 2017-09-15 12:47:45,956 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend, filesystem 2017-09-15 12:47:45,956 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.fs.checkpointdir, file:///home/flink-checkpoints 2017-09-15 12:47:45,970 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager without high-availability 2017-09-15 12:47:45,973 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager on localhost:6123 with execution mode CLUSTER 2017-09-15 12:47:45,993 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: env.java.home, /usr/lib/jvm/java-8-oracle 2017-09-15 12:47:45,995 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.address, localhost 2017-09-15 12:47:45,995 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.rpc.port, 6123 2017-09-15 12:47:45,995 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.heap.mb, 512 2017-09-15 12:47:45,995 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.numberOfTaskSlots, 2 2017-09-15 12:47:45,996 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: taskmanager.memory.preallocate, false 2017-09-15 12:47:45,996 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: jobmanager.web.port, 8081 2017-09-15 12:47:45,996 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend, filesystem 2017-09-15 12:47:45,996 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: state.backend.fs.checkpointdir, file:///home/flink-checkpoints 2017-09-15 12:47:46,045 INFO org.apache.flink.runtime.security.modules.HadoopModule - Hadoop user set to giordano (auth:SIMPLE) 2017-09-15 12:47:46,209 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor system reachable at localhost:6123 2017-09-15 12:47:46,878 INFO akka.event.slf4j.Slf4jLogger - Slf4jLogger started 2017-09-15 12:47:46,982 INFO Remoting - Starting remoting 2017-09-15 12:47:47,392 INFO Remoting - Remoting started; listening on addresses :[akka.tcp://flink@localhost:6123] 2017-09-15 12:47:47,423 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager web frontend 2017-09-15 12:47:47,433 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager log file: /home/giordano/flink-1.3.2/log/flink-giordano-jobmanager-0-gio$ 2017-09-15 12:47:47,434 INFO org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined location of JobManager stdout file: /home/giordano/flink-1.3.2/log/flink-giordano-jobmanager-0-$ 2017-09-15 12:47:47,434 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-f9816186-7918-4475-8359-c3cafb63559a for the web interface files 2017-09-15 12:47:47,435 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Using directory /tmp/flink-web-75aba058-8587-4181-bf16-ee2e68d9b70c for web frontend JAR file uploads 2017-09-15 12:47:47,853 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Web frontend listening at 0:0:0:0:0:0:0:0:8081 2017-09-15 12:47:47,854 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager actor 2017-09-15 12:47:47,876 INFO org.apache.flink.runtime.blob.BlobServer - Created BLOB server storage directory /tmp/blobStore-9ad4e807-069e-4fb7-88b3-410fbdcb5eb0 2017-09-15 12:47:47,879 INFO org.apache.flink.runtime.blob.BlobServer - Started BLOB server at 0.0.0.0:39682 - max concurrent requests: 50 - max backlog: 1000 2017-09-15 12:47:47,896 INFO org.apache.flink.runtime.metrics.MetricRegistry - No metrics reporter configured, no metrics will be exposed/reported. 2017-09-15 12:47:47,906 INFO org.apache.flink.runtime.jobmanager.MemoryArchivist - Started memory archivist akka://flink/user/archive 2017-09-15 12:47:47,914 INFO org.apache.flink.runtime.jobmanager.JobManager - Starting JobManager at akka.tcp://flink@localhost:6123/user/jobmanager. 2017-09-15 12:47:47,929 INFO org.apache.flink.runtime.webmonitor.WebRuntimeMonitor - Starting with JobManager akka.tcp://flink@localhost:6123/user/jobmanager on port 8081 2017-09-15 12:47:47,930 INFO org.apache.flink.runtime.webmonitor.JobManagerRetriever - New leader reachable under akka.tcp://flink@localhost:6123/user/jobmanager:00000000-0000-0000-0000-0000000$ 2017-09-15 12:47:47,970 INFO org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager - Trying to associate with JobManager leader akka.tcp://flink@localhost:6123/user/jobmanag$ 2017-09-15 12:47:48,010 INFO org.apache.flink.runtime.jobmanager.JobManager - JobManager akka.tcp://flink@localhost:6123/user/jobmanager was granted leadership with leader session ID S$ 2017-09-15 12:47:48,034 INFO org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager - Resource Manager associating with leading JobManager Actor[akka://flink/user/jobmanager#$ 2017-09-15 12:47:51,071 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:51,383 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:51,718 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:52,109 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:52,414 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:53,130 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:54,365 INFO org.apache.flink.runtime.clusterframework.standalone.StandaloneResourceManager - TaskManager cf04d1390ff86aba4d1702ef1a0d2b67 has started. 2017-09-15 12:47:54,368 INFO org.apache.flink.runtime.instance.InstanceManager - Registered TaskManager at giordano-2-2-100-1 (akka.tcp://flink@giordano-2-2-100-1:35127/user/taskmanager) $ 2017-09-15 12:47:54,434 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:55,150 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:58,455 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:47:59,172 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:48:01,650 INFO org.apache.flink.runtime.jobmanager.JobManager - Submitting job df1f60ca168364759c69dbe078544346 (Flink Streaming Job). 2017-09-15 12:48:01,768 INFO org.apache.flink.runtime.jobmanager.JobManager - Using restart strategy FixedDelayRestartStrategy(maxNumberRestartAttempts=1, delayBetweenRestartAttempts=0$ 2017-09-15 12:48:01,782 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers via failover strategy: full graph restart 2017-09-15 12:48:01,796 INFO org.apache.flink.runtime.jobmanager.JobManager - Running initialization on master for job Flink Streaming Job (df1f60ca168364759c69dbe078544346). 2017-09-15 12:48:01,796 INFO org.apache.flink.runtime.jobmanager.JobManager - Successfully ran initialization on master in 0 ms. 2017-09-15 12:48:01,830 INFO org.apache.flink.runtime.jobmanager.JobManager - State backend is set to heap memory (checkpoints to filesystem "file:/home/flink-checkpoints") 2017-09-15 12:48:01,843 INFO org.apache.flink.runtime.jobmanager.JobManager - Scheduling job df1f60ca168364759c69dbe078544346 (Flink Streaming Job). 2017-09-15 12:48:01,844 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Flink Streaming Job (df1f60ca168364759c69dbe078544346) switched from state CREATED to RUNNING. 2017-09-15 12:48:01,861 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Timestamps/Watermarks (1/1) (bc0e95e951deb6680cff372a954950d2) switched from CREA$ 2017-09-15 12:48:01,882 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Unnamed (1/1) (7ecc9cea7b0132f9604bce99545b49bd) switched from CREATED to SCHEDULED. 2017-09-15 12:48:01,883 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Learn -> Select -> Process -> (Sink: Cassandra Sink, Sink: Cassandra Sink) (1/1) (d24f111f9720c8e4df77f67a$ 2017-09-15 12:48:01,895 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Timestamps/Watermarks (1/1) (bc0e95e951deb6680cff372a954950d2) switched from SCHE$ 2017-09-15 12:48:01,912 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Source: Custom Source -> Timestamps/Watermarks (1/1) (attempt #0) to giordano-2-2-100-1 2017-09-15 12:48:01,933 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Unnamed (1/1) (7ecc9cea7b0132f9604bce99545b49bd) switched from SCHEDULED to DEPLOYING. 2017-09-15 12:48:01,933 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Map -> Sink: Unnamed (1/1) (attempt #0) to giordano-2-2-100-1 2017-09-15 12:48:01,941 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Learn -> Select -> Process -> (Sink: Cassandra Sink, Sink: Cassandra Sink) (1/1) (d24f111f9720c8e4df77f67a$ 2017-09-15 12:48:01,941 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Learn -> Select -> Process -> (Sink: Cassandra Sink, Sink: Cassandra Sink) (1/1) (attempt #0) to$ 2017-09-15 12:48:03,018 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Learn -> Select -> Process -> (Sink: Cassandra Sink, Sink: Cassandra Sink) (1/1) (d24f111f9720c8e4df77f67a$ 2017-09-15 12:48:03,038 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Timestamps/Watermarks (1/1) (bc0e95e951deb6680cff372a954950d2) switched from DEPL$ 2017-09-15 12:48:03,046 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Map -> Sink: Unnamed (1/1) (7ecc9cea7b0132f9604bce99545b49bd) switched from DEPLOYING to RUNNING. 2017-09-15 12:48:06,474 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:48:07,189 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:48:22,495 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ $flink@giordano-2-2-100-1:6123/]] arriving at [akka.tcp://flink@giordano-2-2-100-1:6123] inbound addresses are [akka.tcp://flink@localhost:6123] 2017-09-15 12:48:52,506 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:48:53,212 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:49:22,524 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:49:23,230 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:49:52,544 ERROR akka.remote.EndpointWriter - dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp://flink@$ 2017-09-15 12:49:59,410 WARN akka.remote.RemoteWatcher - Detected unreachable: [akka.tcp://flink@giordano-2-2-100-1:35127] 2017-09-15 12:49:59,445 INFO org.apache.flink.runtime.jobmanager.JobManager - Task manager akka.tcp://flink@giordano-2-2-100-1:35127/user/taskmanager terminated. 2017-09-15 12:49:59,446 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom Source -> Timestamps/Watermarks (1/1) (bc0e95e951deb6680cff372a954950d2) switched from RUNN$ java.lang.Exception: TaskManager was lost/killed: cf04d1390ff86aba4d1702ef1a0d2b67 @ giordano-2-2-100-1 (dataPort=36806)
it tag as unreachable the task manager (who reside on the same node...) -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/