I enclosed all logs from the run and for this run I used parallelism
one. However, for other runs I checked and found that all parallel
workers were working properly. Is there a simple way to get profiling
information in Flink?
Best,
Habib
On 10/31/2019 2:54 AM, Zhenghua Gao wrote:
I think more runtime information would help figure outwheretheproblem is.
1) how many parallelisms actually working
2) the metrics for each operator
3) the jvm profiling information, etc
*Best Regards,*
*Zhenghua Gao*
On Wed, Oct 30, 2019 at 8:25 PM Habib Mostafaei
<ha...@inet.tu-berlin.de <mailto:ha...@inet.tu-berlin.de>> wrote:
Thanks Gao for the reply. I used the parallelism parameter with
different values like 6 and 8 but still the execution time is not
comparable with a single threaded python script. What would be the
reasonable value for the parallelism?
Best,
Habib
On 10/30/2019 1:17 PM, Zhenghua Gao wrote:
The reason might be the parallelism of your task is only 1,
that's too low.
See [1] to specify proper parallelism for your job, and the
execution time should be reduced significantly.
[1]
https://ci.apache.org/projects/flink/flink-docs-stable/dev/parallel.html
*Best Regards,*
*Zhenghua Gao*
On Tue, Oct 29, 2019 at 9:27 PM Habib Mostafaei
<ha...@inet.tu-berlin.de <mailto:ha...@inet.tu-berlin.de>> wrote:
Hi all,
I am running Flink on a standalone cluster and getting very long
execution time for the streaming queries like WordCount for a
fixed text
file. My VM runs on a Debian 10 with 16 cpu cores and 32GB of
RAM. I
have a text file with size of 2GB. When I run the Flink on a
standalone
cluster, i.e., one JobManager and one taskManager with 25GB
of heapsize,
it took around two hours to finish counting this file while a
simple
python script can do it in around 7 minutes. Just wondering
what is
wrong with my setup. I ran the experiments on a cluster with six
taskManagers, but I still get very long execution time like
25 minutes
or so. I tried to increase the JVM heap size to have lower
execution
time but it did not help. I attached the log file and the Flink
configuration file to this email.
Best,
Habib
2019-10-30 15:59:47,117 INFO org.apache.flink.client.cli.CliFrontend
-
--------------------------------------------------------------------------------
2019-10-30 15:59:47,121 INFO org.apache.flink.client.cli.CliFrontend
- Starting Command Line Client (Version: 1.8.2, Rev:6322618,
Date:04.09.2019 @ 22:07:41 CST)
2019-10-30 15:59:47,121 INFO org.apache.flink.client.cli.CliFrontend
- OS current user: xxx
2019-10-30 15:59:47,122 INFO org.apache.flink.client.cli.CliFrontend
- Current Hadoop/Kerberos user: <no hadoop dependency found>
2019-10-30 15:59:47,122 INFO org.apache.flink.client.cli.CliFrontend
- JVM: OpenJDK 64-Bit Server VM - AdoptOpenJDK - 1.8/25.232-b09
2019-10-30 15:59:47,123 INFO org.apache.flink.client.cli.CliFrontend
- Maximum heap size: 7143 MiBytes
2019-10-30 15:59:47,123 INFO org.apache.flink.client.cli.CliFrontend
- JAVA_HOME: /usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/
2019-10-30 15:59:47,124 INFO org.apache.flink.client.cli.CliFrontend
- No Hadoop Dependency available
2019-10-30 15:59:47,124 INFO org.apache.flink.client.cli.CliFrontend
- JVM Options:
2019-10-30 15:59:47,124 INFO org.apache.flink.client.cli.CliFrontend
- -Dlog.file=/home/xxx/flink-1.8.2/log/flink-xxx-client-xxx.log
2019-10-30 15:59:47,125 INFO org.apache.flink.client.cli.CliFrontend
-
-Dlog4j.configuration=file:/home/xxx/flink-1.8.2/conf/log4j-cli.properties
2019-10-30 15:59:47,125 INFO org.apache.flink.client.cli.CliFrontend
-
-Dlogback.configurationFile=file:/home/xxx/flink-1.8.2/conf/logback.xml
2019-10-30 15:59:47,125 INFO org.apache.flink.client.cli.CliFrontend
- Program Arguments:
2019-10-30 15:59:47,126 INFO org.apache.flink.client.cli.CliFrontend
- run
2019-10-30 15:59:47,126 INFO org.apache.flink.client.cli.CliFrontend
- examples/streaming/WordCount.jar
2019-10-30 15:59:47,126 INFO org.apache.flink.client.cli.CliFrontend
- --input
2019-10-30 15:59:47,127 INFO org.apache.flink.client.cli.CliFrontend
- ../test.txt
2019-10-30 15:59:47,127 INFO org.apache.flink.client.cli.CliFrontend
- Classpath:
/home/xxx/flink-1.8.2/lib/log4j-1.2.17.jar:/home/xxx/flink-1.8.2/lib/slf4j-log4j12-1.7.15.jar:/home/xxx/flink-1.8.2/lib/flink-dist_2.12-1.8.2.jar:::
2019-10-30 15:59:47,127 INFO org.apache.flink.client.cli.CliFrontend
-
--------------------------------------------------------------------------------
2019-10-30 15:59:47,144 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2019-10-30 15:59:47,145 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-10-30 15:59:47,145 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 25000m
2019-10-30 15:59:47,146 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 25000m
2019-10-30 15:59:47,146 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2019-10-30 15:59:47,147 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2019-10-30 15:59:47,160 WARN org.apache.flink.client.cli.CliFrontend
- Could not load CLI class
org.apache.flink.yarn.cli.FlinkYarnSessionCli.
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1230)
at
org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1190)
at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1115)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 5 more
2019-10-30 15:59:47,184 INFO org.apache.flink.core.fs.FileSystem
- Hadoop is not in the classpath/dependencies. The extended set of
supported File Systems via Hadoop is not available.
2019-10-30 15:59:48,640 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create
Hadoop Security Module because Hadoop cannot be found in the Classpath.
2019-10-30 15:59:48,695 INFO org.apache.flink.runtime.security.SecurityUtils
- Cannot install HadoopSecurityContext because Hadoop cannot be
found in the Classpath.
2019-10-30 15:59:48,697 INFO org.apache.flink.client.cli.CliFrontend
- Running 'run' command.
2019-10-30 15:59:48,709 INFO org.apache.flink.client.cli.CliFrontend
- Building program from JAR file
2019-10-30 15:59:48,808 INFO org.apache.flink.configuration.Configuration
- Config uses fallback configuration key 'jobmanager.rpc.address'
instead of key 'rest.address'
2019-10-30 15:59:49,311 INFO org.apache.flink.runtime.rest.RestClient
- Rest client endpoint started.
2019-10-30 15:59:49,318 INFO org.apache.flink.client.cli.CliFrontend
- Starting execution of program
2019-10-30 15:59:49,319 INFO
org.apache.flink.client.program.rest.RestClusterClient - Starting
program in interactive mode (detached: false)
2019-10-30 15:59:49,370 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2019-10-30 15:59:49,370 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-10-30 15:59:49,371 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 25000m
2019-10-30 15:59:49,371 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 25000m
2019-10-30 15:59:49,372 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2019-10-30 15:59:49,372 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2019-10-30 15:59:49,389 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2019-10-30 15:59:49,390 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-10-30 15:59:49,390 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 25000m
2019-10-30 15:59:49,391 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 25000m
2019-10-30 15:59:49,391 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2019-10-30 15:59:49,391 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2019-10-30 15:59:49,466 INFO org.apache.flink.api.java.typeutils.TypeExtractor
- class
org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit does
not contain a setter for field modificationTime
2019-10-30 15:59:49,467 INFO org.apache.flink.api.java.typeutils.TypeExtractor
- Class class
org.apache.flink.streaming.api.functions.source.TimestampedFileInputSplit
cannot be used as a POJO type because not all fields are valid POJO fields, and
must be processed as GenericType. Please read the Flink documentation on "Data
Types & Serialization" for details of the effect on performance.
2019-10-30 15:59:49,827 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2019-10-30 15:59:49,827 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-10-30 15:59:49,827 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 25000m
2019-10-30 15:59:49,828 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 25000m
2019-10-30 15:59:49,828 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2019-10-30 15:59:49,829 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2019-10-30 15:59:49,834 INFO
org.apache.flink.client.program.rest.RestClusterClient - Submitting job
7cde77256615ceb13a1e74010af21c49 (detached: false).
2019-10-30 17:45:11,942 INFO org.apache.flink.client.cli.CliFrontend
- Program execution finished
2019-10-30 17:45:11,959 INFO org.apache.flink.runtime.rest.RestClient
- Shutting down rest endpoint.
2019-10-30 17:45:11,963 INFO org.apache.flink.runtime.rest.RestClient
- Rest endpoint shutdown complete.
2019-10-30 15:59:38,617 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2019-10-30 15:59:38,620 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting
StandaloneSessionClusterEntrypoint (Version: 1.8.2, Rev:6322618,
Date:04.09.2019 @ 22:07:41 CST)
2019-10-30 15:59:38,621 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - OS current
user: xxx
2019-10-30 15:59:38,621 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Current
Hadoop/Kerberos user: <no hadoop dependency found>
2019-10-30 15:59:38,622 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM: OpenJDK
64-Bit Server VM - AdoptOpenJDK - 1.8/25.232-b09
2019-10-30 15:59:38,622 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Maximum heap
size: 23958 MiBytes
2019-10-30 15:59:38,623 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JAVA_HOME:
/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/
2019-10-30 15:59:38,623 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - No Hadoop
Dependency available
2019-10-30 15:59:38,623 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - JVM Options:
2019-10-30 15:59:38,624 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xms25000m
2019-10-30 15:59:38,624 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - -Xmx25000m
2019-10-30 15:59:38,625 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog.file=/home/xxx/flink-1.8.2/log/flink-xxx-standalonesession-0-xxx.log
2019-10-30 15:59:38,625 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlog4j.configuration=file:/home/xxx/flink-1.8.2/conf/log4j.properties
2019-10-30 15:59:38,625 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
-Dlogback.configurationFile=file:/home/xxx/flink-1.8.2/conf/logback.xml
2019-10-30 15:59:38,626 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Program
Arguments:
2019-10-30 15:59:38,626 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - --configDir
2019-10-30 15:59:38,626 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
/home/xxx/flink-1.8.2/conf
2019-10-30 15:59:38,627 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--executionMode
2019-10-30 15:59:38,627 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - cluster
2019-10-30 15:59:38,627 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Classpath:
/home/xxx/flink-1.8.2/lib/log4j-1.2.17.jar:/home/xxx/flink-1.8.2/lib/slf4j-log4j12-1.7.15.jar:/home/xxx/flink-1.8.2/lib/flink-dist_2.12-1.8.2.jar:::
2019-10-30 15:59:38,628 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint -
--------------------------------------------------------------------------------
2019-10-30 15:59:38,631 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Registered UNIX
signal handlers for [TERM, HUP, INT]
2019-10-30 15:59:38,676 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2019-10-30 15:59:38,677 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-10-30 15:59:38,678 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 25000m
2019-10-30 15:59:38,678 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 25000m
2019-10-30 15:59:38,679 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2019-10-30 15:59:38,679 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2019-10-30 15:59:38,922 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting
StandaloneSessionClusterEntrypoint.
2019-10-30 15:59:38,923 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install default
filesystem.
2019-10-30 15:59:38,937 INFO org.apache.flink.core.fs.FileSystem
- Hadoop is not in the classpath/dependencies. The extended set of
supported File Systems via Hadoop is not available.
2019-10-30 15:59:38,956 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Install
security context.
2019-10-30 15:59:38,979 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create
Hadoop Security Module because Hadoop cannot be found in the Classpath.
2019-10-30 15:59:39,006 INFO org.apache.flink.runtime.security.SecurityUtils
- Cannot install HadoopSecurityContext because Hadoop cannot be
found in the Classpath.
2019-10-30 15:59:39,007 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Initializing
cluster services.
2019-10-30 15:59:40,196 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start
actor system at localhost:6123
2019-10-30 15:59:41,453 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2019-10-30 15:59:41,589 INFO akka.remote.Remoting
- Starting remoting
2019-10-30 15:59:41,983 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@localhost:6123]
2019-10-30 15:59:42,039 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system
started at akka.tcp://flink@localhost:6123
2019-10-30 15:59:42,082 INFO org.apache.flink.configuration.Configuration
- Config uses fallback configuration key 'jobmanager.rpc.address'
instead of key 'rest.address'
2019-10-30 15:59:42,100 INFO org.apache.flink.runtime.blob.BlobServer
- Created BLOB server storage directory
/tmp/blobStore-19717373-736d-4713-a247-ca164826068f
2019-10-30 15:59:42,106 INFO org.apache.flink.runtime.blob.BlobServer
- Started BLOB server at 0.0.0.0:42005 - max concurrent requests:
50 - max backlog: 1000
2019-10-30 15:59:42,140 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics
reporter configured, no metrics will be exposed/reported.
2019-10-30 15:59:42,144 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Trying to start
actor system at localhost:0
2019-10-30 15:59:42,210 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2019-10-30 15:59:42,239 INFO akka.remote.Remoting
- Starting remoting
2019-10-30 15:59:42,281 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink-metrics@localhost:33241]
2019-10-30 15:59:42,287 INFO
org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system
started at akka.tcp://flink-metrics@localhost:33241
2019-10-30 15:59:42,298 INFO
org.apache.flink.runtime.dispatcher.FileArchivedExecutionGraphStore -
Initializing FileArchivedExecutionGraphStore: Storage directory
/tmp/executionGraphStore-cc382dd9-e716-433b-a241-c6e9672c7a5e, expiration time
3600000, maximum cache size 52428800 bytes.
2019-10-30 15:59:42,366 INFO org.apache.flink.runtime.blob.TransientBlobCache
- Created BLOB cache storage directory
/tmp/blobStore-4b0a394a-0e2e-4006-b7a1-26c15e58919e
2019-10-30 15:59:42,414 INFO org.apache.flink.configuration.Configuration
- Config uses fallback configuration key 'jobmanager.rpc.address'
instead of key 'rest.address'
2019-10-30 15:59:42,417 WARN
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Upload
directory /tmp/flink-web-e190eefd-84a1-4ab7-a034-fd3e31708280/flink-web-upload
does not exist, or has been deleted externally. Previously uploaded files are
no longer available.
2019-10-30 15:59:42,419 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Created
directory /tmp/flink-web-e190eefd-84a1-4ab7-a034-fd3e31708280/flink-web-upload
for file uploads.
2019-10-30 15:59:42,422 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Starting rest
endpoint.
2019-10-30 15:59:42,943 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined
location of main cluster component log file:
/home/xxx/flink-1.8.2/log/flink-xxx-standalonesession-0-xxx.log
2019-10-30 15:59:42,943 INFO
org.apache.flink.runtime.webmonitor.WebMonitorUtils - Determined
location of main cluster component stdout file:
/home/xxx/flink-1.8.2/log/flink-xxx-standalonesession-0-xxx.out
2019-10-30 15:59:43,302 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Rest endpoint
listening at localhost:8081
2019-10-30 15:59:43,305 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint -
http://localhost:8081 was granted leadership with
leaderSessionID=00000000-0000-0000-0000-000000000000
2019-10-30 15:59:43,305 INFO
org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint - Web frontend
listening at http://localhost:8081.
2019-10-30 15:59:43,464 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at
akka://flink/user/resourcemanager .
2019-10-30 15:59:43,519 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at
akka://flink/user/dispatcher .
2019-10-30 15:59:43,564 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
ResourceManager akka.tcp://flink@localhost:6123/user/resourcemanager was
granted leadership with fencing token 00000000000000000000000000000000
2019-10-30 15:59:43,571 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager - Starting
the SlotManager.
2019-10-30 15:59:43,585 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Dispatcher
akka.tcp://flink@localhost:6123/user/dispatcher was granted leadership with
fencing token 00000000-0000-0000-0000-000000000000
2019-10-30 15:59:43,595 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Recovering all
persisted jobs.
2019-10-30 15:59:47,248 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Ignoring
outdated TaskExecutorGateway connection.
2019-10-30 15:59:47,273 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
Registering TaskManager with ResourceID 5e3d3f309eedf96ec861de138c26e0ac
(akka.tcp://flink@130.149.221.178:34019/user/taskmanager_0) at ResourceManager
2019-10-30 15:59:51,295 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Received
JobGraph submission 7cde77256615ceb13a1e74010af21c49 (Streaming WordCount).
2019-10-30 15:59:51,296 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting job
7cde77256615ceb13a1e74010af21c49 (Streaming WordCount).
2019-10-30 15:59:51,337 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.jobmaster.JobMaster at akka://flink/user/jobmanager_0 .
2019-10-30 15:59:51,356 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Initializing job Streaming WordCount
(7cde77256615ceb13a1e74010af21c49).
2019-10-30 15:59:51,385 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Using restart strategy NoRestartStrategy for Streaming WordCount
(7cde77256615ceb13a1e74010af21c49).
2019-10-30 15:59:51,431 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job recovers
via failover strategy: full graph restart
2019-10-30 15:59:51,503 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Running initialization on master for job Streaming WordCount
(7cde77256615ceb13a1e74010af21c49).
2019-10-30 15:59:51,503 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Successfully ran initialization on master in 0 ms.
2019-10-30 15:59:51,551 INFO org.apache.flink.runtime.jobmaster.JobMaster
- No state backend has been configured, using default (Memory /
JobManager) MemoryStateBackend (data in heap memory / checkpoints to
JobManager) (checkpoints: 'null', savepoints: 'null', asynchronous: TRUE,
maxStateSize: 5242880)
2019-10-30 15:59:51,573 INFO
org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager
runner for job Streaming WordCount (7cde77256615ceb13a1e74010af21c49) was
granted leadership with session id 00000000-0000-0000-0000-000000000000 at
akka.tcp://flink@localhost:6123/user/jobmanager_0.
2019-10-30 15:59:51,580 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Starting execution of job Streaming WordCount
(7cde77256615ceb13a1e74010af21c49) under job master id
00000000000000000000000000000000.
2019-10-30 15:59:51,583 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Streaming
WordCount (7cde77256615ceb13a1e74010af21c49) switched from state CREATED to
RUNNING.
2019-10-30 15:59:51,590 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom
File Source (1/1) (d50e407f6c4eb1a68684062a4f07517b) switched from CREATED to
SCHEDULED.
2019-10-30 15:59:51,618 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot serve
slot request, no ResourceManager connected. Adding as pending request
[SlotRequestId{60266822abf4236d8e336ced6ccb93c1}]
2019-10-30 15:59:51,631 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Split Reader:
Custom File Source -> Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4)
switched from CREATED to SCHEDULED.
2019-10-30 15:59:51,633 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Keyed
Aggregation -> Sink: Print to Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43)
switched from CREATED to SCHEDULED.
2019-10-30 15:59:51,638 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Connecting to ResourceManager
akka.tcp://flink@localhost:6123/user/resourcemanager(00000000000000000000000000000000)
2019-10-30 15:59:51,647 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Resolved ResourceManager address, beginning registration
2019-10-30 15:59:51,647 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Registration at ResourceManager attempt 1 (timeout=100ms)
2019-10-30 15:59:51,651 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
Registering job manager
00000000000000000000000000000...@akka.tcp://flink@localhost:6123/user/jobmanager_0
for job 7cde77256615ceb13a1e74010af21c49.
2019-10-30 15:59:51,664 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
Registered job manager
00000000000000000000000000000...@akka.tcp://flink@localhost:6123/user/jobmanager_0
for job 7cde77256615ceb13a1e74010af21c49.
2019-10-30 15:59:51,670 INFO org.apache.flink.runtime.jobmaster.JobMaster
- JobManager successfully registered at ResourceManager, leader id:
00000000000000000000000000000000.
2019-10-30 15:59:51,671 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Requesting new
slot [SlotRequestId{60266822abf4236d8e336ced6ccb93c1}] and profile
ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1, directMemoryInMB=0,
nativeMemoryInMB=0, networkMemoryInMB=0} from resource manager.
2019-10-30 15:59:51,675 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager - Request
slot with profile ResourceProfile{cpuCores=-1.0, heapMemoryInMB=-1,
directMemoryInMB=0, nativeMemoryInMB=0, networkMemoryInMB=0} for job
7cde77256615ceb13a1e74010af21c49 with allocation id
3618194b1249b0a728ed674b7ddb0e6d.
2019-10-30 15:59:51,840 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom
File Source (1/1) (d50e407f6c4eb1a68684062a4f07517b) switched from SCHEDULED to
DEPLOYING.
2019-10-30 15:59:51,841 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying
Source: Custom File Source (1/1) (attempt #0) to
5e3d3f309eedf96ec861de138c26e0ac @ xxx.inet.tu-berlin.de (dataPort=32991)
2019-10-30 15:59:51,850 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Split Reader:
Custom File Source -> Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4)
switched from SCHEDULED to DEPLOYING.
2019-10-30 15:59:51,851 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Split
Reader: Custom File Source -> Flat Map (1/1) (attempt #0) to
5e3d3f309eedf96ec861de138c26e0ac @ xxx.inet.tu-berlin.de (dataPort=32991)
2019-10-30 15:59:51,859 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Keyed
Aggregation -> Sink: Print to Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43)
switched from SCHEDULED to DEPLOYING.
2019-10-30 15:59:51,860 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying Keyed
Aggregation -> Sink: Print to Std. Out (1/1) (attempt #0) to
5e3d3f309eedf96ec861de138c26e0ac @ xxx.inet.tu-berlin.de (dataPort=32991)
2019-10-30 15:59:52,091 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Keyed
Aggregation -> Sink: Print to Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43)
switched from DEPLOYING to RUNNING.
2019-10-30 15:59:52,104 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom
File Source (1/1) (d50e407f6c4eb1a68684062a4f07517b) switched from DEPLOYING to
RUNNING.
2019-10-30 15:59:52,129 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Split Reader:
Custom File Source -> Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4)
switched from DEPLOYING to RUNNING.
2019-10-30 15:59:52,679 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source: Custom
File Source (1/1) (d50e407f6c4eb1a68684062a4f07517b) switched from RUNNING to
FINISHED.
2019-10-30 17:45:10,950 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Split Reader:
Custom File Source -> Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4)
switched from RUNNING to FINISHED.
2019-10-30 17:45:11,207 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Keyed
Aggregation -> Sink: Print to Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43)
switched from RUNNING to FINISHED.
2019-10-30 17:45:11,211 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job Streaming
WordCount (7cde77256615ceb13a1e74010af21c49) switched from state RUNNING to
FINISHED.
2019-10-30 17:45:11,211 INFO
org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Stopping
checkpoint coordinator for job 7cde77256615ceb13a1e74010af21c49.
2019-10-30 17:45:11,211 INFO
org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore -
Shutting down
2019-10-30 17:45:11,224 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Job
7cde77256615ceb13a1e74010af21c49 reached globally terminal state FINISHED.
2019-10-30 17:45:11,244 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Stopping the JobMaster for job Streaming
WordCount(7cde77256615ceb13a1e74010af21c49).
2019-10-30 17:45:11,256 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Suspending
SlotPool.
2019-10-30 17:45:11,258 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Close ResourceManager connection
6e522eb1e64f2d40e170960440b3e774: JobManager is shutting down..
2019-10-30 17:45:11,258 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Stopping
SlotPool.
2019-10-30 17:45:11,259 INFO
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager -
Disconnect job manager
00000000000000000000000000000...@akka.tcp://flink@localhost:6123/user/jobmanager_0
for job 7cde77256615ceb13a1e74010af21c49 from the resource manager.
2019-10-30 17:45:11,263 INFO
org.apache.flink.runtime.jobmaster.JobManagerRunner -
JobManagerRunner already shutdown.
2019-10-30 15:59:39,607 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
--------------------------------------------------------------------------------
2019-10-30 15:59:39,611 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting
TaskManager (Version: 1.8.2, Rev:6322618, Date:04.09.2019 @ 22:07:41 CST)
2019-10-30 15:59:39,611 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - OS current
user: xxx
2019-10-30 15:59:39,613 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Current
Hadoop/Kerberos user: <no hadoop dependency found>
2019-10-30 15:59:39,613 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM: OpenJDK
64-Bit Server VM - AdoptOpenJDK - 1.8/25.232-b09
2019-10-30 15:59:39,613 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum heap
size: 23976 MiBytes
2019-10-30 15:59:39,614 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JAVA_HOME:
/usr/lib/jvm/adoptopenjdk-8-hotspot-amd64/
2019-10-30 15:59:39,614 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - No Hadoop
Dependency available
2019-10-30 15:59:39,615 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - JVM Options:
2019-10-30 15:59:39,615 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -XX:+UseG1GC
2019-10-30 15:59:39,615 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xms23976M
2019-10-30 15:59:39,616 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - -Xmx23976M
2019-10-30 15:59:39,616 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
-XX:MaxDirectMemorySize=8388607T
2019-10-30 15:59:39,616 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
-Dlog.file=/home/xxx/flink-1.8.2/log/flink-xxx-taskexecutor-0-xxx.log
2019-10-30 15:59:39,617 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
-Dlog4j.configuration=file:/home/xxx/flink-1.8.2/conf/log4j.properties
2019-10-30 15:59:39,617 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
-Dlogback.configurationFile=file:/home/xxx/flink-1.8.2/conf/logback.xml
2019-10-30 15:59:39,617 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Program
Arguments:
2019-10-30 15:59:39,618 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - --configDir
2019-10-30 15:59:39,618 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
/home/xxx/flink-1.8.2/conf
2019-10-30 15:59:39,618 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Classpath:
/home/xxx/flink-1.8.2/lib/log4j-1.2.17.jar:/home/xxx/flink-1.8.2/lib/slf4j-log4j12-1.7.15.jar:/home/xxx/flink-1.8.2/lib/flink-dist_2.12-1.8.2.jar:::
2019-10-30 15:59:39,619 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner -
--------------------------------------------------------------------------------
2019-10-30 15:59:39,622 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Registered UNIX
signal handlers for [TERM, HUP, INT]
2019-10-30 15:59:39,632 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Maximum number
of open file descriptors is 1048576.
2019-10-30 15:59:39,665 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2019-10-30 15:59:39,666 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-10-30 15:59:39,666 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 25000m
2019-10-30 15:59:39,667 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 25000m
2019-10-30 15:59:39,667 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2019-10-30 15:59:39,668 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2019-10-30 15:59:39,689 INFO org.apache.flink.core.fs.FileSystem
- Hadoop is not in the classpath/dependencies. The extended set of
supported File Systems via Hadoop is not available.
2019-10-30 15:59:39,932 INFO
org.apache.flink.runtime.security.modules.HadoopModuleFactory - Cannot create
Hadoop Security Module because Hadoop cannot be found in the Classpath.
2019-10-30 15:59:39,989 INFO org.apache.flink.runtime.security.SecurityUtils
- Cannot install HadoopSecurityContext because Hadoop cannot be
found in the Classpath.
2019-10-30 15:59:41,289 INFO org.apache.flink.configuration.Configuration
- Config uses fallback configuration key 'jobmanager.rpc.address'
instead of key 'rest.address'
2019-10-30 15:59:41,304 INFO
org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to
select the network interface and address to use by connecting to the leading
JobManager.
2019-10-30 15:59:41,305 INFO
org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager
will try to connect for 10000 milliseconds before falling back to heuristics
2019-10-30 15:59:41,315 INFO org.apache.flink.runtime.net.ConnectionUtils
- Retrieved new target address localhost/127.0.0.1:6123.
2019-10-30 15:59:42,114 INFO org.apache.flink.runtime.net.ConnectionUtils
- Trying to connect to address localhost/127.0.0.1:6123
2019-10-30 15:59:42,115 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager
will use hostname/address 'xxx' (130.149.221.178) for communication.
2019-10-30 15:59:42,123 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to start
actor system at 130.149.221.178:0
2019-10-30 15:59:43,212 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2019-10-30 15:59:43,351 INFO akka.remote.Remoting
- Starting remoting
2019-10-30 15:59:43,742 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@130.149.221.178:34019]
2019-10-30 15:59:43,778 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor system
started at akka.tcp://flink@130.149.221.178:34019
2019-10-30 15:59:43,788 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Trying to start
actor system at 130.149.221.178:0
2019-10-30 15:59:43,830 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2019-10-30 15:59:43,851 INFO akka.remote.Remoting
- Starting remoting
2019-10-30 15:59:43,879 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink-metrics@130.149.221.178:40759]
2019-10-30 15:59:43,883 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Actor system
started at akka.tcp://flink-metrics@130.149.221.178:40759
2019-10-30 15:59:43,902 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics
reporter configured, no metrics will be exposed/reported.
2019-10-30 15:59:43,914 INFO org.apache.flink.runtime.blob.PermanentBlobCache
- Created BLOB cache storage directory
/tmp/blobStore-bcad8dce-1395-4f4b-b829-ec466d3f11f6
2019-10-30 15:59:43,920 INFO org.apache.flink.runtime.blob.TransientBlobCache
- Created BLOB cache storage directory
/tmp/blobStore-eaf11d9b-57b5-4d80-9ae5-ef604a691f61
2019-10-30 15:59:43,921 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Starting
TaskManager with ResourceID: 5e3d3f309eedf96ec861de138c26e0ac
2019-10-30 15:59:43,929 INFO
org.apache.flink.runtime.io.network.netty.NettyConfig - NettyConfig
[server address: /130.149.221.178, server port: 0, ssl enabled: false, memory
segment size (bytes): 32768, transport type: NIO, number of server threads: 1
(manual), number of client threads: 1 (manual), server connect backlog: 0 (use
Netty's default), client connect timeout (sec): 120, send/receive buffer size
(bytes): 0 (use Netty's default)]
2019-10-30 15:59:44,276 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Temporary file
directory '/tmp': total 15 GB, usable 9 GB (60.00% usable)
2019-10-30 15:59:45,959 INFO
org.apache.flink.runtime.io.network.buffer.NetworkBufferPool - Allocated 1024
MB for network buffer pool (number of memory segments: 32768, bytes per
segment: 32768).
2019-10-30 15:59:46,100 INFO
org.apache.flink.runtime.io.network.NetworkEnvironment - Starting the
network environment and its components.
2019-10-30 15:59:46,188 INFO
org.apache.flink.runtime.io.network.netty.NettyClient - Successful
initialization (took 85 ms).
2019-10-30 15:59:46,311 INFO
org.apache.flink.runtime.io.network.netty.NettyServer - Successful
initialization (took 122 ms). Listening on SocketAddress /130.149.221.178:32991.
2019-10-30 15:59:46,314 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerServices - Limiting
managed memory to 0.7 of the currently free heap space (16776 MB), memory will
be allocated lazily.
2019-10-30 15:59:46,323 INFO
org.apache.flink.runtime.io.disk.iomanager.IOManager - I/O manager
uses directory /tmp/flink-io-8cc2a96b-c287-43ca-a15a-35a9e1677aaf for spill
files.
2019-10-30 15:59:46,499 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerConfiguration - Messages have
a max timeout of 10000 ms
2019-10-30 15:59:46,523 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
- Starting RPC endpoint for
org.apache.flink.runtime.taskexecutor.TaskExecutor at
akka://flink/user/taskmanager_0 .
2019-10-30 15:59:46,566 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Start job
leader service.
2019-10-30 15:59:46,569 INFO org.apache.flink.runtime.filecache.FileCache
- User file cache uses directory
/tmp/flink-dist-cache-e2ec412f-de7b-489e-87c2-99c0f84c4a7f
2019-10-30 15:59:46,574 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Connecting to
ResourceManager
akka.tcp://flink@localhost:6123/user/resourcemanager(00000000000000000000000000000000).
2019-10-30 15:59:47,094 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Resolved
ResourceManager address, beginning registration
2019-10-30 15:59:47,095 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at
ResourceManager attempt 1 (timeout=100ms)
2019-10-30 15:59:47,234 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Registration at
ResourceManager attempt 2 (timeout=200ms)
2019-10-30 15:59:47,290 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Successful
registration at resource manager
akka.tcp://flink@localhost:6123/user/resourcemanager under registration id
e874a07bfe07800ddedfdaa12cc78d96.
2019-10-30 15:59:51,697 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Receive slot
request 3618194b1249b0a728ed674b7ddb0e6d for job
7cde77256615ceb13a1e74010af21c49 from resource manager with leader id
00000000000000000000000000000000.
2019-10-30 15:59:51,699 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Allocated slot
for 3618194b1249b0a728ed674b7ddb0e6d.
2019-10-30 15:59:51,700 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Add job
7cde77256615ceb13a1e74010af21c49 for job leader monitoring.
2019-10-30 15:59:51,705 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Try to register
at job manager akka.tcp://flink@localhost:6123/user/jobmanager_0 with leader id
00000000-0000-0000-0000-000000000000.
2019-10-30 15:59:51,747 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Resolved
JobManager address, beginning registration
2019-10-30 15:59:51,748 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Registration at
JobManager attempt 1 (timeout=100ms)
2019-10-30 15:59:51,803 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Successful
registration at job manager akka.tcp://flink@localhost:6123/user/jobmanager_0
for job 7cde77256615ceb13a1e74010af21c49.
2019-10-30 15:59:51,806 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Establish
JobManager connection for job 7cde77256615ceb13a1e74010af21c49.
2019-10-30 15:59:51,813 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Offer reserved
slots to the leader of job 7cde77256615ceb13a1e74010af21c49.
2019-10-30 15:59:51,946 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
Source: Custom File Source (1/1).
2019-10-30 15:59:51,948 INFO
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable - Activate slot
3618194b1249b0a728ed674b7ddb0e6d.
2019-10-30 15:59:51,949 INFO org.apache.flink.runtime.taskmanager.Task
- Source: Custom File Source (1/1)
(d50e407f6c4eb1a68684062a4f07517b) switched from CREATED to DEPLOYING.
2019-10-30 15:59:51,950 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task Source:
Custom File Source (1/1) (d50e407f6c4eb1a68684062a4f07517b) [DEPLOYING]
2019-10-30 15:59:51,960 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task Source: Custom File Source (1/1)
(d50e407f6c4eb1a68684062a4f07517b) [DEPLOYING].
2019-10-30 15:59:51,967 INFO org.apache.flink.runtime.blob.BlobClient
- Downloading
7cde77256615ceb13a1e74010af21c49/p-69c1a4c798e039a06e42affd705b3a263014e46d-a9b6a34c16bbefdb0e0a3864cb609a87
from localhost/127.0.0.1:42005
2019-10-30 15:59:51,983 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
Keyed Aggregation -> Sink: Print to Std. Out (1/1).
2019-10-30 15:59:51,985 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Print to Std. Out (1/1)
(016f9ff69941a26f8e2c4664cadc6b43) switched from CREATED to DEPLOYING.
2019-10-30 15:59:51,988 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task Keyed
Aggregation -> Sink: Print to Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43)
[DEPLOYING]
2019-10-30 15:59:51,991 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task Keyed Aggregation -> Sink: Print to
Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43) [DEPLOYING].
2019-10-30 15:59:52,000 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Received task
Split Reader: Custom File Source -> Flat Map (1/1).
2019-10-30 15:59:52,004 INFO org.apache.flink.runtime.taskmanager.Task
- Split Reader: Custom File Source -> Flat Map (1/1)
(6a17c410c3e36f524bb774d2dffed4a4) switched from CREATED to DEPLOYING.
2019-10-30 15:59:52,006 INFO org.apache.flink.runtime.taskmanager.Task
- Creating FileSystem stream leak safety net for task Split Reader:
Custom File Source -> Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4)
[DEPLOYING]
2019-10-30 15:59:52,006 INFO org.apache.flink.runtime.taskmanager.Task
- Loading JAR files for task Split Reader: Custom File Source ->
Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4) [DEPLOYING].
2019-10-30 15:59:52,032 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: Keyed Aggregation -> Sink: Print to
Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43) [DEPLOYING].
2019-10-30 15:59:52,032 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: Split Reader: Custom File Source ->
Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4) [DEPLOYING].
2019-10-30 15:59:52,032 INFO org.apache.flink.runtime.taskmanager.Task
- Registering task at network: Source: Custom File Source (1/1)
(d50e407f6c4eb1a68684062a4f07517b) [DEPLOYING].
2019-10-30 15:59:52,078 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Print to Std. Out (1/1)
(016f9ff69941a26f8e2c4664cadc6b43) switched from DEPLOYING to RUNNING.
2019-10-30 15:59:52,095 INFO org.apache.flink.runtime.taskmanager.Task
- Source: Custom File Source (1/1)
(d50e407f6c4eb1a68684062a4f07517b) switched from DEPLOYING to RUNNING.
2019-10-30 15:59:52,100 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - No state
backend has been configured, using default (Memory / JobManager)
MemoryStateBackend (data in heap memory / checkpoints to JobManager)
(checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize:
5242880)
2019-10-30 15:59:52,100 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - No state
backend has been configured, using default (Memory / JobManager)
MemoryStateBackend (data in heap memory / checkpoints to JobManager)
(checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize:
5242880)
2019-10-30 15:59:52,122 INFO org.apache.flink.runtime.taskmanager.Task
- Split Reader: Custom File Source -> Flat Map (1/1)
(6a17c410c3e36f524bb774d2dffed4a4) switched from DEPLOYING to RUNNING.
2019-10-30 15:59:52,123 INFO
org.apache.flink.streaming.runtime.tasks.StreamTask - No state
backend has been configured, using default (Memory / JobManager)
MemoryStateBackend (data in heap memory / checkpoints to JobManager)
(checkpoints: 'null', savepoints: 'null', asynchronous: TRUE, maxStateSize:
5242880)
2019-10-30 15:59:52,168 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.address, localhost
2019-10-30 15:59:52,169 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 6123
2019-10-30 15:59:52,169 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.heap.size, 25000m
2019-10-30 15:59:52,170 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.heap.size, 25000m
2019-10-30 15:59:52,170 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2019-10-30 15:59:52,170 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: parallelism.default, 1
2019-10-30 15:59:52,290 INFO
org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction
- No state to restore for the ContinuousFileMonitoringFunction.
2019-10-30 15:59:52,291 INFO
org.apache.flink.streaming.api.functions.source.ContinuousFileReaderOperator -
No state to restore for the ContinuousFileReaderOperator (taskIdx=0).
2019-10-30 15:59:52,302 INFO
org.apache.flink.streaming.api.functions.source.ContinuousFileMonitoringFunction
- Forwarding split: [0] file:/home/xxx/test.txt mod@ 1570712020000 : 0 +
2104576302
2019-10-30 15:59:52,313 INFO
org.apache.flink.runtime.state.heap.HeapKeyedStateBackend - Initializing
heap keyed state backend with stream factory.
2019-10-30 15:59:52,639 INFO org.apache.flink.runtime.taskmanager.Task
- Source: Custom File Source (1/1)
(d50e407f6c4eb1a68684062a4f07517b) switched from RUNNING to FINISHED.
2019-10-30 15:59:52,639 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Source: Custom File Source (1/1)
(d50e407f6c4eb1a68684062a4f07517b).
2019-10-30 15:59:52,644 INFO org.apache.flink.runtime.taskmanager.Task
- Ensuring all FileSystem streams are closed for task Source:
Custom File Source (1/1) (d50e407f6c4eb1a68684062a4f07517b) [FINISHED]
2019-10-30 15:59:52,647 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering
task and sending final execution state FINISHED to JobManager for task Source:
Custom File Source d50e407f6c4eb1a68684062a4f07517b.
2019-10-30 17:45:10,943 INFO org.apache.flink.runtime.taskmanager.Task
- Split Reader: Custom File Source -> Flat Map (1/1)
(6a17c410c3e36f524bb774d2dffed4a4) switched from RUNNING to FINISHED.
2019-10-30 17:45:10,943 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Split Reader: Custom File Source ->
Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4).
2019-10-30 17:45:10,944 INFO org.apache.flink.runtime.taskmanager.Task
- Ensuring all FileSystem streams are closed for task Split Reader:
Custom File Source -> Flat Map (1/1) (6a17c410c3e36f524bb774d2dffed4a4)
[FINISHED]
2019-10-30 17:45:10,945 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering
task and sending final execution state FINISHED to JobManager for task Split
Reader: Custom File Source -> Flat Map 6a17c410c3e36f524bb774d2dffed4a4.
2019-10-30 17:45:11,199 INFO org.apache.flink.runtime.taskmanager.Task
- Keyed Aggregation -> Sink: Print to Std. Out (1/1)
(016f9ff69941a26f8e2c4664cadc6b43) switched from RUNNING to FINISHED.
2019-10-30 17:45:11,199 INFO org.apache.flink.runtime.taskmanager.Task
- Freeing task resources for Keyed Aggregation -> Sink: Print to
Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43).
2019-10-30 17:45:11,202 INFO org.apache.flink.runtime.taskmanager.Task
- Ensuring all FileSystem streams are closed for task Keyed
Aggregation -> Sink: Print to Std. Out (1/1) (016f9ff69941a26f8e2c4664cadc6b43)
[FINISHED]
2019-10-30 17:45:11,202 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering
task and sending final execution state FINISHED to JobManager for task Keyed
Aggregation -> Sink: Print to Std. Out 016f9ff69941a26f8e2c4664cadc6b43.
2019-10-30 17:45:11,260 INFO
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable - Free slot
TaskSlot(index:0, state:ACTIVE, resource profile:
ResourceProfile{cpuCores=1.7976931348623157E308, heapMemoryInMB=2147483647,
directMemoryInMB=2147483647, nativeMemoryInMB=2147483647,
networkMemoryInMB=2147483647}, allocationId: 3618194b1249b0a728ed674b7ddb0e6d,
jobId: 7cde77256615ceb13a1e74010af21c49).
2019-10-30 17:45:11,261 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Remove job
7cde77256615ceb13a1e74010af21c49 from job leader monitoring.
2019-10-30 17:45:11,261 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Close
JobManager connection for job 7cde77256615ceb13a1e74010af21c49.
2019-10-30 17:45:11,268 INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Close
JobManager connection for job 7cde77256615ceb13a1e74010af21c49.
2019-10-30 17:45:11,270 INFO
org.apache.flink.runtime.taskexecutor.JobLeaderService - Cannot
reconnect to job 7cde77256615ceb13a1e74010af21c49 because it is not registered.
