Mathieu DESPRIEE created FLINK-22754:
----------------------------------------
Summary: Finished job not always archived properly
Key: FLINK-22754
URL: https://issues.apache.org/jira/browse/FLINK-22754
Project: Flink
Issue Type: Bug
Affects Versions: 1.12.1
Reporter: Mathieu DESPRIEE
Flink batch job, ran on YARN/EMR. Flink v1.12.1
2 different runs, same configuration.
At the end of the first run, the job has been properly archived:
{quote}2021-05-20 07:14:39,479 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Shut
down cluster because application is in SUCCEEDED, diagnostics null.
2021-05-20 07:14:39,479 INFO org.apache.flink.yarn.YarnResourceManagerDriver
[] - Unregister application from the YARN Resource Manager with final status
SUCCEEDED.
2021-05-20 07:14:39,490 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl [] - Waiting for
application to be successfully unregistered.
2021-05-20 07:14:39,501 WARN org.apache.hadoop.util.NativeCodeLoader [] -
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2021-05-20 07:14:40,277 INFO org.apache.hadoop.conf.Configuration.deprecation
[] - fs.s3a.server-side-encryption-key is deprecated. Instead, use
fs.s3a.server-side-encryption.key
2021-05-20 07:14:40,531 WARN com.amazonaws.services.s3.internal.Mimetypes [] -
Unable to find 'mime.types' file in classpath
2021-05-20 07:14:40,727 INFO
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
[] - Closing components.
2021-05-20 07:14:40,728 INFO
org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcess [] -
Stopping JobDispatcherLeaderProcess.
2021-05-20 07:14:40,728 INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping dispatcher
akka.tcp://[email protected]:44219/user/rpc/dispatcher_1.
2021-05-20 07:14:40,728 INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopping all currently
running jobs of dispatcher
akka.tcp://[email protected]:44219/user/rpc/dispatcher_1.
2021-05-20 07:14:40,729 INFO
org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator
[] - Shutting down back pressure request coordinator.
2021-05-20 07:14:40,729 INFO
org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Stopped dispatcher
akka.tcp://[email protected]:44219/user/rpc/dispatcher_1.
2021-05-20 07:14:40,815 INFO *org.apache.flink.runtime.history.FsJobArchivist
[] - Job a2760da3fb41127b8d6f54a1573bd3fa has been archived at
s3://REDACTED-emr/history-server/a2760da3fb41127b8d6f54a1573bd3fa*.
{quote}
At the end of the second run, no mention of "FsJobArchivist"
{quote}2021-05-21 23:49:00,191 INFO
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Shut
down cluster because application is in SUCCEEDED, diagnostics null.
2021-05-21 23:49:00,193 INFO org.apache.flink.yarn.YarnResourceManagerDriver []
- Unregister application from the YARN Resource Manager with final status
SUCCEEDED.
2021-05-21 23:49:00,208 INFO
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl [] - Waiting for
application to be successfully unregistered.
2021-05-21 23:49:01,311 INFO
org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
[] - Closing components.
2021-05-21 23:49:01,312 INFO
org.apache.flink.runtime.dispatcher.runner.JobDispatcherLeaderProcess [] -
Stopping JobDispatcherLeaderProcess.
2021-05-21 23:49:01,312 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher
[] - Stopping dispatcher
akka.tcp://[email protected]:35331/user/rpc/dispatcher_1.
2021-05-21 23:49:01,312 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher
[] - Stopping all currently running jobs of dispatcher
akka.tcp://[email protected]:35331/user/rpc/dispatcher_1.
2021-05-21 23:49:01,312 INFO
org.apache.flink.runtime.rest.handler.legacy.backpressure.BackPressureRequestCoordinator
[] - Shutting down back pressure request coordinator.
2021-05-21 23:49:01,312 INFO org.apache.flink.runtime.dispatcher.MiniDispatcher
[] - Stopped dispatcher
akka.tcp://[email protected]:35331/user/rpc/dispatcher_1.
2021-05-21 23:49:01,337 WARN com.amazonaws.services.s3.internal.Mimetypes [] -
Unable to find 'mime.types' file in classpath
2021-05-21 23:49:01,460 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] -
Closing the SlotManager.
2021-05-21 23:49:01,460 INFO
org.apache.flink.runtime.resourcemanager.slotmanager.SlotManagerImpl [] -
Suspending the SlotManager.
2021-05-21 23:49:01,462 INFO org.apache.flink.runtime.blob.BlobServer [] -
Stopped BLOB server at 0.0.0.0:40643
2021-05-21 23:49:01,464 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Stopping Akka RPC service.
2021-05-21 23:49:01,487 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService
[] - Stopping Akka RPC service.
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
