Re: Re: Flink 1.11 not showing logs
Hi Arnaud, It seems that the TaskExecutor terminated exceptionally. I think you need to check the logs of container_e38_1604477334666_0960_01_04 to figure out why it crashed or shut down. Best, Yang LINZ, Arnaud 于2020年11月16日周一 下午7:11写道： > Hello, > > I'm running Flink 1.10 on a yarn cluster. I have a streaming application, > that, when under heavy load, fails from time to time with this unique error > message in the whole yarn log: > > (...) > 2020-11-15 16:18:42,202 WARN > org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received > late message for now expired checkpoint attempt 63 from task > 4cbc940112a596db54568b24f9209aac of job 1e1717d19bd8ea296314077e42e1c7e5 at > container_e38_1604477334666_0960_01_04 @ xxx (dataPort=33099). > 2020-11-15 16:18:55,043 INFO org.apache.flink.yarn.YarnResourceManager > - Closing TaskExecutor connection > container_e38_1604477334666_0960_01_04 because: The TaskExecutor is > shutting down. > 2020-11-15 16:18:55,087 INFO > org.apache.flink.runtime.executiongraph.ExecutionGraph- Map (7/15) > (c8e92cacddcd4e41f51a2433d07d2153) switched from RUNNING to FAILED. > org.apache.flink.util.FlinkException: The TaskExecutor is shutting down. > > at > org.apache.flink.runtime.taskexecutor.TaskExecutor.onStop(TaskExecutor.java:359) > at > org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:218) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:509) > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:175) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at > scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf > .UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > at > akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > 2020-11-15 16:18:55,092 INFO > org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy > - Calculating tasks to restart to recover the failed task > 2f6467d98899e64a4721f0a7b6a059a8_6. > 2020-11-15 16:18:55,101 INFO > org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy > - 230 tasks should be restarted to recover the failed task > 2f6467d98899e64a4721f0a7b6a059a8_6. > (...) > > What could be the cause of this failure? Why is there no other error > message? > > I've tried to increase the value of heartbeat.timeout, thinking that maybe > it was due to a slow responding mapper, but it did not solve the issue. > > Best regards, > Arnaud > > > > L'intégrité de ce message n'étant pas assurée sur internet, la société > expéditrice ne peut être tenue responsable de son contenu ni de ses pièces > jointes. Toute utilisation ou diffusion non autorisée est interdite. Si > vous n'êtes pas destinataire de ce message, merci de le détruire et > d'avertir l'expéditeur. > > The integrity of this message cannot be guaranteed on the Internet. The > company that sent this message cannot therefore be held liable for its > content nor attachments. Any unauthorized use or dissemination is > prohibited. If you are not the intended recipient of this message, then > please delete it and notify the sender. >
RE: Re: Flink 1.11 not showing logs
Hello, I'm running Flink 1.10 on a yarn cluster. I have a streaming application, that, when under heavy load, fails from time to time with this unique error message in the whole yarn log: (...) 2020-11-15 16:18:42,202 WARN org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Received late message for now expired checkpoint attempt 63 from task 4cbc940112a596db54568b24f9209aac of job 1e1717d19bd8ea296314077e42e1c7e5 at container_e38_1604477334666_0960_01_04 @ xxx (dataPort=33099). 2020-11-15 16:18:55,043 INFO org.apache.flink.yarn.YarnResourceManager - Closing TaskExecutor connection container_e38_1604477334666_0960_01_04 because: The TaskExecutor is shutting down. 2020-11-15 16:18:55,087 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph- Map (7/15) (c8e92cacddcd4e41f51a2433d07d2153) switched from RUNNING to FAILED. org.apache.flink.util.FlinkException: The TaskExecutor is shutting down. at org.apache.flink.runtime.taskexecutor.TaskExecutor.onStop(TaskExecutor.java:359) at org.apache.flink.runtime.rpc.RpcEndpoint.internalCallOnStop(RpcEndpoint.java:218) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor$StartedState.terminate(AkkaRpcActor.java:509) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleControlMessage(AkkaRpcActor.java:175) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at akka.actor.Actor$class.aroundReceive(Actor.scala:517) at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at akka.actor.ActorCell.invoke(ActorCell.scala:561) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at akka.dispatch.Mailbox.run(Mailbox.scala:225) at akka.dispatch.Mailbox.exec(Mailbox.scala:235) at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) 2020-11-15 16:18:55,092 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy - Calculating tasks to restart to recover the failed task 2f6467d98899e64a4721f0a7b6a059a8_6. 2020-11-15 16:18:55,101 INFO org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy - 230 tasks should be restarted to recover the failed task 2f6467d98899e64a4721f0a7b6a059a8_6. (...) What could be the cause of this failure? Why is there no other error message? I've tried to increase the value of heartbeat.timeout, thinking that maybe it was due to a slow responding mapper, but it did not solve the issue. Best regards, Arnaud L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur. The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
Re:Re:Re:Re: Re: Flink 1.11 not showing logs
Hi Yang,
After I copied the logic from `YarnLogConfigUtil` to my own deployer (maybe
call its logic instead of copying is a better option), the logs now can show
normally.
Thanks again for the kind help.
At 2020-11-16 17:28:47, "马阳阳" wrote:
Hi Yang,
I checked the `YarnLogConfigUtil`, it does some work to set the configuration
for log.
Should I copy the logic to my deployer?
At 2020-11-16 17:21:07, "马阳阳" wrote:
Hi Yang,
Thank you for you reply.
I set the value for "$internal.deployment.config-dir" to the Flink
configuration directory.
And the configuration showed on Flink web UI. But it still not work. So I
wonder what should
I set as the value for "$internal.deployment.config-dir"?
At 2020-11-16 16:43:11, "Yang Wang" wrote:
If you are using your own deployer(aka a java program calls the Flink client
API to submit Flink jobs),
you need to check the jobmanager configuration in webUI whether
"$internal.yarn.log-config-file"
is correctly set. If not, maybe you need to set
"$internal.deployment.config-dir" in your deployer,
not simply set the FLINK_CONF_DIR environment. Because your deployer needs to
do some configuration setting
which CliFrontend has done. Please have a try and share more feedback.
Best,
Yang
马阳阳 于2020年11月16日周一 下午2:47写道：
Hi Yang,
We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any
taskmanager/jobmanager logs.
I have checked the log4j.properties file, and it's in the right format. And the
FLINK_CONF_DIR is set.
When checking the java dynamic options of task manager, I found that the log
related options are not
set.
This is the output when ussuing "ps -ef | grep ".
yarn 31049 30974 9 13:57 ?00:03:31
/usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902
-XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456
org.apache.flink.yarn.YarnTaskExecutorRunner -D
taskmanager.memory.framework.off-heap.size=134217728b -D
taskmanager.memory.network.max=134217730b -D
taskmanager.memory.network.min=134217730b -D
taskmanager.memory.framework.heap.size=134217728b -D
taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D
taskmanager.memory.task.heap.size=402653174b -D
taskmanager.memory.task.off-heap.size=0b --configDir .
-Djobmanager.rpc.address=dhpdn09-113
-Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0
-Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63
-Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113
My question is, what maybe the problem for this? And any suggestions?
By the way, we submit the program from Java program instead of from the command
line.
Thanks.
ps: I sent the mail to spark user mail list un-attentionally. So I resent it to
the Flink user mail list. Sorry for the inconvenience to @Yang Wang
At 2020-11-03 20:56:19, "Yang Wang" wrote:
You could issue "ps -ef | grep container_id_for_some_tm". And then you will
find the
following java options about log4j.
-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log
-Dlog4j.configuration=file:./log4j.properties
-Dlog4j.configurationFile=file:./log4j.properties
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午11:37写道：
Sure. I will check that and get back to you. could you please share how to
check java dynamic options?
Best,
Diwakar
On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
If you have already updated the log4j.properties, and it still could not work,
then I
suggest to log in the Yarn NodeManager machine and check the log4j.properties
in the container workdir is correct. Also you could have a look at the java
dynamic
options are correctly set.
I think it should work if the log4j.properties and java dynamic options are set
correctly.
BTW, could you share the new yarn logs?
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午4:32写道：
Hi Yang,
Thank you so much for taking a look at the log files. I changed my
log4j.properties. Below is the actual file that I got from EMR 6.1.0
distribution of flink 1.11. I observed that it is different from Flink 1.11
that i downloaded so i changed it. Still I didn't see any logs.
Actual
log4j.rootLogger=INFO,file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS} %-5p
%-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
modified : commented the above and added new logging from actual flink
application log4.properties file
#log4j.rootLogger=INFO,file
# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
Re:Re:Re: Re: Flink 1.11 not showing logs
Hi Yang,
I checked the `YarnLogConfigUtil`, it does some work to set the configuration
for log.
Should I copy the logic to my deployer?
At 2020-11-16 17:21:07, "马阳阳" wrote:
Hi Yang,
Thank you for you reply.
I set the value for "$internal.deployment.config-dir" to the Flink
configuration directory.
And the configuration showed on Flink web UI. But it still not work. So I
wonder what should
I set as the value for "$internal.deployment.config-dir"?
At 2020-11-16 16:43:11, "Yang Wang" wrote:
If you are using your own deployer(aka a java program calls the Flink client
API to submit Flink jobs),
you need to check the jobmanager configuration in webUI whether
"$internal.yarn.log-config-file"
is correctly set. If not, maybe you need to set
"$internal.deployment.config-dir" in your deployer,
not simply set the FLINK_CONF_DIR environment. Because your deployer needs to
do some configuration setting
which CliFrontend has done. Please have a try and share more feedback.
Best,
Yang
马阳阳 于2020年11月16日周一 下午2:47写道：
Hi Yang,
We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any
taskmanager/jobmanager logs.
I have checked the log4j.properties file, and it's in the right format. And the
FLINK_CONF_DIR is set.
When checking the java dynamic options of task manager, I found that the log
related options are not
set.
This is the output when ussuing "ps -ef | grep ".
yarn 31049 30974 9 13:57 ?00:03:31
/usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902
-XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456
org.apache.flink.yarn.YarnTaskExecutorRunner -D
taskmanager.memory.framework.off-heap.size=134217728b -D
taskmanager.memory.network.max=134217730b -D
taskmanager.memory.network.min=134217730b -D
taskmanager.memory.framework.heap.size=134217728b -D
taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D
taskmanager.memory.task.heap.size=402653174b -D
taskmanager.memory.task.off-heap.size=0b --configDir .
-Djobmanager.rpc.address=dhpdn09-113
-Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0
-Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63
-Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113
My question is, what maybe the problem for this? And any suggestions?
By the way, we submit the program from Java program instead of from the command
line.
Thanks.
ps: I sent the mail to spark user mail list un-attentionally. So I resent it to
the Flink user mail list. Sorry for the inconvenience to @Yang Wang
At 2020-11-03 20:56:19, "Yang Wang" wrote:
You could issue "ps -ef | grep container_id_for_some_tm". And then you will
find the
following java options about log4j.
-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log
-Dlog4j.configuration=file:./log4j.properties
-Dlog4j.configurationFile=file:./log4j.properties
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午11:37写道：
Sure. I will check that and get back to you. could you please share how to
check java dynamic options?
Best,
Diwakar
On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
If you have already updated the log4j.properties, and it still could not work,
then I
suggest to log in the Yarn NodeManager machine and check the log4j.properties
in the container workdir is correct. Also you could have a look at the java
dynamic
options are correctly set.
I think it should work if the log4j.properties and java dynamic options are set
correctly.
BTW, could you share the new yarn logs?
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午4:32写道：
Hi Yang,
Thank you so much for taking a look at the log files. I changed my
log4j.properties. Below is the actual file that I got from EMR 6.1.0
distribution of flink 1.11. I observed that it is different from Flink 1.11
that i downloaded so i changed it. Still I didn't see any logs.
Actual
log4j.rootLogger=INFO,file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS} %-5p
%-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
modified : commented the above and added new logging from actual flink
application log4.properties file
#log4j.rootLogger=INFO,file
# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS} %-5p
%-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
Re:Re: Re: Flink 1.11 not showing logs
Hi Yang,
Thank you for you reply.
I set the value for "$internal.deployment.config-dir" to the Flink
configuration directory.
And the configuration showed on Flink web UI. But it still not work. So I
wonder what should
I set as the value for "$internal.deployment.config-dir"?
At 2020-11-16 16:43:11, "Yang Wang" wrote:
If you are using your own deployer(aka a java program calls the Flink client
API to submit Flink jobs),
you need to check the jobmanager configuration in webUI whether
"$internal.yarn.log-config-file"
is correctly set. If not, maybe you need to set
"$internal.deployment.config-dir" in your deployer,
not simply set the FLINK_CONF_DIR environment. Because your deployer needs to
do some configuration setting
which CliFrontend has done. Please have a try and share more feedback.
Best,
Yang
马阳阳 于2020年11月16日周一 下午2:47写道：
Hi Yang,
We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any
taskmanager/jobmanager logs.
I have checked the log4j.properties file, and it's in the right format. And the
FLINK_CONF_DIR is set.
When checking the java dynamic options of task manager, I found that the log
related options are not
set.
This is the output when ussuing "ps -ef | grep ".
yarn 31049 30974 9 13:57 ?00:03:31
/usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902
-XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456
org.apache.flink.yarn.YarnTaskExecutorRunner -D
taskmanager.memory.framework.off-heap.size=134217728b -D
taskmanager.memory.network.max=134217730b -D
taskmanager.memory.network.min=134217730b -D
taskmanager.memory.framework.heap.size=134217728b -D
taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D
taskmanager.memory.task.heap.size=402653174b -D
taskmanager.memory.task.off-heap.size=0b --configDir .
-Djobmanager.rpc.address=dhpdn09-113
-Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0
-Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63
-Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113
My question is, what maybe the problem for this? And any suggestions?
By the way, we submit the program from Java program instead of from the command
line.
Thanks.
ps: I sent the mail to spark user mail list un-attentionally. So I resent it to
the Flink user mail list. Sorry for the inconvenience to @Yang Wang
At 2020-11-03 20:56:19, "Yang Wang" wrote:
You could issue "ps -ef | grep container_id_for_some_tm". And then you will
find the
following java options about log4j.
-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log
-Dlog4j.configuration=file:./log4j.properties
-Dlog4j.configurationFile=file:./log4j.properties
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午11:37写道：
Sure. I will check that and get back to you. could you please share how to
check java dynamic options?
Best,
Diwakar
On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
If you have already updated the log4j.properties, and it still could not work,
then I
suggest to log in the Yarn NodeManager machine and check the log4j.properties
in the container workdir is correct. Also you could have a look at the java
dynamic
options are correctly set.
I think it should work if the log4j.properties and java dynamic options are set
correctly.
BTW, could you share the new yarn logs?
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午4:32写道：
Hi Yang,
Thank you so much for taking a look at the log files. I changed my
log4j.properties. Below is the actual file that I got from EMR 6.1.0
distribution of flink 1.11. I observed that it is different from Flink 1.11
that i downloaded so i changed it. Still I didn't see any logs.
Actual
log4j.rootLogger=INFO,file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS} %-5p
%-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
modified : commented the above and added new logging from actual flink
application log4.properties file
#log4j.rootLogger=INFO,file
# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS} %-5p
%-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender
# Uncomment this
Re: Re: Flink 1.11 not showing logs
If you are using your own deployer(aka a java program calls the Flink
client API to submit Flink jobs),
you need to check the jobmanager configuration in webUI whether "
$internal.yarn.log-config-file"
is correctly set. If not, maybe you need to set "
$internal.deployment.config-dir" in your deployer,
not simply set the FLINK_CONF_DIR environment. Because your deployer needs
to do some configuration setting
which CliFrontend has done. Please have a try and share more feedback.
Best,
Yang
马阳阳 于2020年11月16日周一 下午2:47写道：
> Hi Yang,
> We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any
> taskmanager/jobmanager logs.
> I have checked the log4j.properties file, and it's in the right format.
> And the FLINK_CONF_DIR is set.
> When checking the java dynamic options of task manager, I found that the
> log related options are not
> set.
> This is the output when ussuing "ps -ef | grep ".
>
> yarn 31049 30974 9 13:57 ?00:03:31
> /usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902
> -XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456
> org.apache.flink.yarn.YarnTaskExecutorRunner -D
> taskmanager.memory.framework.off-heap.size=134217728b -D
> taskmanager.memory.network.max=134217730b -D
> taskmanager.memory.network.min=134217730b -D
> taskmanager.memory.framework.heap.size=134217728b -D
> taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D
> taskmanager.memory.task.heap.size=402653174b -D
> taskmanager.memory.task.off-heap.size=0b --configDir .
> -Djobmanager.rpc.address=dhpdn09-113
> -Dtaskmanager.resource-id=container_1604585185669_635512_01_000713
> -Dweb.port=0
> -Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63
> -Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113
>
> My question is, what maybe the problem for this? And any suggestions?
>
> By the way, we submit the program from Java program instead of from the
> command line.
>
> Thanks.
>
> ps: I sent the mail to spark user mail list un-attentionally. So I resent
> it to the Flink user mail list. Sorry for the inconvenience to @Yang Wang
>
>
>
>
>
>
>
> At 2020-11-03 20:56:19, "Yang Wang" wrote:
>
> You could issue "ps -ef | grep container_id_for_some_tm". And then you
> will find the
> following java options about log4j.
>
>
> -Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log
> -Dlog4j.configuration=file:./log4j.properties
> -Dlog4j.configurationFile=file:./log4j.properties
>
> Best,
> Yang
>
> Diwakar Jha 于2020年11月2日周一 下午11:37写道：
>
>> Sure. I will check that and get back to you. could you please share how
>> to check java dynamic options?
>>
>> Best,
>> Diwakar
>>
>> On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
>>
>>> If you have already updated the log4j.properties, and it still could not
>>> work, then I
>>> suggest to log in the Yarn NodeManager machine and check the
>>> log4j.properties
>>> in the container workdir is correct. Also you could have a look at the
>>> java dynamic
>>> options are correctly set.
>>>
>>> I think it should work if the log4j.properties and java dynamic options
>>> are set correctly.
>>>
>>> BTW, could you share the new yarn logs?
>>>
>>> Best,
>>> Yang
>>>
>>> Diwakar Jha 于2020年11月2日周一 下午4:32写道：
>>>
Hi Yang,
Thank you so much for taking a look at the log files. I changed my
log4j.properties. Below is the actual file that I got from EMR 6.1.0
distribution of flink 1.11. I observed that it is different from Flink 1.11
that i downloaded so i changed it. Still I didn't see any logs.
*Actual*
log4j.rootLogger=INFO,file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd
HH:mm:ss,SSS} %-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel
handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
*modified : *commented the above and added new logging from
actual flink application log4.properties file
#log4j.rootLogger=INFO,file
# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{-MM-dd
HH:mm:ss,SSS} %-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel
handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender
#
Re:Re: Flink 1.11 not showing logs
Hi Yang,
We run a self-compiled Flink-1.12-SNAPSHOT, and could not see any
taskmanager/jobmanager logs.
I have checked the log4j.properties file, and it's in the right format. And the
FLINK_CONF_DIR is set.
When checking the java dynamic options of task manager, I found that the log
related options are not
set.
This is the output when ussuing "ps -ef | grep ".
yarn 31049 30974 9 13:57 ?00:03:31
/usr/lib/jvm/jdk1.8.0_121/bin/java -Xmx536870902 -Xms536870902
-XX:MaxDirectMemorySize=268435458 -XX:MaxMetaspaceSize=268435456
org.apache.flink.yarn.YarnTaskExecutorRunner -D
taskmanager.memory.framework.off-heap.size=134217728b -D
taskmanager.memory.network.max=134217730b -D
taskmanager.memory.network.min=134217730b -D
taskmanager.memory.framework.heap.size=134217728b -D
taskmanager.memory.managed.size=536870920b -D taskmanager.cpu.cores=1.0 -D
taskmanager.memory.task.heap.size=402653174b -D
taskmanager.memory.task.off-heap.size=0b --configDir .
-Djobmanager.rpc.address=dhpdn09-113
-Dtaskmanager.resource-id=container_1604585185669_635512_01_000713 -Dweb.port=0
-Dweb.tmpdir=/tmp/flink-web-1d373ec2-0cbe-49b8-9592-3ac1d207ad63
-Djobmanager.rpc.port=40093 -Drest.address=dhpdn09-113
My question is, what maybe the problem for this? And any suggestions?
By the way, we submit the program from Java program instead of from the command
line.
Thanks.
ps: I sent the mail to spark user mail list un-attentionally. So I resent it to
the Flink user mail list. Sorry for the inconvenience to @Yang Wang
At 2020-11-03 20:56:19, "Yang Wang" wrote:
You could issue "ps -ef | grep container_id_for_some_tm". And then you will
find the
following java options about log4j.
-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log
-Dlog4j.configuration=file:./log4j.properties
-Dlog4j.configurationFile=file:./log4j.properties
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午11:37写道：
Sure. I will check that and get back to you. could you please share how to
check java dynamic options?
Best,
Diwakar
On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
If you have already updated the log4j.properties, and it still could not work,
then I
suggest to log in the Yarn NodeManager machine and check the log4j.properties
in the container workdir is correct. Also you could have a look at the java
dynamic
options are correctly set.
I think it should work if the log4j.properties and java dynamic options are set
correctly.
BTW, could you share the new yarn logs?
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午4:32写道：
Hi Yang,
Thank you so much for taking a look at the log files. I changed my
log4j.properties. Below is the actual file that I got from EMR 6.1.0
distribution of flink 1.11. I observed that it is different from Flink 1.11
that i downloaded so i changed it. Still I didn't see any logs.
Actual
log4j.rootLogger=INFO,file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS} %-5p
%-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
modified : commented the above and added new logging from actual flink
application log4.properties file
#log4j.rootLogger=INFO,file
# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS} %-5p
%-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
# Suppress the irrelevant
Re: Flink 1.11 not showing logs
If you have set the environment FLINK_CONF_DIR, then it will have a higher
priority.
I think that could be why you changed the log4j.properties in the conf
directory but it does
not take effect.
Yes, if you have changed the log4j.properties, you need to relaunch the
Flink application.
Although we have a ticket to support updating the log level via rest
API[1], we do not have
much process on this. And I think you could keep an eye on this and share
your thoughts.
[1]. https://issues.apache.org/jira/browse/FLINK-16478
Best,
Yang
Diwakar Jha 于2020年11月12日周四 上午5:38写道：
> HI Yang,
>
> I'm able to see taskmanage and jobmanager logs after I changed the
> log4j.properties file (/usr/lib/flink/conf). Thank you!
> I updated the file as shown below. I had to kill the app ( yarn
> application -kill ) and start flink job again to get the logs. This
> doesn't seem like an efficient way. I was wondering if there's a more
> simpler way to do it in production. let me know, please!
>
> *Actual*
> log4j.rootLogger=INFO,file
>
> # Log all infos in the given file
> log4j.appender.file=org.apache.log4j.FileAppender
> log4j.appender.file.file=${log.file}
> log4j.appender.file.append=false
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # suppress the irrelevant (wrong) warnings from the netty channel handler
> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>
>
> *modified : *commented the above and added new logging from actual flink
> application log4.properties file
>
> #log4j.rootLogger=INFO,file
>
> # Log all infos in the given file
> #log4j.appender.file=org.apache.log4j.FileAppender
> #log4j.appender.file.file=${log.file}
> #log4j.appender.file.append=false
> #log4j.appender.file.layout=org.apache.log4j.PatternLayout
> #log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # suppress the irrelevant (wrong) warnings from the netty channel handler
> #log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>
> # This affects logging for both user code and Flink
> rootLogger.level = INFO
> rootLogger.appenderRef.file.ref = MainAppender
>
> # Uncomment this if you want to _only_ change Flink's logging
> #logger.flink.name = org.apache.flink
> #logger.flink.level = INFO
>
> # The following lines keep the log level of common libraries/connectors on
> # log level INFO. The root logger does not override this. You have to
> manually
> # change the log levels here.
> logger.akka.name = akka
> logger.akka.level = INFO
> logger.kafka.name= org.apache.kafka
> logger.kafka.level = INFO
> logger.hadoop.name = org.apache.hadoop
> logger.hadoop.level = INFO
> logger.zookeeper.name = org.apache.zookeeper
> logger.zookeeper.level = INFO
>
> # Log all infos in the given file
> appender.main.name = MainAppender
> appender.main.type = File
> appender.main.append = false
> appender.main.fileName = ${sys:log.file}
> appender.main.layout.type = PatternLayout
> appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x -
> %m%n
>
> # Suppress the irrelevant (wrong) warnings from the Netty channel handler
> logger.netty.name =
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
> logger.netty.level = OFF
>
>
>
> On Tue, Nov 3, 2020 at 4:56 AM Yang Wang wrote:
>
>> You could issue "ps -ef | grep container_id_for_some_tm". And then you
>> will find the
>> following java options about log4j.
>>
>>
>> -Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log
>> -Dlog4j.configuration=file:./log4j.properties
>> -Dlog4j.configurationFile=file:./log4j.properties
>>
>> Best,
>> Yang
>>
>> Diwakar Jha 于2020年11月2日周一 下午11:37写道：
>>
>>> Sure. I will check that and get back to you. could you please share how
>>> to check java dynamic options?
>>>
>>> Best,
>>> Diwakar
>>>
>>> On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
>>>
If you have already updated the log4j.properties, and it still could
not work, then I
suggest to log in the Yarn NodeManager machine and check the
log4j.properties
in the container workdir is correct. Also you could have a look at the
java dynamic
options are correctly set.
I think it should work if the log4j.properties and java dynamic options
are set correctly.
BTW, could you share the new yarn logs?
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午4:32写道：
>
>
> Hi Yang,
>
> Thank you so much for taking a look at the log files. I changed my
> log4j.properties. Below is the actual file that I got from EMR 6.1.0
> distribution of flink 1.11. I observed that it is different from Flink
> 1.11
> that i downloaded so i changed it. Still I didn't see any logs.
>
> *Actual*
> log4j.rootLogger=INFO,file
>
> # Log all infos in the given
Re: Flink 1.11 not showing logs
HI Yang,
I'm able to see taskmanage and jobmanager logs after I changed the
log4j.properties file (/usr/lib/flink/conf). Thank you!
I updated the file as shown below. I had to kill the app ( yarn
application -kill ) and start flink job again to get the logs. This
doesn't seem like an efficient way. I was wondering if there's a more
simpler way to do it in production. let me know, please!
*Actual*
log4j.rootLogger=INFO,file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
*modified : *commented the above and added new logging from actual flink
application log4.properties file
#log4j.rootLogger=INFO,file
# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to
manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x -
%m%n
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name =
org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
On Tue, Nov 3, 2020 at 4:56 AM Yang Wang wrote:
> You could issue "ps -ef | grep container_id_for_some_tm". And then you
> will find the
> following java options about log4j.
>
>
> -Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log
> -Dlog4j.configuration=file:./log4j.properties
> -Dlog4j.configurationFile=file:./log4j.properties
>
> Best,
> Yang
>
> Diwakar Jha 于2020年11月2日周一 下午11:37写道：
>
>> Sure. I will check that and get back to you. could you please share how
>> to check java dynamic options?
>>
>> Best,
>> Diwakar
>>
>> On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
>>
>>> If you have already updated the log4j.properties, and it still could not
>>> work, then I
>>> suggest to log in the Yarn NodeManager machine and check the
>>> log4j.properties
>>> in the container workdir is correct. Also you could have a look at the
>>> java dynamic
>>> options are correctly set.
>>>
>>> I think it should work if the log4j.properties and java dynamic options
>>> are set correctly.
>>>
>>> BTW, could you share the new yarn logs?
>>>
>>> Best,
>>> Yang
>>>
>>> Diwakar Jha 于2020年11月2日周一 下午4:32写道：
>>>
Hi Yang,
Thank you so much for taking a look at the log files. I changed my
log4j.properties. Below is the actual file that I got from EMR 6.1.0
distribution of flink 1.11. I observed that it is different from Flink 1.11
that i downloaded so i changed it. Still I didn't see any logs.
*Actual*
log4j.rootLogger=INFO,file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd
HH:mm:ss,SSS} %-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel
handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
*modified : *commented the above and added new logging from
actual flink application log4.properties file
#log4j.rootLogger=INFO,file
# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
Re: Flink 1.11 not showing logs
You could issue "ps -ef | grep container_id_for_some_tm". And then you will
find the
following java options about log4j.
-Dlog.file=/var/log/hadoop-yarn/containers/application_xx/container_xx/taskmanager.log
-Dlog4j.configuration=file:./log4j.properties
-Dlog4j.configurationFile=file:./log4j.properties
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午11:37写道：
> Sure. I will check that and get back to you. could you please share how to
> check java dynamic options?
>
> Best,
> Diwakar
>
> On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
>
>> If you have already updated the log4j.properties, and it still could not
>> work, then I
>> suggest to log in the Yarn NodeManager machine and check the
>> log4j.properties
>> in the container workdir is correct. Also you could have a look at the
>> java dynamic
>> options are correctly set.
>>
>> I think it should work if the log4j.properties and java dynamic options
>> are set correctly.
>>
>> BTW, could you share the new yarn logs?
>>
>> Best,
>> Yang
>>
>> Diwakar Jha 于2020年11月2日周一 下午4:32写道：
>>
>>>
>>>
>>> Hi Yang,
>>>
>>> Thank you so much for taking a look at the log files. I changed my
>>> log4j.properties. Below is the actual file that I got from EMR 6.1.0
>>> distribution of flink 1.11. I observed that it is different from Flink 1.11
>>> that i downloaded so i changed it. Still I didn't see any logs.
>>>
>>> *Actual*
>>> log4j.rootLogger=INFO,file
>>>
>>> # Log all infos in the given file
>>> log4j.appender.file=org.apache.log4j.FileAppender
>>> log4j.appender.file.file=${log.file}
>>> log4j.appender.file.append=false
>>> log4j.appender.file.layout=org.apache.log4j.PatternLayout
>>> log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
>>> %-5p %-60c %x - %m%n
>>>
>>> # suppress the irrelevant (wrong) warnings from the netty channel handler
>>> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>>>
>>>
>>> *modified : *commented the above and added new logging from
>>> actual flink application log4.properties file
>>>
>>> #log4j.rootLogger=INFO,file
>>>
>>> # Log all infos in the given file
>>> #log4j.appender.file=org.apache.log4j.FileAppender
>>> #log4j.appender.file.file=${log.file}
>>> #log4j.appender.file.append=false
>>> #log4j.appender.file.layout=org.apache.log4j.PatternLayout
>>> #log4j.appender.file.layout.ConversionPattern=%d{-MM-dd
>>> HH:mm:ss,SSS} %-5p %-60c %x - %m%n
>>>
>>> # suppress the irrelevant (wrong) warnings from the netty channel handler
>>> #log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>>>
>>> # This affects logging for both user code and Flink
>>> rootLogger.level = INFO
>>> rootLogger.appenderRef.file.ref = MainAppender
>>>
>>> # Uncomment this if you want to _only_ change Flink's logging
>>> #logger.flink.name = org.apache.flink
>>> #logger.flink.level = INFO
>>>
>>> # The following lines keep the log level of common libraries/connectors
>>> on
>>> # log level INFO. The root logger does not override this. You have to
>>> manually
>>> # change the log levels here.
>>> logger.akka.name = akka
>>> logger.akka.level = INFO
>>> logger.kafka.name= org.apache.kafka
>>> logger.kafka.level = INFO
>>> logger.hadoop.name = org.apache.hadoop
>>> logger.hadoop.level = INFO
>>> logger.zookeeper.name = org.apache.zookeeper
>>> logger.zookeeper.level = INFO
>>>
>>> # Log all infos in the given file
>>> appender.main.name = MainAppender
>>> appender.main.type = File
>>> appender.main.append = false
>>> appender.main.fileName = ${sys:log.file}
>>> appender.main.layout.type = PatternLayout
>>> appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x
>>> - %m%n
>>>
>>> # Suppress the irrelevant (wrong) warnings from the Netty channel handler
>>> logger.netty.name =
>>> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
>>> logger.netty.level = OFF
>>>
>>> **
>>> I also think its related to the log4j setting but I'm not able to figure
>>> it out.
>>> Please let me know if you want any other log files or configuration.
>>>
>>> Thanks.
>>>
>>> On Sun, Nov 1, 2020 at 10:06 PM Yang Wang wrote:
>>>
Hi Diwakar Jha,
From the logs you have provided, everything seems working as expected.
The JobManager and TaskManager
java processes have been started with correct dynamic options,
especially for the logging.
Could you share the content of $FLINK_HOME/conf/log4j.properties? I
think there's something wrong with the
log4j config file. For example, it is a log4j1 format. But we are using
log4j2 in Flink 1.11.
Best,
Yang
Diwakar Jha 于2020年11月2日周一 上午1:57写道：
> Hi
> I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine
> but i'm not seeing any taskmanager/jobmanager logs.
> I see the below error in stdout.
> 18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR
>
Re: Flink 1.11 not showing logs
Sure. I will check that and get back to you. could you please share how to
check java dynamic options?
Best,
Diwakar
On Mon, Nov 2, 2020 at 1:33 AM Yang Wang wrote:
> If you have already updated the log4j.properties, and it still could not
> work, then I
> suggest to log in the Yarn NodeManager machine and check the
> log4j.properties
> in the container workdir is correct. Also you could have a look at the
> java dynamic
> options are correctly set.
>
> I think it should work if the log4j.properties and java dynamic options
> are set correctly.
>
> BTW, could you share the new yarn logs?
>
> Best,
> Yang
>
> Diwakar Jha 于2020年11月2日周一 下午4:32写道：
>
>>
>>
>> Hi Yang,
>>
>> Thank you so much for taking a look at the log files. I changed my
>> log4j.properties. Below is the actual file that I got from EMR 6.1.0
>> distribution of flink 1.11. I observed that it is different from Flink 1.11
>> that i downloaded so i changed it. Still I didn't see any logs.
>>
>> *Actual*
>> log4j.rootLogger=INFO,file
>>
>> # Log all infos in the given file
>> log4j.appender.file=org.apache.log4j.FileAppender
>> log4j.appender.file.file=${log.file}
>> log4j.appender.file.append=false
>> log4j.appender.file.layout=org.apache.log4j.PatternLayout
>> log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
>> %-5p %-60c %x - %m%n
>>
>> # suppress the irrelevant (wrong) warnings from the netty channel handler
>> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>>
>>
>> *modified : *commented the above and added new logging from actual flink
>> application log4.properties file
>>
>> #log4j.rootLogger=INFO,file
>>
>> # Log all infos in the given file
>> #log4j.appender.file=org.apache.log4j.FileAppender
>> #log4j.appender.file.file=${log.file}
>> #log4j.appender.file.append=false
>> #log4j.appender.file.layout=org.apache.log4j.PatternLayout
>> #log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
>> %-5p %-60c %x - %m%n
>>
>> # suppress the irrelevant (wrong) warnings from the netty channel handler
>> #log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>>
>> # This affects logging for both user code and Flink
>> rootLogger.level = INFO
>> rootLogger.appenderRef.file.ref = MainAppender
>>
>> # Uncomment this if you want to _only_ change Flink's logging
>> #logger.flink.name = org.apache.flink
>> #logger.flink.level = INFO
>>
>> # The following lines keep the log level of common libraries/connectors on
>> # log level INFO. The root logger does not override this. You have to
>> manually
>> # change the log levels here.
>> logger.akka.name = akka
>> logger.akka.level = INFO
>> logger.kafka.name= org.apache.kafka
>> logger.kafka.level = INFO
>> logger.hadoop.name = org.apache.hadoop
>> logger.hadoop.level = INFO
>> logger.zookeeper.name = org.apache.zookeeper
>> logger.zookeeper.level = INFO
>>
>> # Log all infos in the given file
>> appender.main.name = MainAppender
>> appender.main.type = File
>> appender.main.append = false
>> appender.main.fileName = ${sys:log.file}
>> appender.main.layout.type = PatternLayout
>> appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x
>> - %m%n
>>
>> # Suppress the irrelevant (wrong) warnings from the Netty channel handler
>> logger.netty.name =
>> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
>> logger.netty.level = OFF
>>
>> **
>> I also think its related to the log4j setting but I'm not able to figure
>> it out.
>> Please let me know if you want any other log files or configuration.
>>
>> Thanks.
>>
>> On Sun, Nov 1, 2020 at 10:06 PM Yang Wang wrote:
>>
>>> Hi Diwakar Jha,
>>>
>>> From the logs you have provided, everything seems working as expected.
>>> The JobManager and TaskManager
>>> java processes have been started with correct dynamic options,
>>> especially for the logging.
>>>
>>> Could you share the content of $FLINK_HOME/conf/log4j.properties? I
>>> think there's something wrong with the
>>> log4j config file. For example, it is a log4j1 format. But we are using
>>> log4j2 in Flink 1.11.
>>>
>>>
>>> Best,
>>> Yang
>>>
>>> Diwakar Jha 于2020年11月2日周一 上午1:57写道：
>>>
Hi
I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine
but i'm not seeing any taskmanager/jobmanager logs.
I see the below error in stdout.
18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR
org.apache.flink.runtime.rest.handler.taskmanager.
TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor
container_160404508_0001_01_04.
java.util.concurrent.CompletionException: org.apache.flink.util.
FlinkException: The file LOG does not exist on the TaskExecutor.
I'm stuck at this step for a couple of days now and not able to migrate
to Flink 1.11. I would appreciate it if anyone can help me.
i have the following setup :
a) i'm deploying flink using yarn. I
Re: Flink 1.11 not showing logs
If you have already updated the log4j.properties, and it still could not
work, then I
suggest to log in the Yarn NodeManager machine and check the
log4j.properties
in the container workdir is correct. Also you could have a look at the java
dynamic
options are correctly set.
I think it should work if the log4j.properties and java dynamic options are
set correctly.
BTW, could you share the new yarn logs?
Best,
Yang
Diwakar Jha 于2020年11月2日周一 下午4:32写道：
>
>
> Hi Yang,
>
> Thank you so much for taking a look at the log files. I changed my
> log4j.properties. Below is the actual file that I got from EMR 6.1.0
> distribution of flink 1.11. I observed that it is different from Flink 1.11
> that i downloaded so i changed it. Still I didn't see any logs.
>
> *Actual*
> log4j.rootLogger=INFO,file
>
> # Log all infos in the given file
> log4j.appender.file=org.apache.log4j.FileAppender
> log4j.appender.file.file=${log.file}
> log4j.appender.file.append=false
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # suppress the irrelevant (wrong) warnings from the netty channel handler
> log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>
>
> *modified : *commented the above and added new logging from actual flink
> application log4.properties file
>
> #log4j.rootLogger=INFO,file
>
> # Log all infos in the given file
> #log4j.appender.file=org.apache.log4j.FileAppender
> #log4j.appender.file.file=${log.file}
> #log4j.appender.file.append=false
> #log4j.appender.file.layout=org.apache.log4j.PatternLayout
> #log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # suppress the irrelevant (wrong) warnings from the netty channel handler
> #log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
>
> # This affects logging for both user code and Flink
> rootLogger.level = INFO
> rootLogger.appenderRef.file.ref = MainAppender
>
> # Uncomment this if you want to _only_ change Flink's logging
> #logger.flink.name = org.apache.flink
> #logger.flink.level = INFO
>
> # The following lines keep the log level of common libraries/connectors on
> # log level INFO. The root logger does not override this. You have to
> manually
> # change the log levels here.
> logger.akka.name = akka
> logger.akka.level = INFO
> logger.kafka.name= org.apache.kafka
> logger.kafka.level = INFO
> logger.hadoop.name = org.apache.hadoop
> logger.hadoop.level = INFO
> logger.zookeeper.name = org.apache.zookeeper
> logger.zookeeper.level = INFO
>
> # Log all infos in the given file
> appender.main.name = MainAppender
> appender.main.type = File
> appender.main.append = false
> appender.main.fileName = ${sys:log.file}
> appender.main.layout.type = PatternLayout
> appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x -
> %m%n
>
> # Suppress the irrelevant (wrong) warnings from the Netty channel handler
> logger.netty.name =
> org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
> logger.netty.level = OFF
>
> **
> I also think its related to the log4j setting but I'm not able to figure
> it out.
> Please let me know if you want any other log files or configuration.
>
> Thanks.
>
> On Sun, Nov 1, 2020 at 10:06 PM Yang Wang wrote:
>
>> Hi Diwakar Jha,
>>
>> From the logs you have provided, everything seems working as expected.
>> The JobManager and TaskManager
>> java processes have been started with correct dynamic options, especially
>> for the logging.
>>
>> Could you share the content of $FLINK_HOME/conf/log4j.properties? I think
>> there's something wrong with the
>> log4j config file. For example, it is a log4j1 format. But we are using
>> log4j2 in Flink 1.11.
>>
>>
>> Best,
>> Yang
>>
>> Diwakar Jha 于2020年11月2日周一 上午1:57写道：
>>
>>> Hi
>>> I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine
>>> but i'm not seeing any taskmanager/jobmanager logs.
>>> I see the below error in stdout.
>>> 18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR
>>> org.apache.flink.runtime.rest.handler.taskmanager.
>>> TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor
>>> container_160404508_0001_01_04.
>>> java.util.concurrent.CompletionException: org.apache.flink.util.
>>> FlinkException: The file LOG does not exist on the TaskExecutor.
>>>
>>> I'm stuck at this step for a couple of days now and not able to migrate
>>> to Flink 1.11. I would appreciate it if anyone can help me.
>>> i have the following setup :
>>> a) i'm deploying flink using yarn. I have attached yarn application id
>>> logs.
>>> c) stsd setup
>>>
>>> metrics.reporters: stsd
>>> metrics.reporter.stsd.factory.class:
>>> org.apache.flink.metrics.statsd.StatsDReporterFactory
>>> metrics.reporter.stsd.host: localhost
>>> metrics.reporter.stsd.port: 8125
>>>
>>>
Re: Flink 1.11 not showing logs
Hi Yang,
Thank you so much for taking a look at the log files. I changed my
log4j.properties. Below is the actual file that I got from EMR 6.1.0
distribution of flink 1.11. I observed that it is different from Flink 1.11
that i downloaded so i changed it. Still I didn't see any logs.
*Actual*
log4j.rootLogger=INFO,file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
*modified : *commented the above and added new logging from actual flink
application log4.properties file
#log4j.rootLogger=INFO,file
# Log all infos in the given file
#log4j.appender.file=org.apache.log4j.FileAppender
#log4j.appender.file.file=${log.file}
#log4j.appender.file.append=false
#log4j.appender.file.layout=org.apache.log4j.PatternLayout
#log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# suppress the irrelevant (wrong) warnings from the netty channel handler
#log4j.logger.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,file
# This affects logging for both user code and Flink
rootLogger.level = INFO
rootLogger.appenderRef.file.ref = MainAppender
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to
manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Log all infos in the given file
appender.main.name = MainAppender
appender.main.type = File
appender.main.append = false
appender.main.fileName = ${sys:log.file}
appender.main.layout.type = PatternLayout
appender.main.layout.pattern = %d{-MM-dd HH:mm:ss,SSS} %-5p %-60c %x -
%m%n
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name =
org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
**
I also think its related to the log4j setting but I'm not able to figure it
out.
Please let me know if you want any other log files or configuration.
Thanks.
On Sun, Nov 1, 2020 at 10:06 PM Yang Wang wrote:
> Hi Diwakar Jha,
>
> From the logs you have provided, everything seems working as expected. The
> JobManager and TaskManager
> java processes have been started with correct dynamic options, especially
> for the logging.
>
> Could you share the content of $FLINK_HOME/conf/log4j.properties? I think
> there's something wrong with the
> log4j config file. For example, it is a log4j1 format. But we are using
> log4j2 in Flink 1.11.
>
>
> Best,
> Yang
>
> Diwakar Jha 于2020年11月2日周一 上午1:57写道：
>
>> Hi
>> I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but
>> i'm not seeing any taskmanager/jobmanager logs.
>> I see the below error in stdout.
>> 18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR
>> org.apache.flink.runtime.rest.handler.taskmanager.
>> TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor
>> container_160404508_0001_01_04.
>> java.util.concurrent.CompletionException: org.apache.flink.util.
>> FlinkException: The file LOG does not exist on the TaskExecutor.
>>
>> I'm stuck at this step for a couple of days now and not able to migrate
>> to Flink 1.11. I would appreciate it if anyone can help me.
>> i have the following setup :
>> a) i'm deploying flink using yarn. I have attached yarn application id
>> logs.
>> c) stsd setup
>>
>> metrics.reporters: stsd
>> metrics.reporter.stsd.factory.class:
>> org.apache.flink.metrics.statsd.StatsDReporterFactory
>> metrics.reporter.stsd.host: localhost
>> metrics.reporter.stsd.port: 8125
>>
>>
Re: Flink 1.11 not showing logs
Hi Diwakar Jha, >From the logs you have provided, everything seems working as expected. The JobManager and TaskManager java processes have been started with correct dynamic options, especially for the logging. Could you share the content of $FLINK_HOME/conf/log4j.properties? I think there's something wrong with the log4j config file. For example, it is a log4j1 format. But we are using log4j2 in Flink 1.11. Best, Yang Diwakar Jha 于2020年11月2日周一 上午1:57写道： > Hi > I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but > i'm not seeing any taskmanager/jobmanager logs. > I see the below error in stdout. > 18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR > org.apache.flink.runtime.rest.handler.taskmanager. > TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor > container_160404508_0001_01_04. > java.util.concurrent.CompletionException: org.apache.flink.util. > FlinkException: The file LOG does not exist on the TaskExecutor. > > I'm stuck at this step for a couple of days now and not able to migrate to > Flink 1.11. I would appreciate it if anyone can help me. > i have the following setup : > a) i'm deploying flink using yarn. I have attached yarn application id > logs. > c) stsd setup > > metrics.reporters: stsd > metrics.reporter.stsd.factory.class: > org.apache.flink.metrics.statsd.StatsDReporterFactory > metrics.reporter.stsd.host: localhost > metrics.reporter.stsd.port: 8125 > >
Flink 1.11 not showing logs
Hi I'm running Flink 1.11 on EMR 6.1.0. I can see my job is running fine but i'm not seeing any taskmanager/jobmanager logs. I see the below error in stdout. 18:29:19.834 [flink-akka.actor.default-dispatcher-28] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_160404508_0001_01_04. java.util.concurrent.CompletionException: org.apache.flink.util. FlinkException: The file LOG does not exist on the TaskExecutor. I'm stuck at this step for a couple of days now and not able to migrate to Flink 1.11. I would appreciate it if anyone can help me. i have the following setup : a) i'm deploying flink using yarn. I have attached yarn application id logs. c) stsd setup metrics.reporters: stsd metrics.reporter.stsd.factory.class: org.apache.flink.metrics.statsd.StatsDReporterFactory metrics.reporter.stsd.host: localhost metrics.reporter.stsd.port: 8125 yarn Description: Binary data
