Interesting that we came across this problem at the same time. We have
observed this with Lyft's K8s operator which uses the Rest API for job
submission, much like the Flink dashboard.
Note that you can restore the original stdout/stderr in your program:
private static void restoreStdOutAndStdErr() {
System.setOut(new PrintStream(
new FileOutputStream(FileDescriptor.out)));
System.setErr(new PrintStream(
new FileOutputStream(FileDescriptor.err)));
}
Just call restoreStdOutAndStdErr() before you start building the Flink
job. Of course, this is just meant to be a workaround.
I think an acceptable solution is to always print upon execution. For
the plan preview we may keep the existing behavior.
Cheers,
Max
On 07.01.20 17:39, Dawid Wysakowicz wrote:
A quick update. The suppression of stdout/stderr actually might soon be
dropped, see: https://issues.apache.org/jira/browse/FLINK-15504
Best,
Dawid
On 07/01/2020 07:17, Yang Wang wrote:
Hi Bajaj,
I have tested just as you say, and find that the logs in the user
class could not show up when
using ConsoleAppender. If using FileAppender instead, everything goes
well.
It is so weird and i have no idea how to debug it.
Best,
Yang
Bajaj, Abhinav <abhinav.ba...@here.com
<mailto:abhinav.ba...@here.com>> 于2020年1月7日周二 上午4:28写道：
Hi,
Thanks much for the responses.
Let me add some more details and clarify my question.
_Setup_
* I used the WikipediaAnalysis example and added a log in main
method.
……
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment see =
StreamExecutionEnvironment./getExecutionEnvironment/();
/LOG/.info("Info log for test");
DataStream<WikipediaEditEvent> edits = see.addSource(new
WikipediaEditsSource());
……
* I am using the Flink 1.7.1 distribution and starting
jobmanager and taskmanager locally using the below commands –
o ./bin/jobmanager.sh start-foreground
o ./bin/taskmanager.sh start-foreground
o Both jobmanager and taskmanager log in the console now
o JVM options are correctly set and verified from jobmanager
& taskmanager logs
* I submit the WikipediaAnalysis job from Flink dashboard and
checked the jobmanager logs
_Run 1_: Flink is using the default log4j logging
* Jobmanager logs the added info log from the job
o 2020-01-06 11:55:37,422 INFO wikiedits.WikipediaAnalysis -
Info log for test
_Run 2_: Flink is setup to use logback as suggested in Flink
documentation here
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster>
* Jobmanger does not log the added info log from the job
So, it seems there is a logging behavior difference between using
log4j & logback in Flink.
Is this expected or a known difference?
Thanks again,
Abhinav Bajaj
_PS_: Ahh. I see how my email was confusing the first time.
Hopefully this one is better :P
*From: *Dawid Wysakowicz <dwysakow...@apache.org
<mailto:dwysakow...@apache.org>>
*Date: *Monday, January 6, 2020 at 5:13 AM
*Cc: *"Bajaj, Abhinav" <abhinav.ba...@here.com
<mailto:abhinav.ba...@here.com>>, "user@flink.apache.org
<mailto:user@flink.apache.org>" <user@flink.apache.org
<mailto:user@flink.apache.org>>
*Subject: *Re: Flink logging issue with logback
Hi Bajaj,
I am not entirely sure what is the actual issue you are seeking
help, but let me comment on your observations.
Ad. 1
If you log to the console from the main method this is an expected
behavior in both cases (log4j, logback). The std out is being
overwritten for the execution of the main method
If you log to a file logs should work in both cases. I checked
that myself and actually the logs appeared in the jobmanager logs
as long as they are executed before the env.execute(). I observed
though weird behavior of the Web UI, as it does not always update
the logs that are displayed. How did you check the logs? If you
checked through the Web UI could you try to check the file directly?
Ad. 2 Yes this is expected. Operators are executed on taskmanager
and that's why they log there.
Ad. 3 See Ad. 1
Best,
Dawid
On 06/01/2020 07:07, vino yang wrote:
Hi Bajaj,
>> Logs from main method(outside of job graph) do not show up
in jobmanager logs.
IMO, it's normal phenomena.
Other ideas, please check the JVM options mentioned by Yang.
Best,
Vino
Yang Wang <danrtsey...@gmail.com
<mailto:danrtsey...@gmail.com>> 于2020年1月6日周一 上午11:18写道：
Hi Bajaj, Abhinav,
Could you share the start-command of jobmanager and
taskmanager. If it is started correctly, we
will have a the following jvm options.
-Dlog.file=/path/of/taskmanager.log
-Dlogback.configurationFile=file:///path/of/logback.xml
Best,
Yang
Bajaj, Abhinav <abhinav.ba...@here.com
<mailto:abhinav.ba...@here.com>> 于2020年1月4日周六 上午
7:23写道：
Hi,
I am investigating a logging issue with Flink.
_Setup_
* Using Flink-1.7.1 using logback as suggested in
Flink documentation here
<https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster>.
* Submitting the Flink job from the Flink dashboard.
_Observations_
* Logs from main method(outside of job graph) do not
show up in jobmanager logs.
* Logs from the operators like map or custom
operators do show up in the taskmanager logs.
* Logs from main method do show up in jobmanager
logs when using log4j in place of logback.
Has anyone else noticed similar behavior or is this a
known issue with logback integration in Flink?
Any suggestions on potential workaround or fix?
Appreciate your time and help.
~ Abhinav Bajaj
