A quick update. The suppression of stdout/stderr actually might soon be dropped, see: https://issues.apache.org/jira/browse/FLINK-15504
Best, Dawid On 07/01/2020 07:17, Yang Wang wrote: > Hi Bajaj, > > I have tested just as you say, and find that the logs in the user > class could not show up when > using ConsoleAppender. If using FileAppender instead, everything goes > well. > > It is so weird and i have no idea how to debug it. > Best, > Yang > > Bajaj, Abhinav <abhinav.ba...@here.com > <mailto:abhinav.ba...@here.com>> 于2020年1月7日周二 上午4:28写道： > > Hi, > > > > Thanks much for the responses. > > Let me add some more details and clarify my question. > > > > _Setup_ > > * I used the WikipediaAnalysis example and added a log in main > method. > > …… > > public static void main(String[] args) throws Exception { > StreamExecutionEnvironment see = > StreamExecutionEnvironment./getExecutionEnvironment/(); > /LOG/.info("Info log for test"); > > DataStream<WikipediaEditEvent> edits = see.addSource(new > WikipediaEditsSource()); > > …… > > * I am using the Flink 1.7.1 distribution and starting > jobmanager and taskmanager locally using the below commands – > o ./bin/jobmanager.sh start-foreground > o ./bin/taskmanager.sh start-foreground > o Both jobmanager and taskmanager log in the console now > o JVM options are correctly set and verified from jobmanager > & taskmanager logs > > > > * I submit the WikipediaAnalysis job from Flink dashboard and > checked the jobmanager logs > > > > _Run 1_: Flink is using the default log4j logging > > * Jobmanager logs the added info log from the job > o 2020-01-06 11:55:37,422 INFO > wikiedits.WikipediaAnalysis > - Info log for test > > > > _Run 2_: Flink is setup to use logback as suggested in Flink > documentation here > > <https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster> > > * Jobmanger does not log the added info log from the job > > > > So, it seems there is a logging behavior difference between using > log4j & logback in Flink. > > Is this expected or a known difference? > > > > Thanks again, > > Abhinav Bajaj > > > > > > _PS_: Ahh. I see how my email was confusing the first time. > Hopefully this one is better :P > > > > > > *From: *Dawid Wysakowicz <dwysakow...@apache.org > <mailto:dwysakow...@apache.org>> > *Date: *Monday, January 6, 2020 at 5:13 AM > *Cc: *"Bajaj, Abhinav" <abhinav.ba...@here.com > <mailto:abhinav.ba...@here.com>>, "user@flink.apache.org > <mailto:user@flink.apache.org>" <user@flink.apache.org > <mailto:user@flink.apache.org>> > *Subject: *Re: Flink logging issue with logback > > > > Hi Bajaj, > > I am not entirely sure what is the actual issue you are seeking > help, but let me comment on your observations. > > Ad. 1 > > If you log to the console from the main method this is an expected > behavior in both cases (log4j, logback). The std out is being > overwritten for the execution of the main method > > If you log to a file logs should work in both cases. I checked > that myself and actually the logs appeared in the jobmanager logs > as long as they are executed before the env.execute(). I observed > though weird behavior of the Web UI, as it does not always update > the logs that are displayed. How did you check the logs? If you > checked through the Web UI could you try to check the file directly? > > Ad. 2 Yes this is expected. Operators are executed on taskmanager > and that's why they log there. > > Ad. 3 See Ad. 1 > > Best, > > Dawid > > > > On 06/01/2020 07:07, vino yang wrote: > > Hi Bajaj, > > > > >> Logs from main method(outside of job graph) do not show up > in jobmanager logs. > > > > IMO, it's normal phenomena. > > > > Other ideas, please check the JVM options mentioned by Yang. > > > > Best, > > Vino > > > > > > Yang Wang <danrtsey...@gmail.com > <mailto:danrtsey...@gmail.com>> 于2020年1月6日周一 上午11:18写道： > > Hi Bajaj, Abhinav, > > > > Could you share the start-command of jobmanager and > taskmanager. If it is started correctly, we > > will have a the following jvm options. > > > > -Dlog.file=/path/of/taskmanager.log > -Dlogback.configurationFile=file:///path/of/logback.xml > > > > > > > > Best, > > Yang > > > > Bajaj, Abhinav <abhinav.ba...@here.com > <mailto:abhinav.ba...@here.com>> 于2020年1月4日周六 > 上午7:23写道： > > Hi, > > > > I am investigating a logging issue with Flink. > > > > _Setup_ > > * Using Flink-1.7.1 using logback as suggested in > Flink documentation here > > <https://ci.apache.org/projects/flink/flink-docs-stable/dev/best_practices.html#use-logback-when-running-flink-on-a-cluster>. > * Submitting the Flink job from the Flink dashboard. > > > > _Observations_ > > * Logs from main method(outside of job graph) do not > show up in jobmanager logs. > * Logs from the operators like map or custom > operators do show up in the taskmanager logs. > * Logs from main method do show up in jobmanager > logs when using log4j in place of logback. > > > > Has anyone else noticed similar behavior or is this a > known issue with logback integration in Flink? > > Any suggestions on potential workaround or fix? > > > > Appreciate your time and help. > > > > ~ Abhinav Bajaj > > >
signature.asc
Description: OpenPGP digital signature
