Re: Flink Standalone cluster - logging problem
Hi,
Are you logging from your own operator implementations, and you expect these
log messages to end up in a file prefixed with XYZ-? If that is the case,
modifying log4j-cli.properties will not be expedient as I wrote earlier.
You should modify the log4j.properties on all hosts that are running the
JobManager (JM) and TaskManagers (TM). Consequently, the log files can only
be
found on the hosts that are running the JM and TMs.
Furthermore, I see a problem with the following line in your log4j
configuration:
log4j.appender.file2.file=XYZ-${log.file}
Here, ${log.file} can be an absolute path, which means you would end up
prefixing the whole path instead of just the filename.
If this does not help, please share a minimum working example with us.
Best,
Gary
On Mon, Feb 11, 2019 at 12:02 PM simpleusr wrote:
> Hi Gary,
>
> By "job logs" I mean all the loggers under a subpackage of
> com.mycompany.xyz
> .
>
> We are using ./bin/flink run command for job execution thats why I modified
> log4j-cli.properties. Modification of log4j.properties also did not help...
>
> Regards
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Flink Standalone cluster - logging problem
Hi Selveraj, This did not help either. Thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink Standalone cluster - logging problem
Could you pls try modifying conf/logback.xml . Regards, Selvaraj C On Mon, Feb 11, 2019 at 4:32 PM simpleusr wrote: > Hi Gary, > > By "job logs" I mean all the loggers under a subpackage of > com.mycompany.xyz > . > > We are using ./bin/flink run command for job execution thats why I modified > log4j-cli.properties. Modification of log4j.properties also did not help... > > Regards > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ > -- Regards, Selvaraj C
Re: Flink Standalone cluster - logging problem
Hi Gary, By "job logs" I mean all the loggers under a subpackage of com.mycompany.xyz . We are using ./bin/flink run command for job execution thats why I modified log4j-cli.properties. Modification of log4j.properties also did not help... Regards -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink Standalone cluster - logging problem
Hi,
Can you define what you mean by "job logs"? For code that is run on the
cluster, i.e., JM or TM, you should add your config to log4j.properties. The
log4j-cli.properties file is only used by the Flink CLI process.
Best,
Gary
On Mon, Feb 11, 2019 at 7:39 AM simpleusr wrote:
> Hi Chesnay,
>
> below is the content for my log4j-cli.properties file. I expect my job logs
> (packaged under com.mycompany.xyz to be written to file2 appender. However
> no file generated with prefix XYZ. I restarted the cluster , canceled
> resubmitted several times but none of them helped.
>
>
> /
> log4j.rootLogger=INFO, file
>
> # Log all infos in the given file
> log4j.appender.file=org.apache.log4j.FileAppender
> log4j.appender.file.file=${log.file}
> log4j.appender.file.append=false
> log4j.appender.file.layout=org.apache.log4j.PatternLayout
> log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> log4j.appender.file2=org.apache.log4j.FileAppender
> log4j.appender.file2.file=XYZ-${log.file}
> log4j.appender.file2.append=false
> log4j.appender.file2.layout=org.apache.log4j.PatternLayout
> log4j.appender.file2.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
>
> # Log output from org.apache.flink.yarn to the console. This is used by the
> # CliFrontend class when using a per-job YARN cluster.
> log4j.logger.org.apache.flink.yarn=INFO, console
> log4j.logger.org.apache.flink.yarn.cli.FlinkYarnSessionCli=INFO, console
> log4j.logger.org.apache.hadoop=INFO, console
>
> log4j.appender.console=org.apache.log4j.ConsoleAppender
> log4j.appender.console.layout=org.apache.log4j.PatternLayout
> log4j.appender.console.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
> %-5p %-60c %x - %m%n
>
> # suppress the warning that hadoop native libraries are not loaded
> (irrelevant for the client)
> log4j.logger.org.apache.hadoop.util.NativeCodeLoader=OFF
>
> # suppress the irrelevant (wrong) warnings from the netty channel handler
>
> log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,
> file
>
>
> log4j.logger.com.hazelcast=INFO, file2
> log4j.logger.com.mycompany.xyz=DEBUG, file2/
>
>
>
> --
> Sent from:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
>
Re: Flink Standalone cluster - logging problem
Hi Chesnay,
below is the content for my log4j-cli.properties file. I expect my job logs
(packaged under com.mycompany.xyz to be written to file2 appender. However
no file generated with prefix XYZ. I restarted the cluster , canceled
resubmitted several times but none of them helped.
/
log4j.rootLogger=INFO, file
# Log all infos in the given file
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.file=${log.file}
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
log4j.appender.file2=org.apache.log4j.FileAppender
log4j.appender.file2.file=XYZ-${log.file}
log4j.appender.file2.append=false
log4j.appender.file2.layout=org.apache.log4j.PatternLayout
log4j.appender.file2.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# Log output from org.apache.flink.yarn to the console. This is used by the
# CliFrontend class when using a per-job YARN cluster.
log4j.logger.org.apache.flink.yarn=INFO, console
log4j.logger.org.apache.flink.yarn.cli.FlinkYarnSessionCli=INFO, console
log4j.logger.org.apache.hadoop=INFO, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{-MM-dd HH:mm:ss,SSS}
%-5p %-60c %x - %m%n
# suppress the warning that hadoop native libraries are not loaded
(irrelevant for the client)
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=OFF
# suppress the irrelevant (wrong) warnings from the netty channel handler
log4j.logger.org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline=ERROR,
file
log4j.logger.com.hazelcast=INFO, file2
log4j.logger.com.mycompany.xyz=DEBUG, file2/
--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: Flink Standalone cluster - logging problem
What exactly are you expecting to happen? On 08.02.2019 15:06, simpleusr wrote: We are using standalone cluster and submittig jobs through command line client. As stated in https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html , we are editing log4j-cli.properties but this does not make any effect? Anybody seen that before? Regards -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Flink Standalone cluster - logging problem
We are using standalone cluster and submittig jobs through command line client. As stated in https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/logging.html , we are editing log4j-cli.properties but this does not make any effect? Anybody seen that before? Regards -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
