Hi Chesnay,
Thanks for the reply.
1. The internal logging framework is built upon slf4j/log4j2 (The same one
Flink uses, but it comes with an additional POM dependency). I have added such
dependency in the Flink application POM file. But it seems only to work locally
in IDE. When it is in the Flink cluster environment, it can’t work.
2. I tried to only add the configmap and put a single jar into lib/ folder,
and it seems it still can’t find the classpath. How should I organize the
folder structure? /lib/internal-logging/xxx.jar or this jar file must be
directly under /lib, something like /lib/xxx.jar?
3. I got you point, I guess it is stilling using Flink default logging
classpath and that causes the issue of not recognizing the internal framework?
How to check the classpath of the Flink logging? Could you share me some
blogs..? I am not familiar with this.
Best,
Fuyao
From: Chesnay Schepler <ches...@apache.org>
Date: Tuesday, September 28, 2021 at 07:06
To: Fuyao Li <fuyao...@oracle.com>, user <user@flink.apache.org>
Cc: Rohit Gupta <rohit.c.gu...@oracle.com>
Subject: [External] : Re: How to enable customize logging library based on
SLF4J for Flink deployment in Kubernetes
Could you clarify whether this internal framework uses a custom slfj4/log4j2
version, or is it just using what Flink comes with?
Did you only add the configmap and put a single jar into lib, or did you make
other changes in Flink?
Can you remove just the configmap, start the cluster, and provide us with the
classpath that Flink is logging?
On 25/09/2021 01:57, Fuyao Li wrote:
Hi Flink Community,
I am trying enable a company internal logging framework built upon SLF4J and
log4j. This logging framework has another separate jar and specific logging
configurations. After debugging, I am able to make Flink application running
correctly in the local IDE with the internal logging framework after adding
related SLF4J, log4j dependencies, and logging framework dependencies.
However, I still run into errors when I deploy this into the Kubernetes
environment. I tried to add the logging framework jar to /opt/flink/lib/
folder, but it doesn’t help much. I am not sure which part I am missing here. I
have attached relevant information below. Thanks for your help.
This is the log4j2-console.properties I proposed, I have injected this as a
configmap (mounted to /opt/flink/conf inside the pod using a Flink native
Kubernetes Operator I build).
Such configuration will run correctly in Local IDE and generate logs in the
internal logging framework expected shape. (I have rename it to
log4j2.properties and put it into resources/ folder during local debug.)
packages = oracle.spectra.logging.base
status = WARN
monitorInterval = 30
shutdownHook = disable
rootLogger.level = ${sys:spectra-log-level:-INFO}
rootLogger.appenderRef.asyncC.ref = AsyncCAppender
rootLogger.appenderRef.asyncF.ref = AsyncFAppender
appender.asyncC.name = AsyncCAppender
appender.asyncC.type = Async
appender.asyncC.bufferSize = 256
appender.asyncC.appenderRef.type = AppenderRef
appender.asyncC.appenderRef.ref = JSONLogConsoleAppender
# Log all infos to the console
appender.console.name = JSONLogConsoleAppender
appender.console.target = SYSTEM_OUT
appender.console.type = Console
appender.console.layout.type = SpectraJsonLayout
appender.console.layout.compact = true
appender.console.layout.eventEol = true
appender.asyncF.name = AsyncFAppender
appender.asyncF.type = Async
appender.asyncF.bufferSize = 256
appender.asyncF.appenderRef.type = AppenderRef
appender.asyncF.appenderRef.ref = RollingFileAppender
# Log all infos in the given rolling file
appender.rolling.type = RollingFile
appender.rolling.name = RollingFileAppender
appender.rolling.fileName = ${sys:log.file}
appender.rolling.filePattern = ${sys:log.file}.%i
appender.rolling.layout.type = SpectraJsonLayout
appender.rolling.layout.compact = false
appender.rolling.layout.eventEol = true
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 10
# Uncomment this if you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override this. You have to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name =
org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
This is the error I got from the Job Manager pod in the Kubernetes.
sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o: Read-only file
system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file
system
sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn: Read-only file
system
/docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml: Read-only file
system
/docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp: Read-only
file system
Starting kubernetes-application as a console application on host
faw-poc-demo-67864b696b-9zbbc.
2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type for
SpectraJsonLayout
2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type for
SpectraJsonLayout
2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,689 main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.RollingFileAppender for element
RollingFile: java.lang.NullPointerException java.lang.NullPointerException
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method in class
org.apache.logging.log4j.core.appender.RollingFileAppender for element
RollingFile: java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.RollingFileAppender
java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.RollingFileAppender
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for SpectraJsonLayout
2021-09-24 23:42:53,758 main ERROR Could not create plugin of type class
org.apache.logging.log4j.core.appender.ConsoleAppender for element Console:
java.lang.NullPointerException java.lang.NullPointerException
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method in class
org.apache.logging.log4j.core.appender.ConsoleAppender for element Console:
java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.ConsoleAppender
java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.ConsoleAppender
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,766 main ERROR Null object returned for RollingFile in
Appenders.
2021-09-24 23:42:53,766 main ERROR Null object returned for Console in
Appenders.
2021-09-24 23:42:53,771 main ERROR No appender named RollingFileAppender was
configured
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: org.apache.logging.log4j.core.config.ConfigurationException: No
appenders are available for AsyncAppender AsyncFAppender
at
org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
As you can see, the SpectraJsonLayout that is defined in the logging framework
jar is somehow not recognized. I am not sure how to solve this? Could anyone
share some information. Really appreciate your help!
Best regards,
Fuyao
