Could you clarify whether this internal framework uses a custom
slfj4/log4j2 version, or is it just using what Flink comes with?
Did you only add the configmap and put a single jar into lib, or did you
make other changes in Flink?
Can you remove just the configmap, start the cluster, and provide us
with the classpath that Flink is logging?
On 25/09/2021 01:57, Fuyao Li wrote:
Hi Flink Community,
I am trying enable a company internal logging framework built upon
SLF4J and log4j. This logging framework has another separate jar and
specific logging configurations. After debugging, I am able to make
Flink application running correctly in the local IDE with the internal
logging framework after adding related SLF4J, log4j dependencies, and
logging framework dependencies.
However, I still run into errors when I deploy this into the
Kubernetes environment. I tried to add the logging framework jar to
/opt/flink/lib/ folder, but it doesn’t help much. I am not sure which
part I am missing here. I have attached relevant information below.
Thanks for your help.
This is the log4j2-console.properties I proposed, I have injected this
as a configmap (mounted to /opt/flink/conf inside the pod using a
Flink native Kubernetes Operator I build).
Such configuration will run correctly in Local IDE and generate logs
in the internal logging framework expected shape. (I have rename it to
log4j2.properties and put it into resources/ folder during local debug.)
packages = oracle.spectra.logging.base
status = WARN
monitorInterval = 30
shutdownHook = disable
rootLogger.level = ${sys:spectra-log-level:-INFO}
rootLogger.appenderRef.asyncC.ref = AsyncCAppender
rootLogger.appenderRef.asyncF.ref = AsyncFAppender
appender.asyncC.name = AsyncCAppender
appender.asyncC.type = Async
appender.asyncC.bufferSize = 256
appender.asyncC.appenderRef.type = AppenderRef
appender.asyncC.appenderRef.ref = JSONLogConsoleAppender
# Log all infos to the console
appender.console.name = JSONLogConsoleAppender
appender.console.target = SYSTEM_OUT
appender.console.type = Console
appender.console.layout.type = SpectraJsonLayout
appender.console.layout.compact = *true*
appender.console.layout.eventEol = *true*
appender.asyncF.name = AsyncFAppender
appender.asyncF.type = Async
appender.asyncF.bufferSize = 256
appender.asyncF.appenderRef.type = AppenderRef
appender.asyncF.appenderRef.ref = RollingFileAppender
# Log all infos in the given rolling file
appender.rolling.type = RollingFile
appender.rolling.name = RollingFileAppender
appender.rolling.fileName = ${sys:log.file}
appender.rolling.filePattern = ${sys:log.file}.%i
appender.rolling.layout.type = SpectraJsonLayout
appender.rolling.layout.compact = *false*
appender.rolling.layout.eventEol = *true*
appender.rolling.policies.type = Policies
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size=100MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.max = 10
# Uncomment *this**if*you want to _only_ change Flink's logging
#logger.flink.name = org.apache.flink
#logger.flink.level = INFO
# The following lines keep the log level of common libraries/connectors on
# log level INFO. The root logger does not override *this*. You have
to manually
# change the log levels here.
logger.akka.name = akka
logger.akka.level = INFO
logger.kafka.name= org.apache.kafka
logger.kafka.level = INFO
logger.hadoop.name = org.apache.hadoop
logger.hadoop.level = INFO
logger.zookeeper.name = org.apache.zookeeper
logger.zookeeper.level = INFO
# Suppress the irrelevant (wrong) warnings from the Netty channel handler
logger.netty.name =
org.apache.flink.shaded.akka.org.jboss.netty.channel.DefaultChannelPipeline
logger.netty.level = OFF
This is the error I got from the Job Manager pod in the Kubernetes.
sed: couldn't open temporary file /opt/flink/conf/sedAHNLHl: Read-only
file system
sed: couldn't open temporary file /opt/flink/conf/sedBkNR6o: Read-only
file system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml:
Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedMGJAkn: Read-only
file system
/docker-entrypoint.sh: line 86: /opt/flink/conf/flink-conf.yaml:
Read-only file system
/docker-entrypoint.sh: line 88: /opt/flink/conf/flink-conf.yaml.tmp:
Read-only file system
Starting kubernetes-application as a console application on host
faw-poc-demo-67864b696b-9zbbc.
2021-09-24 23:42:53,579 main ERROR Unable to locate plugin type for
SpectraJsonLayout
2021-09-24 23:42:53,582 main ERROR Unable to locate plugin type for
SpectraJsonLayout
2021-09-24 23:42:53,675 main ERROR Unable to locate plugin for
SpectraJsonLayout
2021-09-24 23:42:53,689 main ERROR Could not create plugin of type
class org.apache.logging.log4j.core.appender.RollingFileAppender for
element RollingFile: java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,751 main ERROR Unable to invoke factory method in
class org.apache.logging.log4j.core.appender.RollingFileAppender for
element RollingFile: java.lang.IllegalStateException: No factory
method found for class
org.apache.logging.log4j.core.appender.RollingFileAppender
java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.RollingFileAppender
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,754 main ERROR Unable to locate plugin for
SpectraJsonLayout
2021-09-24 23:42:53,758 main ERROR Could not create plugin of type
class org.apache.logging.log4j.core.appender.ConsoleAppender for
element Console: java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.findNamedNode(PluginElementVisitor.java:104)
at
org.apache.logging.log4j.core.config.plugins.visitors.PluginElementVisitor.visit(PluginElementVisitor.java:88)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.injectFields(PluginBuilder.java:185)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:121)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,760 main ERROR Unable to invoke factory method in
class org.apache.logging.log4j.core.appender.ConsoleAppender for
element Console: java.lang.IllegalStateException: No factory method
found for class org.apache.logging.log4j.core.appender.ConsoleAppender
java.lang.IllegalStateException: No factory method found for class
org.apache.logging.log4j.core.appender.ConsoleAppender
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.findFactoryMethod(PluginBuilder.java:234)
at
org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:134)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
2021-09-24 23:42:53,766 main ERROR Null object returned for
RollingFile in Appenders.
2021-09-24 23:42:53,766 main ERROR Null object returned for Console in
Appenders.
2021-09-24 23:42:53,771 main ERROR No appender named
RollingFileAppender was configured
Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by:
org.apache.logging.log4j.core.config.ConfigurationException: No
appenders are available for AsyncAppender AsyncFAppender
at
org.apache.logging.log4j.core.appender.AsyncAppender.start(AsyncAppender.java:123)
at
org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:304)
at
org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
at
org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
at
org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at
org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at
org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
at
org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
at
org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
at
org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
at
org.apache.flink.runtime.entrypoint.ClusterEntrypoint.<clinit>(ClusterEntrypoint.java:108)
As you can see, the SpectraJsonLayout that is defined in the logging
framework jar is somehow not recognized. I am not sure how to solve
this? Could anyone share some information. Really appreciate your help!
Best regards,
Fuyao
