hmm....yes then it is indeed weird that it can't find the logger, but
their error messages are notorious for being misleading in my experience.
Can you set the log4j2.debug system property (to any value, even an
empty string) and try again?
If that doesn't reveal anything I would try the following:
* run a job without bundling any logging classes
* if that works, add one jar at a time to the fat jar and see if it
continues to work
* check what happens if the logging classes are added to lib/ instead of
being bundled in the fat jar
You can also test all of this with a standalone cluster and a dummy job
if that makes things easier; I don't think there is anything
Kubernetes-specific going on.
On 15/02/2022 18:18, jonas eyob wrote:
1. Ok, thanks!
2. We are using application mode. No changes to the distribution other
than updating the log4j-console.properties file.
content of /lib/:
* flink-csv-1.14.3.jar
* flink-json-1.14.3.jar
* flink-table_2.12-1.14.3.jar
* log4j-api-2.17.1.jar
* log4j-slf4j-impl-2.17.1.jar
* flink-dist_2.12-1.14.3.jar
* flink-shaded-zookeeper-3.4.14.jar
* log4j-1.2-api-2.17.1.jar
* log4j-core-2.17.1.jar
Den tis 15 feb. 2022 kl 16:30 skrev Chesnay Schepler <ches...@apache.org>:
1) You either need to modify the log4j-console.properties file, or
explicitly set the log4j.configurationFile property to point to
your .xml file.
2)
Have you made modifications to the distribution (e.g., removing
other logging jars from the lib directory)?
Are you using application mode, or session clusters?
On 15/02/2022 16:41, jonas eyob wrote:
Hey,
We are deploying our Flink Cluster on a standalone Kubernetes
with the longrunning job written in scala.
We recently upgraded our Flink cluster from 1.12 to 1.14.3 -
after which we started seeing a few problems related to logging
which I have been struggling to fix for the past days).
Related is also an attempt to add, we are also attempting to add
a Sentry integration for our error logs.
PROBLEM 1 - Error logs not being sent to Sentry.
We are bundling our code and dependencies into a FAT jar, which
includes a log4j2.xml specifying the Sentry Appender. But if I
understand the documentation
<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/advanced/logging/#configuring-log4j-2>
correctly our log4j2.xml won't be picked up by Flink as it
already defines a set of default logging configurations files
(e.g. log4j and logback).
Q: How does Flink resolve logging configurations to use?
I can see the following JVM override params provided when running
in our dockerized version locally.
-Dlog.file=/opt/flink/log/flink--taskexecutor-0-thoros-taskmanager-6b9785d4df-c28n4.log
2022-02-15 10:01:59,826 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] -
-Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2022-02-15 10:01:59,827 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] -
-Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2022-02-15 10:01:59,830 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] -
-Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
Content of the log4j2.xml (path: src/main/resources):
<?xml version="1.0" encoding="UTF-8"?> <Configuration
status="warn"
packages="org.apache.logging.log4j.core,io.sentry.log4j2">
<Appenders> <Console name="Console" target="SYSTEM_OUT">
<PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level
%logger{36} - %msg%n"/> </Console> <Sentry name="Sentry"
dsn="<DSN>" minimumEventLevel="ERROR" /> </Appenders> <Loggers>
<Root level="info"> <AppenderRef ref="Sentry"/> <AppenderRef
ref="Console"/> </Root> </Loggers> </Configuration>
For our kubernetes deployment we have followed the reference
example here
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/#common-cluster-resource-definitions.
My assumption is that I would need to also provide the
Sentry-related configuration to the "log4-console.properties" for
it to be picked up by the Taskmanager and JobManager?
PROBLEM 2:
ERROR StatusLogger Log4j2 could not find a logging implementation.
Please add log4j-core to the classpath. Using SimpleLogger to log
to the console
I am not sure what's going on here. Following dependencies are
bundled with the FAT jar
"com.typesafe.scala-logging" %%"scala-logging" % scalaLoggingVersion, "org.slf4j" %"slf4j-api" %"1.7.33", "org.apache.logging.log4j" %"log4j-slf4j-impl" %"2.17.0",
"org.apache.logging.log4j" %"log4j-core" %"2.17.0", "org.apache.logging.log4j" %%"log4j-api-scala" %"12.0", "io.sentry" %"sentry-log4j2" %"5.6.0",
Confused about what is going on here, possible this might not be
Flink related matter but I am not sure..any tips on how to best
debug this would be much appreciated.
--
*Thanks,*
*Jonas*
--
*Med Vänliga Hälsningar*
/Jonas Eyob/
