hailin0 opened a new issue, #2725: URL: https://github.com/apache/incubator-seatunnel/issues/2725
### Search before asking - [X] I had searched in the [feature](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description # ST-Engine integration log ecosystem proposal ## Backgroud & Motivation Currently ST-Engine has not integrated log framework management engine and connector logs. I think it is possible to integrate [slf4j](https://www.slf4j.org/) + [log4j2](https://logging.apache.org/log4j/2.x/) to manage logs output ## Overall Design ### Java log ecosystem & bridger #### ecosystem <img src="https://www.slf4j.org/images/concrete-bindings.png"/> #### bridger <img src="https://www.slf4j.org/images/legacy.png"/> ### Integration hazelcast log ST-Engine manage the cluster nodes using [hazelcast](https://hazelcast.com/). hazelcast supports integration with slf4j, log4j1.x, log4j2.x, jdk-logging, and provides http endpoint to dynamically manage logger level. We chose to integrate log4j2.x and get the ability to [dynamically manage loggger level](https://docs.hazelcast.com/hazelcast/5.1/maintain-cluster/logging#dynamically-changing-log-levels). - Configure [`hazelcast.logging.type: log4j2`](https://docs.hazelcast.com/hazelcast/5.1/maintain-cluster/logging#logging-for-client-and-embedded-mode) into config files - Add `log4j-api`、`log4j-core`、`log4j-slf4j-impl` package & config file into classpath ### Integration ST-Engine dependencies package log If ST-Engine dependencie packages use other log systems to output logs, e.g. slf4j、log4j1、 log4j2、logback、commons-logging. We can use [log bridge](https://www.slf4j.org/legacy.html) to integrate other logging systems： - Add `slf4j-api` package into classpath - Add `log4j-slf4j-impl` package into classpath - Exclude `commons-logging`、`slf4j-jcl` from classpath, Add `jcl-over-slf4j`(commons-logging bridge) into classpath - Exclude `logback-classic`、`logback-core` from classpath - Exclude `log4j`、`slf4j-log4j12`、`slf4j-reload4j` from classpath, Add `log4j-over-slf4j`(log4j1 bridge) into classpath - Exclude ` slf4j-jdk14` from classpath, Add `jul-to-slf4j`(jdk-logging bridge) into classpath - optional, [low performance](https://www.slf4j.org/legacy.html#jul-to-slf4j) ST-Engine `ChildFirstClassLoader` should not load packages contains with `slf4j` or `log4j` or `commons-logging` keyword. [reference link](https://github.com/apache/flink/blob/a91501f37262b461f7c645defe805686f7e274dd/flink-core/src/main/java/org/apache/flink/configuration/CoreOptions.java#L106) ### Integration connector log Logging system used by each engine： - [flink use slf4j + log4j2](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/advanced/logging/) by default - [spark use slf4j + log4j1](https://spark.apache.org/docs/2.4.6/configuration.html#configuring-logging) by default - ST-Engine use slf4j + log4j2 by default If connector and their dependencie packages use other log systems to output logs, e.g. slf4j、log4j1、 log4j2、logback、commons-logging. Exclude log system packages from connector shade package： - Exclude `commons-logging`、`slf4j-jcl` from connector shade package - Exclude `logback-classic`、`logback-core` from connector shade package - Exclude `log4j`、`slf4j-log4j12` from connector shade package - Exclude `log4j-api`、`log4j-core`、`log4j-slf4j-impl` from connector shade package - Exclude `slf4j` from connector shade package Add [log bridge](https://www.slf4j.org/legacy.html) to engine server libs. ### Link the logs of same job in st-engine We use [slf4j mdc](https://www.slf4j.org/manual.html#mdc) to link jobs and logs. Define the slf4j mdc key values: - ST-JID: indicates job ID - ST-TID: indicates task ID - ST-PID: indicates job pipeline ID Inject metadata(ST-JID, ST-TID, ST-PID) to slf4j mdc at job start and cleanup slf4j mdc after job ends Configure the output of mdc key-values for log pattern: ```text [%X{ST-JID, ST-TID, ST-PID}] [%p] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c:%L - %m%n ``` ## Rejected proposal ### Job output single log file The connector logs cannot be output to same log file as st-engine job logs, connector and dependency packages logger is declared by user custom. ### Usage Scenario _No response_ ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
![https://www.slf4j.org/images/concrete-bindings.png"/](https://www.slf4j.org/images/concrete-bindings.png"/){kind=link}
![https://www.slf4j.org/images/legacy.png"/](https://www.slf4j.org/images/legacy.png"/){kind=link}