[
https://issues.apache.org/jira/browse/SPARK-42479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699041#comment-17699041
]
Sean R. Owen commented on SPARK-42479:
--------------------------------------
This is because you're pointing at some local path not visible on the workers.
It's quite expected
> Log4j2 doesn't works with Spark 3.3.0
> -------------------------------------
>
> Key: SPARK-42479
> URL: https://issues.apache.org/jira/browse/SPARK-42479
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, Spark Submit
> Affects Versions: 3.2.1, 3.3.0
> Reporter: Pratik Malani
> Priority: Major
>
> Hi All,
> Was trying to run spark application on the cluster mode using log4j2 and
> Spark 3.3.0.
> When I run the below spark-submit command, only one worker (out of 3) starts
> executing the job.
> {code:java}
> // code placeholder
> spark-submit --master spark://spark-master-svc:7077 \
> --conf spark.cores.max=4 \
> --conf spark.sql.broadcastTimeout=3600 \
> --conf spark.executor.cores=1 \
> --jars /opt/spark/work-dir/<main_jar_name>.jar \
> --deploy-mode cluster \
> --class <class_name> \
> --properties-file /opt/spark/conf/spark-defaults.conf \
> --conf
> spark.driver.extraJavaOptions="-Dcom.amazonaws.sdk.disableCertChecking=true
> -Dlog4j.configurationFile=file:/opt/spark/work-dir/<log4j2>.properties" \
> --conf
> spark.executor.extraJavaOptions="-Dcom.amazonaws.sdk.disableCertChecking=true
> -Dlog4j.configurationFile=file:/opt/spark/work-dir/<log4j2>.properties" \
> --files "/opt/spark/work-dir/<log4j2>.properties" \
> /opt/spark/work-dir/<jar_name>.jar
> /opt/spark/work-dir/application.properties >> /var/log/containers/hourly.log
> 2>&1 {code}
> It means, on only one worker, I can see the driver logs and the other workers
> are idle and there are no app or executors logs created on other workers.
> Below is the log4j2.properties file being used.
> {code:java}
> // code placeholder
> rootLogger.level = INFO
> rootLogger.appenderRef.rolling.ref = loggerId
> appender.rolling.type = RollingFile
> appender.rolling.name = loggerId
> appender.rolling.fileName=/var/log/containers/hourly.log
> appender.rolling.filePattern=hourly-.%d{yyyyMMdd}.log.gz
> appender.rolling.layout.type = PatternLayout
> appender.rolling.layout.pattern=%d [%t] %-5p (%F:%L) - %m%n
> appender.rolling.policies.type = Policies
> appender.rolling.policies.size.type = TimeBasedTriggeringPolicy
> appender.rolling.strategy.type = DefaultRolloverStrategy
> appender.rolling.strategy.max = 5
> logger.spark.name = org.apache.spark
> logger.spark.level = WARN
> logger.spark.additivity = false
> logger.spark.repl.SparkIMain$exprTyper.level = INFO
> logger.spark.repl.SparkILoop$SparkILoopInterpreter.level = INFO
> # Settings to quiet third party logs that are too verbose
> logger.jetty.name = org.eclipse.jetty
> logger.jetty.level = WARN
> logger.jetty.util.component.AbstractLifeCycle.level = ERROR
> logger.parquet.name = org.apache.parquet
> logger.parquet.level = ERROR
> logger.kafka.name = org.apache.kafka
> logger.kafka.level = WARN
> logger.kafka.clients.consumer.internals.Fetcher.level=WARN {code}
> All log4j2 jars are included in the Spark home classpath under the jars
> directory.
> * log4j-1.2-api-2.17.2.jar
> * log4j-api-2.17.2.jar
> * log4j-api-scala_2.12-12.0.jar
> * log4j-core-2.17.2.jar
> * log4j-slf4j-impl-2.17.2.jar
> Can you please check and let me know whether I need to add or update anything
> to start the job in cluster mode supporting log4j2
> Note : Things work fine with log4j1
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
