Hi Sanket,
Driver and executor logs are written to stdout by default, it can be
configured using SPARK_HOME/conf/log4j.properties file. The file including
the entire SPARK_HOME/conf is auto propogateded to all driver and executor
container and mounted as volume.
Thanks
On Mon, 9 Oct, 2023, 5:37
Hi All,
We are trying to send the spark logs using fluent-bit. We validated that
fluent-bit is able to move logs of all other pods except the driver/executor
pods.
It would be great if someone can guide us where should I look for spark logs in
Spark on Kubernetes with client/cluster mode
