Hi! I have a question about logs and have not seen the answer through internet. I have a spark submit process and I configure a custom log configuration to it using the next params: --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=customlog4j.properties" --driver-java-options '-Dlog4j.configuration=customlog4j.properties' So far is working great, but now I have to add kerberos params like this: --conf spark.hadoop.fs.hdfs.impl.disable.cache=true --keytab <keytab path> --principal <user@domain> When I add those params (and in particular, --principal) my custom log file starts being ignored and the default spark log file is being used insted. Change the default spark configuration file or path is a little difficult for us since we can't access that cluster. Is there any way to keep using my custom log configuration the same way? I am doing something wrong?
Thanks for the help!
