Hoping someone can answer this. I am unable to override and use a Custom log4j.properties on Amazon EMR. I am running Spark on EMR (Yarn) and have tried all the below combinations in the Spark-Submit to try and use the custom log4j.
In Client mode --driver-java-options "-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties" In Cluster mode --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties" I have also tried picking from local filesystem using file://// instead of hdfs. None of this seem to work. However, I can get this working when running on my local Yarn setup. Any ideas? I have also posted on Stackoverflow (link below) http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr
