Thanks for your response Jonathan. Yes, this works. I also added another way of achieving this to the Stackoverflow post. Thanks for the help.
On Tue, Feb 28, 2017 at 11:58 PM, Jonathan Kelly <jonathaka...@gmail.com> wrote: > Prithish, > > I saw you posted this on SO, so I responded there just now. See > http://stackoverflow.com/questions/42452622/custom- > log4j-properties-on-aws-emr/42516161#42516161 > > In short, an hdfs:// path can't be used to configure log4j because log4j > knows nothing about hdfs. Instead, since you are using EMR, you should use > the Configuration API when creating your cluster to configure the > spark-log4j configuration classification. See http://docs.aws.amazon. > com/emr/latest/ReleaseGuide/emr-configure-apps.html for more info. > > ~ Jonathan > > On Sun, Feb 26, 2017 at 8:17 PM Prithish <prith...@gmail.com> wrote: > >> Steve, I tried that, but didn't work. Any other ideas? >> >> On Mon, Feb 27, 2017 at 1:42 AM, Steve Loughran <ste...@hortonworks.com> >> wrote: >> >> try giving a resource of a file in the JAR, e.g add a file >> "log4j-debugging.properties into the jar, and give a config option of >> -Dlog4j.configuration=/log4j-debugging.properties (maybe also try >> without the "/") >> >> >> On 26 Feb 2017, at 16:31, Prithish <prith...@gmail.com> wrote: >> >> Hoping someone can answer this. >> >> I am unable to override and use a Custom log4j.properties on Amazon EMR. >> I am running Spark on EMR (Yarn) and have tried all the below combinations >> in the Spark-Submit to try and use the custom log4j. >> >> In Client mode >> --driver-java-options "-Dlog4j.configuration=hdfs:// >> host:port/user/hadoop/log4j.properties" >> >> In Cluster mode >> --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host: >> port/user/hadoop/log4j.properties" >> >> I have also tried picking from local filesystem using file://// instead >> of hdfs. None of this seem to work. However, I can get this working when >> running on my local Yarn setup. >> >> Any ideas? >> >> I have also posted on Stackoverflow (link below) >> http://stackoverflow.com/questions/42452622/custom- >> log4j-properties-on-aws-emr >> >> >> >>
