Re: Custom log4j.properties on AWS EMR
Thanks for your response Jonathan. Yes, this works. I also added another way of achieving this to the Stackoverflow post. Thanks for the help. On Tue, Feb 28, 2017 at 11:58 PM, Jonathan Kellywrote: > Prithish, > > I saw you posted this on SO, so I responded there just now. See > http://stackoverflow.com/questions/42452622/custom- > log4j-properties-on-aws-emr/42516161#42516161 > > In short, an hdfs:// path can't be used to configure log4j because log4j > knows nothing about hdfs. Instead, since you are using EMR, you should use > the Configuration API when creating your cluster to configure the > spark-log4j configuration classification. See http://docs.aws.amazon. > com/emr/latest/ReleaseGuide/emr-configure-apps.html for more info. > > ~ Jonathan > > On Sun, Feb 26, 2017 at 8:17 PM Prithish wrote: > >> Steve, I tried that, but didn't work. Any other ideas? >> >> On Mon, Feb 27, 2017 at 1:42 AM, Steve Loughran >> wrote: >> >> try giving a resource of a file in the JAR, e.g add a file >> "log4j-debugging.properties into the jar, and give a config option of >> -Dlog4j.configuration=/log4j-debugging.properties (maybe also try >> without the "/") >> >> >> On 26 Feb 2017, at 16:31, Prithish wrote: >> >> Hoping someone can answer this. >> >> I am unable to override and use a Custom log4j.properties on Amazon EMR. >> I am running Spark on EMR (Yarn) and have tried all the below combinations >> in the Spark-Submit to try and use the custom log4j. >> >> In Client mode >> --driver-java-options "-Dlog4j.configuration=hdfs:// >> host:port/user/hadoop/log4j.properties" >> >> In Cluster mode >> --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host: >> port/user/hadoop/log4j.properties" >> >> I have also tried picking from local filesystem using file: instead >> of hdfs. None of this seem to work. However, I can get this working when >> running on my local Yarn setup. >> >> Any ideas? >> >> I have also posted on Stackoverflow (link below) >> http://stackoverflow.com/questions/42452622/custom- >> log4j-properties-on-aws-emr >> >> >> >>
Re: Custom log4j.properties on AWS EMR
Prithish, I saw you posted this on SO, so I responded there just now. See http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr/42516161#42516161 In short, an hdfs:// path can't be used to configure log4j because log4j knows nothing about hdfs. Instead, since you are using EMR, you should use the Configuration API when creating your cluster to configure the spark-log4j configuration classification. See http://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-configure-apps.html for more info. ~ Jonathan On Sun, Feb 26, 2017 at 8:17 PM Prithishwrote: > Steve, I tried that, but didn't work. Any other ideas? > > On Mon, Feb 27, 2017 at 1:42 AM, Steve Loughran > wrote: > > try giving a resource of a file in the JAR, e.g add a file > "log4j-debugging.properties into the jar, and give a config option of > -Dlog4j.configuration=/log4j-debugging.properties (maybe also try without > the "/") > > > On 26 Feb 2017, at 16:31, Prithish wrote: > > Hoping someone can answer this. > > I am unable to override and use a Custom log4j.properties on Amazon EMR. I > am running Spark on EMR (Yarn) and have tried all the below combinations in > the Spark-Submit to try and use the custom log4j. > > In Client mode > --driver-java-options "-Dlog4j.configuration= > hdfs://host:port/user/hadoop/log4j.properties" > > In Cluster mode > --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration= > hdfs://host:port/user/hadoop/log4j.properties" > > I have also tried picking from local filesystem using file: instead > of hdfs. None of this seem to work. However, I can get this working when > running on my local Yarn setup. > > Any ideas? > > I have also posted on Stackoverflow (link below) > > http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr > > > >
Re: Custom log4j.properties on AWS EMR
Steve, I tried that, but didn't work. Any other ideas? On Mon, Feb 27, 2017 at 1:42 AM, Steve Loughranwrote: > try giving a resource of a file in the JAR, e.g add a file > "log4j-debugging.properties into the jar, and give a config option of > -Dlog4j.configuration=/log4j-debugging.properties (maybe also try > without the "/") > > > On 26 Feb 2017, at 16:31, Prithish wrote: > > Hoping someone can answer this. > > I am unable to override and use a Custom log4j.properties on Amazon EMR. I > am running Spark on EMR (Yarn) and have tried all the below combinations in > the Spark-Submit to try and use the custom log4j. > > In Client mode > --driver-java-options "-Dlog4j.configuration=hdfs:// > host:port/user/hadoop/log4j.properties" > > In Cluster mode > --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host: > port/user/hadoop/log4j.properties" > > I have also tried picking from local filesystem using file: instead > of hdfs. None of this seem to work. However, I can get this working when > running on my local Yarn setup. > > Any ideas? > > I have also posted on Stackoverflow (link below) > http://stackoverflow.com/questions/42452622/custom- > log4j-properties-on-aws-emr > > >
Re: Custom log4j.properties on AWS EMR
try giving a resource of a file in the JAR, e.g add a file "log4j-debugging.properties into the jar, and give a config option of -Dlog4j.configuration=/log4j-debugging.properties (maybe also try without the "/") On 26 Feb 2017, at 16:31, Prithish> wrote: Hoping someone can answer this. I am unable to override and use a Custom log4j.properties on Amazon EMR. I am running Spark on EMR (Yarn) and have tried all the below combinations in the Spark-Submit to try and use the custom log4j. In Client mode --driver-java-options "-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties" In Cluster mode --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties" I have also tried picking from local filesystem using file: instead of hdfs. None of this seem to work. However, I can get this working when running on my local Yarn setup. Any ideas? I have also posted on Stackoverflow (link below) http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr
Custom log4j.properties on AWS EMR
Hoping someone can answer this. I am unable to override and use a Custom log4j.properties on Amazon EMR. I am running Spark on EMR (Yarn) and have tried all the below combinations in the Spark-Submit to try and use the custom log4j. In Client mode --driver-java-options "-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties" In Cluster mode --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=hdfs://host:port/user/hadoop/log4j.properties" I have also tried picking from local filesystem using file: instead of hdfs. None of this seem to work. However, I can get this working when running on my local Yarn setup. Any ideas? I have also posted on Stackoverflow (link below) http://stackoverflow.com/questions/42452622/custom-log4j-properties-on-aws-emr