Hi Sabarish Thanks for your help, i was able to get the logs from archive, is there a way i can adjust archival policy, say i want to persist the logs of last 2 jobs on resource manage and archive others on the file system.
On Mon, Feb 22, 2016 at 12:25 PM Sabarish Sasidharan < sabarish.sasidha...@manthan.com> wrote: > Your logs are getting archived in your logs bucket in S3. > > > http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-debugging.html > > Regards > Sab > > On Mon, Feb 22, 2016 at 12:14 PM, HARSH TAKKAR <takkarha...@gmail.com> > wrote: > >> Hi >> >> In am using an EMR cluster for running my spark jobs, but after the job >> finishes logs disappear, >> >> I have added a log4j.properties in my jar, but all the logs still >> redirects to EMR resource manager which vanishes after jobs completes, is >> there a way i could redirect the logs to a location in file syatem, I am >> working on price points and its very critical for me to maintain logs. >> >> Just to add i get following error when my application starts. >> >> java.io.FileNotFoundException: /etc/spark/conf/log4j.properties (No such >> file or directory) >> at java.io.FileInputStream.open(Native Method) >> at java.io.FileInputStream.<init>(FileInputStream.java:146) >> at java.io.FileInputStream.<init>(FileInputStream.java:101) >> at >> sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) >> at >> sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) >> at >> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557) >> at >> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) >> at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) >> at org.apache.spark.Logging$class.initializeLogging(Logging.scala:122) >> at >> org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:107) >> at org.apache.spark.Logging$class.log(Logging.scala:51) >> at >> org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:607) >> at >> org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:621) >> at >> org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) >> >> >> > > > -- > > Architect - Big Data > Ph: +91 99805 99458 > > Manthan Systems | *Company of the year - Analytics (2014 Frost and > Sullivan India ICT)* > +++ >
