Thanks for that info Jian. You said, "there are no job logs generated on the server that is running the job.". So am I correct in assuming the logs will be in the dir specified by yarn.nodemanager.log-dirs on the datanodes?
I am quite confused as to where the logs for each specific part of the ecosystem reside. If I were to run a job and I wanted to tail the job log as it was running, where would I find that log? Thanks for your help. On Mon, Mar 3, 2014 at 11:46 AM, Jian He <j...@hortonworks.com> wrote: > Note that node manager will not keep the finished applications and only > show running apps, so the UI won't show the finished apps. > Conversely, job history server UI will only show the finished apps but > not the running apps. > > bq. there are no job logs generated on the server that is running the job. > by default, the local logs will be deleted after job finished. you can > config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of > the logs. > > Jian > > > On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sfhad...@gmail.com> wrote: > >> Hadoop 2.2.0 >> CentOS 6.4 >> Viewing UI in various browsers. >> >> I am having a problem where no information is visible in my Job History >> UI. I run test jobs, they complete without error, but no information ever >> populates the nodemanager or jobhistory server UI. >> >> Also, there are no job logs generated on the server that is running the >> job. >> >> I have the following settings configured: >> yarn.nodemanager.local-dirs >> yarn.nodemanager.log-dirs >> yarn.log.server.url >> >> ...plus the basic yarn log dir. I get output in regards to the daemons >> but very little in regards to the job. All I get that refers to the >> jobhistory server is the following (so it appears to be functioning >> properly): >> >> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty >> bound to port 19888 >> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26 >> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract >> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory >> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp >> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started >> SelectChannelConnector@server:19888 >> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web >> app /jobhistory started at 19888 >> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps: >> Registered webapp guice modules >> >> I have a feeling this is a misconfiguration but I cannot figure out what >> setting is missing or wrong. >> >> Other than not being able to see any of the jobs in the UIs, everything >> appears to be working correctly so this is quite confusing. >> >> Any help is appreciated. >> > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
