That explains a lot. Thanks for the information. I appreciate your help.
On Mon, Mar 3, 2014 at 7:47 PM, Jian He <j...@hortonworks.com> wrote: > > You said, "there are no job logs generated on the server that is > running the job.". > that was quoting your previous sentence and answer your question.. > > > If I were to run a job and I wanted to tail the job log as it was > running, where would I find that log? > 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and > look for logs in local dirs specified by yarn.nodemanager.log-dirs. > Or > 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is > to aggregate those NM local logs and upload them to HDFS once application > is finished.Then you can use yarn logs command or simply go the history UI > to see the logs. > You can find good explanation from > http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/ > > Thanks. > > > On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sfhad...@gmail.com> wrote: > >> Thanks for that info Jian. >> >> You said, "there are no job logs generated on the server that is running >> the job.". So am I correct in assuming the logs will be in the dir >> specified by yarn.nodemanager.log-dirs on the datanodes? >> >> I am quite confused as to where the logs for each specific part of the >> ecosystem reside. >> >> If I were to run a job and I wanted to tail the job log as it was >> running, where would I find that log? >> >> Thanks for your help. >> >> >> On Mon, Mar 3, 2014 at 11:46 AM, Jian He <j...@hortonworks.com> wrote: >> >>> Note that node manager will not keep the finished applications and >>> only show running apps, so the UI won't show the finished apps. >>> Conversely, job history server UI will only show the finished apps but >>> not the running apps. >>> >>> bq. there are no job logs generated on the server that is running the >>> job. >>> by default, the local logs will be deleted after job finished. you can >>> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion >>> of the logs. >>> >>> Jian >>> >>> >>> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sfhad...@gmail.com> wrote: >>> >>>> Hadoop 2.2.0 >>>> CentOS 6.4 >>>> Viewing UI in various browsers. >>>> >>>> I am having a problem where no information is visible in my Job History >>>> UI. I run test jobs, they complete without error, but no information ever >>>> populates the nodemanager or jobhistory server UI. >>>> >>>> Also, there are no job logs generated on the server that is running the >>>> job. >>>> >>>> I have the following settings configured: >>>> yarn.nodemanager.local-dirs >>>> yarn.nodemanager.log-dirs >>>> yarn.log.server.url >>>> >>>> ...plus the basic yarn log dir. I get output in regards to the daemons >>>> but very little in regards to the job. All I get that refers to the >>>> jobhistory server is the following (so it appears to be functioning >>>> properly): >>>> >>>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty >>>> bound to port 19888 >>>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26 >>>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract >>>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory >>>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp >>>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started >>>> SelectChannelConnector@server:19888 >>>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web >>>> app /jobhistory started at 19888 >>>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps: >>>> Registered webapp guice modules >>>> >>>> I have a feeling this is a misconfiguration but I cannot figure out >>>> what setting is missing or wrong. >>>> >>>> Other than not being able to see any of the jobs in the UIs, everything >>>> appears to be working correctly so this is quite confusing. >>>> >>>> Any help is appreciated. >>>> >>> >>> >>> CONFIDENTIALITY NOTICE >>> NOTICE: This message is intended for the use of the individual or entity >>> to which it is addressed and may contain information that is confidential, >>> privileged and exempt from disclosure under applicable law. If the reader >>> of this message is not the intended recipient, you are hereby notified that >>> any printing, copying, dissemination, distribution, disclosure or >>> forwarding of this communication is strictly prohibited. If you have >>> received this communication in error, please contact the sender immediately >>> and delete it from your system. Thank You. >> >> >> > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
