> You said, "there are no job logs generated on the server that is running the job.". that was quoting your previous sentence and answer your question..
> If I were to run a job and I wanted to tail the job log as it was running, where would I find that log? 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and look for logs in local dirs specified by yarn.nodemanager.log-dirs. Or 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is to aggregate those NM local logs and upload them to HDFS once application is finished.Then you can use yarn logs command or simply go the history UI to see the logs. You can find good explanation from http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/ Thanks. On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop <sfhad...@gmail.com> wrote: > Thanks for that info Jian. > > You said, "there are no job logs generated on the server that is running > the job.". So am I correct in assuming the logs will be in the dir > specified by yarn.nodemanager.log-dirs on the datanodes? > > I am quite confused as to where the logs for each specific part of the > ecosystem reside. > > If I were to run a job and I wanted to tail the job log as it was running, > where would I find that log? > > Thanks for your help. > > > On Mon, Mar 3, 2014 at 11:46 AM, Jian He <j...@hortonworks.com> wrote: > >> Note that node manager will not keep the finished applications and only >> show running apps, so the UI won't show the finished apps. >> Conversely, job history server UI will only show the finished apps but >> not the running apps. >> >> bq. there are no job logs generated on the server that is running the >> job. >> by default, the local logs will be deleted after job finished. you can >> config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of >> the logs. >> >> Jian >> >> >> On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop <sfhad...@gmail.com> wrote: >> >>> Hadoop 2.2.0 >>> CentOS 6.4 >>> Viewing UI in various browsers. >>> >>> I am having a problem where no information is visible in my Job History >>> UI. I run test jobs, they complete without error, but no information ever >>> populates the nodemanager or jobhistory server UI. >>> >>> Also, there are no job logs generated on the server that is running the >>> job. >>> >>> I have the following settings configured: >>> yarn.nodemanager.local-dirs >>> yarn.nodemanager.log-dirs >>> yarn.log.server.url >>> >>> ...plus the basic yarn log dir. I get output in regards to the daemons >>> but very little in regards to the job. All I get that refers to the >>> jobhistory server is the following (so it appears to be functioning >>> properly): >>> >>> 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty >>> bound to port 19888 >>> 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26 >>> 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract >>> jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory >>> to /tmp/Jetty_server_19888_jobhistory____v7gnnv/webapp >>> 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started >>> SelectChannelConnector@server:19888 >>> 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web >>> app /jobhistory started at 19888 >>> 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps: >>> Registered webapp guice modules >>> >>> I have a feeling this is a misconfiguration but I cannot figure out what >>> setting is missing or wrong. >>> >>> Other than not being able to see any of the jobs in the UIs, everything >>> appears to be working correctly so this is quite confusing. >>> >>> Any help is appreciated. >>> >> >> >> CONFIDENTIALITY NOTICE >> NOTICE: This message is intended for the use of the individual or entity >> to which it is addressed and may contain information that is confidential, >> privileged and exempt from disclosure under applicable law. If the reader >> of this message is not the intended recipient, you are hereby notified that >> any printing, copying, dissemination, distribution, disclosure or >> forwarding of this communication is strictly prohibited. If you have >> received this communication in error, please contact the sender immediately >> and delete it from your system. Thank You. > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
