Re: Not information in Job History UI
That explains a lot. Thanks for the information. I appreciate your help. On Mon, Mar 3, 2014 at 7:47 PM, Jian He j...@hortonworks.com wrote: You said, there are no job logs generated on the server that is running the job.. that was quoting your previous sentence and answer your question.. If I were to run a job and I wanted to tail the job log as it was running, where would I find that log? 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and look for logs in local dirs specified by yarn.nodemanager.log-dirs. Or 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is to aggregate those NM local logs and upload them to HDFS once application is finished.Then you can use yarn logs command or simply go the history UI to see the logs. You can find good explanation from http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/ Thanks. On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop sfhad...@gmail.com wrote: Thanks for that info Jian. You said, there are no job logs generated on the server that is running the job.. So am I correct in assuming the logs will be in the dir specified by yarn.nodemanager.log-dirs on the datanodes? I am quite confused as to where the logs for each specific part of the ecosystem reside. If I were to run a job and I wanted to tail the job log as it was running, where would I find that log? Thanks for your help. On Mon, Mar 3, 2014 at 11:46 AM, Jian He j...@hortonworks.com wrote: Note that node manager will not keep the finished applications and only show running apps, so the UI won't show the finished apps. Conversely, job history server UI will only show the finished apps but not the running apps. bq. there are no job logs generated on the server that is running the job. by default, the local logs will be deleted after job finished. you can config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of the logs. Jian On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop sfhad...@gmail.com wrote: Hadoop 2.2.0 CentOS 6.4 Viewing UI in various browsers. I am having a problem where no information is visible in my Job History UI. I run test jobs, they complete without error, but no information ever populates the nodemanager or jobhistory server UI. Also, there are no job logs generated on the server that is running the job. I have the following settings configured: yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.log.server.url ...plus the basic yarn log dir. I get output in regards to the daemons but very little in regards to the job. All I get that refers to the jobhistory server is the following (so it appears to be functioning properly): 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 19888 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started SelectChannelConnector@server:19888 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /jobhistory started at 19888 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules I have a feeling this is a misconfiguration but I cannot figure out what setting is missing or wrong. Other than not being able to see any of the jobs in the UIs, everything appears to be working correctly so this is quite confusing. Any help is appreciated. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Not information in Job History UI
Hadoop 2.2.0 CentOS 6.4 Viewing UI in various browsers. I am having a problem where no information is visible in my Job History UI. I run test jobs, they complete without error, but no information ever populates the nodemanager or jobhistory server UI. Also, there are no job logs generated on the server that is running the job. I have the following settings configured: yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.log.server.url ...plus the basic yarn log dir. I get output in regards to the daemons but very little in regards to the job. All I get that refers to the jobhistory server is the following (so it appears to be functioning properly): 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 19888 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started SelectChannelConnector@server:19888 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /jobhistory started at 19888 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules I have a feeling this is a misconfiguration but I cannot figure out what setting is missing or wrong. Other than not being able to see any of the jobs in the UIs, everything appears to be working correctly so this is quite confusing. Any help is appreciated.
Re: Not information in Job History UI
Note that node manager will not keep the finished applications and only show running apps, so the UI won't show the finished apps. Conversely, job history server UI will only show the finished apps but not the running apps. bq. there are no job logs generated on the server that is running the job. by default, the local logs will be deleted after job finished. you can config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of the logs. Jian On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop sfhad...@gmail.com wrote: Hadoop 2.2.0 CentOS 6.4 Viewing UI in various browsers. I am having a problem where no information is visible in my Job History UI. I run test jobs, they complete without error, but no information ever populates the nodemanager or jobhistory server UI. Also, there are no job logs generated on the server that is running the job. I have the following settings configured: yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.log.server.url ...plus the basic yarn log dir. I get output in regards to the daemons but very little in regards to the job. All I get that refers to the jobhistory server is the following (so it appears to be functioning properly): 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 19888 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started SelectChannelConnector@server:19888 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /jobhistory started at 19888 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules I have a feeling this is a misconfiguration but I cannot figure out what setting is missing or wrong. Other than not being able to see any of the jobs in the UIs, everything appears to be working correctly so this is quite confusing. Any help is appreciated. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Not information in Job History UI
Thanks for that info Jian. You said, there are no job logs generated on the server that is running the job.. So am I correct in assuming the logs will be in the dir specified by yarn.nodemanager.log-dirs on the datanodes? I am quite confused as to where the logs for each specific part of the ecosystem reside. If I were to run a job and I wanted to tail the job log as it was running, where would I find that log? Thanks for your help. On Mon, Mar 3, 2014 at 11:46 AM, Jian He j...@hortonworks.com wrote: Note that node manager will not keep the finished applications and only show running apps, so the UI won't show the finished apps. Conversely, job history server UI will only show the finished apps but not the running apps. bq. there are no job logs generated on the server that is running the job. by default, the local logs will be deleted after job finished. you can config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of the logs. Jian On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop sfhad...@gmail.com wrote: Hadoop 2.2.0 CentOS 6.4 Viewing UI in various browsers. I am having a problem where no information is visible in my Job History UI. I run test jobs, they complete without error, but no information ever populates the nodemanager or jobhistory server UI. Also, there are no job logs generated on the server that is running the job. I have the following settings configured: yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.log.server.url ...plus the basic yarn log dir. I get output in regards to the daemons but very little in regards to the job. All I get that refers to the jobhistory server is the following (so it appears to be functioning properly): 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 19888 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started SelectChannelConnector@server:19888 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /jobhistory started at 19888 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules I have a feeling this is a misconfiguration but I cannot figure out what setting is missing or wrong. Other than not being able to see any of the jobs in the UIs, everything appears to be working correctly so this is quite confusing. Any help is appreciated. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Not information in Job History UI
You said, there are no job logs generated on the server that is running the job.. that was quoting your previous sentence and answer your question.. If I were to run a job and I wanted to tail the job log as it was running, where would I find that log? 1) set yarn.nodemanager.delete.debug-delay-sec to be a larger value, and look for logs in local dirs specified by yarn.nodemanager.log-dirs. Or 2) enable log aggregation yarn.log-aggregation-enable. Log aggregation is to aggregate those NM local logs and upload them to HDFS once application is finished.Then you can use yarn logs command or simply go the history UI to see the logs. You can find good explanation from http://hortonworks.com/blog/simplifying-user-logs-management-and-access-in-yarn/ Thanks. On Mon, Mar 3, 2014 at 4:29 PM, SF Hadoop sfhad...@gmail.com wrote: Thanks for that info Jian. You said, there are no job logs generated on the server that is running the job.. So am I correct in assuming the logs will be in the dir specified by yarn.nodemanager.log-dirs on the datanodes? I am quite confused as to where the logs for each specific part of the ecosystem reside. If I were to run a job and I wanted to tail the job log as it was running, where would I find that log? Thanks for your help. On Mon, Mar 3, 2014 at 11:46 AM, Jian He j...@hortonworks.com wrote: Note that node manager will not keep the finished applications and only show running apps, so the UI won't show the finished apps. Conversely, job history server UI will only show the finished apps but not the running apps. bq. there are no job logs generated on the server that is running the job. by default, the local logs will be deleted after job finished. you can config yarn.nodemanager.delete.debug-delay-sec, to delay the deletion of the logs. Jian On Mon, Mar 3, 2014 at 10:45 AM, SF Hadoop sfhad...@gmail.com wrote: Hadoop 2.2.0 CentOS 6.4 Viewing UI in various browsers. I am having a problem where no information is visible in my Job History UI. I run test jobs, they complete without error, but no information ever populates the nodemanager or jobhistory server UI. Also, there are no job logs generated on the server that is running the job. I have the following settings configured: yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.log.server.url ...plus the basic yarn log dir. I get output in regards to the daemons but very little in regards to the job. All I get that refers to the jobhistory server is the following (so it appears to be functioning properly): 2014-02-18 11:43:06,824 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 19888 2014-02-18 11:43:06,824 INFO org.mortbay.log: jetty-6.1.26 2014-02-18 11:43:06,847 INFO org.mortbay.log: Extract jar:file:/usr/lib/hadoop-yarn/hadoop-yarn-common-2.1.0.2.0.5.0-67.jar!/webapps/jobhistory to /tmp/Jetty_server_19888_jobhistoryv7gnnv/webapp 2014-02-18 11:43:07,085 INFO org.mortbay.log: Started SelectChannelConnector@server:19888 2014-02-18 11:43:07,085 INFO org.apache.hadoop.yarn.webapp.WebApps: Web app /jobhistory started at 19888 2014-02-18 11:43:07,477 INFO org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules I have a feeling this is a misconfiguration but I cannot figure out what setting is missing or wrong. Other than not being able to see any of the jobs in the UIs, everything appears to be working correctly so this is quite confusing. Any help is appreciated. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
