[ http://issues.apache.org/jira/browse/HADOOP-489?page=comments#action_12431388 ] Michel Tourn commented on HADOOP-489: -------------------------------------
>>About (c) and providing a http url to the log file which contains >>the log for this task. on the client screen >>Would that solve your second requirement Michel? Yes and no. With access to such a URL I would not ask the uer to go browse it. But I could have HadoopStreaming retrieve the content then stream it to the user like "tail -f". >polling for the last N lines would not be easy Use HTTP Keep-Alive to do this. I.e. just keep the GET socket open and wait for this "slow webserver" to send you more lines as they get available. Now the scalability story: =================== when you have access to all this real-time log information, a "smart" client will not blindly echo everything from every Task. It will just show you what is wrong "by example" Assuming we have a way to do some classification of the Tasks into categories: "failed", "still going" and "successfully finished": o pick the first one in each category o by default, ignore log URLs for the other logs o do a merged "tail -f" with a banned on the client. "tail -f " is implemented as HTTP Keep-Alive as described. 3 HTTP sockets at most. Yes, all this assumes some smart-client support. But we can easily package the code as reusable logic to be used from any user-written JobSubmitter main class. One of these clients is HadoopStreaming. If a client application does not use this support code: it is just ignoring the log URLs made available, but otherwise works as usual. > Seperating user logs from system logs in map reduce > --------------------------------------------------- > > Key: HADOOP-489 > URL: http://issues.apache.org/jira/browse/HADOOP-489 > Project: Hadoop > Issue Type: Improvement > Components: mapred > Reporter: Mahadev konar > Assigned To: Mahadev konar > Priority: Minor > > Currently the user logs are a part of system logs in mapreduce. Anything > logged by the user is logged into the tasktracker log files. This create two > issues- > 1) The system log files get cluttered with user output. If the user outputs a > large amount of logs, the system logs need to be cleaned up pretty often. > 2) For the user, it is difficult to get to each of the machines and look for > the logs his/her job might have generated. > I am proposing three solutions to the problem. All of them have issues with > it - > Solution 1. > Output the user logs on the user screen as part of the job submission > process. > Merits- > This will prevent users from printing large amount of logs and the user can > get runtime feedback on what is wrong with his/her job. > Issues - > This proposal will use the framework bandwidth while running jobs for the > user. The user logs will need to pass from the tasks to the tasktrackers, > from the tasktrackers to the jobtrackers and then from the jobtrackers to the > jobclient using a lot of framework bandwidth if the user is printing out too > much data. > Solution 2. > Output the user logs onto a dfs directory and then concatenate these files. > Each task can create a file for the output in the log direcotyr for a given > user and jobid. > Issues - > This will create a huge amount of small files in DFS which later can be > concatenated into a single file. Also there is this issue that who would > concatenate these files into a single file? This could be done by the > framework (jobtracker) as part of the cleanup for the jobs - might stress the > jobtracker. > > Solution 3. > Put the user logs into a seperate user log file in the log directory on each > tasktrackers. We can provide some tools to query these local log files. We > could have commands like for jobid j and for taskid t get me the user log > output. These tools could run as a seperate map reduce program with each map > grepping the user log files and a single recude aggregating these logs in to > a single dfs file. > Issues- > This does sound like more work for the user. Also, the output might not be > complete since a tasktracker might have went down after it ran the job. > Any thoughts? -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
