[jira] Commented: (MAPREDUCE-1713) Utilities for system tests specific.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865415#action_12865415 ] Konstantin Boudnik commented on MAPREDUCE-1713: --- bq. AbstractDaemonCluster.java should import MRCluster class the one above should clearly said "should NOT import". > Utilities for system tests specific. > > > Key: MAPREDUCE-1713 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1713 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1713-ydist-security.patch, 1713-ydist-security.patch, > systemtestutils_MR1713.patch, utilsforsystemtest_1713.patch > > > 1. A method for restarting the daemon with new configuration. > public static void restartCluster(Hashtable props, String > confFile) throws Exception; > 2. A method for resetting the daemon with default configuration. > public void resetCluster() throws Exception; > 3. A method for waiting until daemon to stop. > public void waitForClusterToStop() throws Exception; > 4. A method for waiting until daemon to start. > public void waitForClusterToStart() throws Exception; > 5. A method for checking the job whether it has started or not. > public boolean isJobStarted(JobID id) throws IOException; > 6. A method for checking the task whether it has started or not. > public boolean isTaskStarted(TaskInfo taskInfo) throws IOException; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1713) Utilities for system tests specific.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865387#action_12865387
]
Konstantin Boudnik commented on MAPREDUCE-1713:
---
- Looking almost good. However, {{AbstractDaemonCluster.java}} should import
MRCluster class. It's a wrong dependency.
- Is it possible that {{chkDaemonStart.get(ind)}} returns null and then calling
{{join()}} on it will produce NPE?
- why {{clusterManager}} is used in some places and isn't in others (on more
than one occasion)?
{noformat}
+ clusterManager.stop(newConfDir);
+} else {
+ stop();
{noformat}
- reduce the level of logging for daemon thread helper classes. E.g.
{noformat}
+ LOG.info("Waiting for Daemon " + daemon.getHostName()
+ + " to stop.");
{noformat}
will be clogging the log files with pretty much useless information. Use debug
instead, Another info message might be added before {{while}} loop; warning
message needs to be added if wait was unsuccessful.
Same makes sense for daemon start threads, I guess.
- shorter form of {{for}} loop can be used perhaps for
{{waitForClusterToStop()}} and {{waitForClusterToStart()}}. Instead of
{noformat}
+for (int ind = 0; ind < chkDaemonStart.size(); ind++){
+ try {
+chkDaemonStart.get(ind).join();
+ }catch(InterruptedException intExp) {
+ }
+}
{noformat}
{noformat}
+for (AbstractDaemonClient daemonThread : chkDaemonStart){
+ try {
+daemonThread.join();
+ }catch(InterruptedException intExp) {
+ }
+}
{noformat}
- missing descriptions in
{{+ * @throws IOException}}
- incorrect JavaDocs
{noformat}
+ * @param cluster daemon cluster instance
+ * @param props config attributes
+ * @throws IOException
+ */
+ public void restartClusterWithNewConfig(Hashtable props,
+ String configFile) throws IOException {
{noformat}
- more of the same
{noformat}
+ * @param taskInfo task information
+ * @param wovenClient jobtracker protocol
+ * @return true if task is running.
+ * @throws IOException
+ */
+ public boolean isTaskStarted(TaskInfo taskInfo) throws IOException {
{noformat}
- and more
{noformat}
+ /**
+ * It uses to check whether job is started or not.
+ * @param jInfo job information
+ * @param wovenClient jobtracker protocol
+ * @return true if job is running.
+ * @throws IOException
+ */
+ public boolean isJobStarted(JobID id) throws IOException {
{noformat}
> Utilities for system tests specific.
>
>
> Key: MAPREDUCE-1713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1713
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
>Reporter: Vinay Kumar Thota
>Assignee: Vinay Kumar Thota
> Attachments: 1713-ydist-security.patch, 1713-ydist-security.patch,
> systemtestutils_MR1713.patch, utilsforsystemtest_1713.patch
>
>
> 1. A method for restarting the daemon with new configuration.
> public static void restartCluster(Hashtable props, String
> confFile) throws Exception;
> 2. A method for resetting the daemon with default configuration.
> public void resetCluster() throws Exception;
> 3. A method for waiting until daemon to stop.
> public void waitForClusterToStop() throws Exception;
> 4. A method for waiting until daemon to start.
> public void waitForClusterToStart() throws Exception;
> 5. A method for checking the job whether it has started or not.
> public boolean isJobStarted(JobID id) throws IOException;
> 6. A method for checking the task whether it has started or not.
> public boolean isTaskStarted(TaskInfo taskInfo) throws IOException;
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-220) Collecting cpu and memory usage for MapReduce tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865353#action_12865353 ] Scott Chen commented on MAPREDUCE-220: -- Had some offline discussion with Dhruba. We can make COUNTER_UPDATE_INTERVAL configurable here (it is currently hard coded to 1 min). Then we can reconfigure it to submit newer information to the scheduler in our case. This should solve our problem. > Collecting cpu and memory usage for MapReduce tasks > --- > > Key: MAPREDUCE-220 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-220 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: task, tasktracker >Reporter: Hong Tang >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-220-v1.txt, MAPREDUCE-220.txt > > > It would be nice for TaskTracker to collect cpu and memory usage for > individual Map or Reduce tasks over time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1481) Streaming should swallow IOExceptions when closing clientOut
[ https://issues.apache.org/jira/browse/MAPREDUCE-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-1481: --- Attachment: TestStreamingDontConsumeInput.java quickly built a testcase for this, though no patch yet - the code is somewhat spaghetti. > Streaming should swallow IOExceptions when closing clientOut > > > Key: MAPREDUCE-1481 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1481 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: TestStreamingDontConsumeInput.java > > > in PipeMapRed.mapRedFinished, streaming flushes and closes clientOut_, the > handle to the subprocess's stdin. If the subprocess has already exited or > closed its stdin, this will generate a Broken Pipe IOException. This causes > us to skip waitOutputThreads, which is incorrect, since the subprocess may > have data still written from stdout that needs to be read. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1733) Authentication between pipes processes and java counterparts.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-1733: Attachment: MR-1733-y20.2.patch > Authentication between pipes processes and java counterparts. > - > > Key: MAPREDUCE-1733 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1733 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Attachments: MR-1733-y20.1.patch, MR-1733-y20.2.patch > > > The connection between a pipe process and its parent java process should be > authenticated. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1683) Remove JNI calls from ClusterStatus cstr
[
https://issues.apache.org/jira/browse/MAPREDUCE-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865343#action_12865343
]
Luke Lu commented on MAPREDUCE-1683:
Looks like JobClient.getClusterStatus(boolean detailed) won't behave as
expected, as there is no way to get jobtracker ClusterStatus with jobtracker
memory info from the new ClientProtocol due to MAPREDUCE-777.
> Remove JNI calls from ClusterStatus cstr
>
>
> Key: MAPREDUCE-1683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 0.20.2
>Reporter: Chris Douglas
>Assignee: Luke Lu
> Attachments: MAPREDUCE-1683_part2_yhadoop_20_10.patch,
> MAPREDUCE-1683_yhadoop_20_9.patch, MAPREDUCE-1683_yhadoop_20_S.patch
>
>
> The {{ClusterStatus}} constructor makes two JNI calls to the {{Runtime}} to
> fetch memory information. {{ClusterStatus}} instances are often created
> inside the {{JobTracker}} to obtain other, unrelated metrics (sometimes from
> schedulers' inner loops). Given that this information is related to the
> {{JobTracker}} process and not the cluster, the metrics are also available
> via {{JvmMetrics}}, and the jsps can gather this information for themselves:
> these fields can be removed from {{ClusterStatus}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1759) Exception message for unauthorized user doing killJob, killTask, setJobPriority needs to be improved
[ https://issues.apache.org/jira/browse/MAPREDUCE-1759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-1759: Attachment: 1759.20S.1.patch Attaching patch for earlier version of hadoop. Not for commit here. > Exception message for unauthorized user doing killJob, killTask, > setJobPriority needs to be improved > > > Key: MAPREDUCE-1759 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1759 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Affects Versions: 0.22.0 >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1759.20S.1.patch > > > The Exception message contains ADMINISTER_JOBS or MODIFY_JOB as the operation > name when an unauthorized user tries to do killTask, killJob or > setJobPriority operations. This needs to be changed so that user gets the > correct operation instead of ADMINISTER_JOBS or MODIFY_JOB. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1758) Building blocks for the herriot test cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865231#action_12865231 ] Konstantin Boudnik commented on MAPREDUCE-1758: --- Jeff, this is the system test framework tracked by HADOOP-6332. This is a proposed name after [James Herriot | http://en.wikipedia.org/wiki/James_Herriot] aka Alf Wight, who was a famous veterinarian and writer (thanks Arun Murphy for digging up this name). We though it'd be cool to use this name for the framework because it designated to 'cure' our little elephant :) All Herriot code is available in a form of Y20 patches at the above JIRA. I have started working on forward port last night and now patches for all components of Hadoop's trunk are available but aren't finished yet: there some known issues and gaps. I hope to finish the work in a next few days. Any reviews and comments will be very helpful. > Building blocks for the herriot test cases > > > Key: MAPREDUCE-1758 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1758 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan >Priority: Minor > Attachments: bb_patch.txt > > > There is so much commonality in the test cases that we are writing, so it is > pertinent to create reusable code. The common methods will be added to > herriot framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1693) Process tree clean up of either a failed task or killed task tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1693: - Attachment: 1693-ydist_security.patch Latest patch for Yahoo distribution security branch and this patch requires the MAPREDUCE-1713. > Process tree clean up of either a failed task or killed task tests. > --- > > Key: MAPREDUCE-1693 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1693 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1693-ydist_security.patch, 1693-ydist_security.patch, > taskchildskilling_1693.diff, taskchildskilling_1693.diff, > taskchildskilling_1693.patch, taskchildskilling_1693.patch, > taskchildskilling_1693.patch, taskchildskilling_1693.patch, > taskchildskilling_1693.patch, taskchildskilling_1693.patch > > > The following scenarios covered in the test. > 1. Run a job which spawns subshells in the tasks. Kill one of the task. All > the child process of the killed task must be killed. > 2. Run a job which spawns subshells in tasks. Fail one of the task. All the > child process of the killed task must be killed along with the task after its > failure. > 3. Check process tree cleanup on paritcular task-tracker when we use > -kill-task and -fail-task with both map and reduce. > 4. Submit a job which would spawn child processes and each of the child > processes exceeds the memory limits. Let the job complete . Check if all the > child processes are killed, the overall job should fail. > l)Submit a job which would spawn child processes and each of the child > processes exceeds the memory limits. Kill/fail the job while in progress. > Check if all the child processes are killed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1710) Process tree clean up of exceeding memory limit tasks.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1710: - Attachment: 1710-ydist_security.patch Uploading latest patch for Yahoo distribution security branch and this patch requires the MAPREDUCE-1693 and MAPREDUCE-1713. > Process tree clean up of exceeding memory limit tasks. > -- > > Key: MAPREDUCE-1710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1710 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1710-ydist_security.patch, 1710-ydist_security.patch, > memorylimittask_1710.patch, memorylimittask_1710.patch, > memorylimittask_1710.patch, memorylimittask_1710.patch, > memorylimittask_1710.patch > > > 1. Submit a job which would spawn child processes and each of the child > processes exceeds the memory limits. Let the job complete . Check if all the > child processes are killed, the overall job should fail. > 2. Submit a job which would spawn child processes and each of the child > processes exceeds the memory limits. Kill/fail the job while in progress. > Check if all the child processes are killed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-118) Job.getJobID() will always return null
[ https://issues.apache.org/jira/browse/MAPREDUCE-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865117#action_12865117 ] Hadoop QA commented on MAPREDUCE-118: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443941/patch-118-4.txt against trunk revision 941564. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 27 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/175/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/175/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/175/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/175/console This message is automatically generated. > Job.getJobID() will always return null > -- > > Key: MAPREDUCE-118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-118 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.20.1 >Reporter: Amar Kamat >Assignee: Amareshwari Sriramadasu >Priority: Blocker > Fix For: 0.21.0 > > Attachments: patch-118-0.20.txt, patch-118-0.21.txt, patch-118-1.txt, > patch-118-2.txt, patch-118-3.txt, patch-118-4.txt, patch-118.txt > > > JobContext is used for a read-only view of job's info. Hence all the readonly > fields in JobContext are set in the constructor. Job extends JobContext. When > a Job is created, jobid is not known and hence there is no way to set JobID > once Job is created. JobID is obtained only when the JobClient queries the > jobTracker for a job-id., which happens later i.e upon job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1713) Utilities for system tests specific.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1713: - Attachment: 1713-ydist-security.patch Uploading latest patch for Yahoo distribution security branch by addressing cos comments. > Utilities for system tests specific. > > > Key: MAPREDUCE-1713 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1713 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1713-ydist-security.patch, 1713-ydist-security.patch, > systemtestutils_MR1713.patch, utilsforsystemtest_1713.patch > > > 1. A method for restarting the daemon with new configuration. > public static void restartCluster(Hashtable props, String > confFile) throws Exception; > 2. A method for resetting the daemon with default configuration. > public void resetCluster() throws Exception; > 3. A method for waiting until daemon to stop. > public void waitForClusterToStop() throws Exception; > 4. A method for waiting until daemon to start. > public void waitForClusterToStart() throws Exception; > 5. A method for checking the job whether it has started or not. > public boolean isJobStarted(JobID id) throws IOException; > 6. A method for checking the task whether it has started or not. > public boolean isTaskStarted(TaskInfo taskInfo) throws IOException; -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-736) Undefined variable is treated as string.
[
https://issues.apache.org/jira/browse/MAPREDUCE-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Balaji Rajagopalan updated MAPREDUCE-736:
-
Attachment: hadoop_env.txt
I modified the unit test case, run this test to make it fixes the original
issue with mapred.child.env
> Undefined variable is treated as string.
>
>
> Key: MAPREDUCE-736
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-736
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Suman Sehgal
>Priority: Minor
> Attachments: hadoop_env.txt
>
>
> This issue is related to HADOOP-2838.
> For X=$X:Y : Append Y to X (which should be taken from the tasktracker) , if
> we append to an undefined variable then value for undefined variable should
> be displayed as blank
> e.g. NEW_PATH=$NEW_PATH2:/tmp should be displayed as
> ":/tmp" in child's environment
> while that variable is being displayed as a string ("$NEW_PATH2:/tmp") in the
> environemnt.
> This is happening in case of default task-controller only. This scenario
> works fine with linux task-controller.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-118) Job.getJobID() will always return null
[ https://issues.apache.org/jira/browse/MAPREDUCE-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-118: -- Status: Patch Available (was: Open) patch-118-4.txt applies to branch 0.21 also. All the tests passed on branch 0.21 > Job.getJobID() will always return null > -- > > Key: MAPREDUCE-118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-118 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.20.1 >Reporter: Amar Kamat >Assignee: Amareshwari Sriramadasu >Priority: Blocker > Fix For: 0.21.0 > > Attachments: patch-118-0.20.txt, patch-118-0.21.txt, patch-118-1.txt, > patch-118-2.txt, patch-118-3.txt, patch-118-4.txt, patch-118.txt > > > JobContext is used for a read-only view of job's info. Hence all the readonly > fields in JobContext are set in the constructor. Job extends JobContext. When > a Job is created, jobid is not known and hence there is no way to set JobID > once Job is created. JobID is obtained only when the JobClient queries the > jobTracker for a job-id., which happens later i.e upon job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-118) Job.getJobID() will always return null
[ https://issues.apache.org/jira/browse/MAPREDUCE-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-118: -- Attachment: patch-118-4.txt Missed another test failure in earlier patch. Fixed that. > Job.getJobID() will always return null > -- > > Key: MAPREDUCE-118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-118 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.20.1 >Reporter: Amar Kamat >Assignee: Amareshwari Sriramadasu >Priority: Blocker > Fix For: 0.21.0 > > Attachments: patch-118-0.20.txt, patch-118-0.21.txt, patch-118-1.txt, > patch-118-2.txt, patch-118-3.txt, patch-118-4.txt, patch-118.txt > > > JobContext is used for a read-only view of job's info. Hence all the readonly > fields in JobContext are set in the constructor. Job extends JobContext. When > a Job is created, jobid is not known and hence there is no way to set JobID > once Job is created. JobID is obtained only when the JobClient queries the > jobTracker for a job-id., which happens later i.e upon job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1648) Use rolling to limit tasklogs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12865054#action_12865054 ] Hadoop QA commented on MAPREDUCE-1648: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12443921/MAPREDUCE-1648-v4.patch against trunk revision 941564. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 5 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/174/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/174/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/174/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/174/console This message is automatically generated. > Use rolling to limit tasklogs > - > > Key: MAPREDUCE-1648 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1648 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Reporter: Guilin Sun > Attachments: mapreduce-1648-design.pdf, MAPREDUCE-1648-v4.patch, > syslog-baidu-v2.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > There are at least two types of task-logs: syslog and stdlog > Task-Jvm outputs syslog by log4j with TaskLogAppender, TaskLogAppender looks > just like "tail -c", it stores last N byte/line logs in memory(via queue), > and do real output only if all logs is commit and Appender is going to close. > The common problem of TaskLogAppender and 'tail -c' is keep everything in > memory and user can't see any log output while task is in progress. > So I'm going to try RollingFileAppender instead of TaskLogAppender, use > MaxFileSize&MaxBackupIndex to limit log file size. > RollingFileAppender is also suitable for stdout/stderr, just redirect > stdout/stderr to log4j via LoggingOutputStream, no client code have to be > changed, and RollingFileAppender seems better than 'tail -c' too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
