[jira] Commented: (MAPREDUCE-1978) [Rumen] TraceBuilder should provide recursive input folder scanning
[
https://issues.apache.org/jira/browse/MAPREDUCE-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915143#action_12915143
]
Vinay Kumar Thota commented on MAPREDUCE-1978:
--
Patch looks good to me. However, why don't you use -r option instead of
-recursive? In general -r represents the recursive mode right.
> [Rumen] TraceBuilder should provide recursive input folder scanning
> ---
>
> Key: MAPREDUCE-1978
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1978
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
>Reporter: Amar Kamat
>Assignee: Ravi Gummadi
> Attachments: 1978.patch
>
>
> Currently, {{TraceBuilder}} assumes that the input is either jobhistory files
> or a folders containing jobhistory files directly underneath the specified
> folder. There could be a use cases where the input folder could contain
> sub-folders containing jobhistory files. Rumen should support such input
> folders.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2008) write Junit test cases that run the Hadoop examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915141#action_12915141 ] Amareshwari Sriramadasu commented on MAPREDUCE-2008: bq. System.exit call is commented in all the examples. That should be removed. I meant "It should not be commented.". > write Junit test cases that run the Hadoop examples > --- > > Key: MAPREDUCE-2008 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2008 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: examples, test >Affects Versions: 0.20.2 > Environment: N/A >Reporter: Saikat Kanjilal >Priority: Minor > Fix For: 0.22.0 > > Attachments: HADOOP-MR-2008.patch > > Original Estimate: 20h > Remaining Estimate: 20h > > I will be adding junit test cases that run the hadoop examples, specifically > I'll be adding a test suite that tests everything under the > src/examples/org/apache/hadoop/examples package. Since this is my first > contribution to the project I'll need some help to figure out how to submit > this -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2008) write Junit test cases that run the Hadoop examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-2008: --- Status: Open (was: Patch Available) Saikat, some comments on the patch: * The tests should not run by calling Main directly. They should be run as ToolRunner().run. See TestTeraSort for example. * System.exit call is commented in all the examples. That should be removed. * The input for the examples is never generated. Are they running on empty input? * There should be a validation for every test to ensure that the example is run as expected and has produced the expected output. > write Junit test cases that run the Hadoop examples > --- > > Key: MAPREDUCE-2008 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2008 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: examples, test >Affects Versions: 0.20.2 > Environment: N/A >Reporter: Saikat Kanjilal >Priority: Minor > Fix For: 0.22.0 > > Attachments: HADOOP-MR-2008.patch > > Original Estimate: 20h > Remaining Estimate: 20h > > I will be adding junit test cases that run the hadoop examples, specifically > I'll be adding a test suite that tests everything under the > src/examples/org/apache/hadoop/examples package. Since this is my first > contribution to the project I'll need some help to figure out how to submit > this -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1517) streaming should support running on background
[ https://issues.apache.org/jira/browse/MAPREDUCE-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bochun Bai updated MAPREDUCE-1517: -- Attachment: contrib-streaming-background-with-test-with-license2.patch Previous patch contains only test changes. Sorry for that. This is a full patch. > streaming should support running on background > -- > > Key: MAPREDUCE-1517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1517 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/streaming >Reporter: Bochun Bai > Attachments: > contrib-streaming-background-with-test-with-license.patch, > contrib-streaming-background-with-test-with-license2.patch, > contrib-streaming-background-with-test.patch, > contrib-streaming-background.patch, contrib-streaming-background.patch, > hadoop-mapred-patch-logs.tar.gz > > > StreamJob submit the job and use a while loop monitor the progress. > I prefer it running on background. > Just add "&" at the end of command is a alternative solution, but it keeps a > java process on client machine. > When submit hundreds jobs at the same time, the client machine is overloaded. > Adding a -background option to StreamJob, tell it only submit and don't > monitor the progress. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1517) streaming should support running on background
[ https://issues.apache.org/jira/browse/MAPREDUCE-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915124#action_12915124 ] Amareshwari Sriramadasu commented on MAPREDUCE-1517: Your patch is missing other files. Can you regenerate the patch properly? > streaming should support running on background > -- > > Key: MAPREDUCE-1517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1517 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/streaming >Reporter: Bochun Bai > Attachments: > contrib-streaming-background-with-test-with-license.patch, > contrib-streaming-background-with-test.patch, > contrib-streaming-background.patch, contrib-streaming-background.patch, > hadoop-mapred-patch-logs.tar.gz > > > StreamJob submit the job and use a while loop monitor the progress. > I prefer it running on background. > Just add "&" at the end of command is a alternative solution, but it keeps a > java process on client machine. > When submit hundreds jobs at the same time, the client machine is overloaded. > Adding a -background option to StreamJob, tell it only submit and don't > monitor the progress. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1517) streaming should support running on background
[ https://issues.apache.org/jira/browse/MAPREDUCE-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bochun Bai updated MAPREDUCE-1517: -- Attachment: contrib-streaming-background-with-test-with-license.patch license header added. > streaming should support running on background > -- > > Key: MAPREDUCE-1517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1517 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/streaming >Reporter: Bochun Bai > Attachments: > contrib-streaming-background-with-test-with-license.patch, > contrib-streaming-background-with-test.patch, > contrib-streaming-background.patch, contrib-streaming-background.patch, > hadoop-mapred-patch-logs.tar.gz > > > StreamJob submit the job and use a while loop monitor the progress. > I prefer it running on background. > Just add "&" at the end of command is a alternative solution, but it keeps a > java process on client machine. > When submit hundreds jobs at the same time, the client machine is overloaded. > Adding a -background option to StreamJob, tell it only submit and don't > monitor the progress. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1597) combinefileinputformat does not work with non-splittable files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1597: --- Attachment: patch-1597-ydist.txt Patch for Yahoo! distribution, on top of MAPREDUCE-2046. > combinefileinputformat does not work with non-splittable files > -- > > Key: MAPREDUCE-1597 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1597 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Namit Jain >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1597-ydist.txt, patch-1597.txt > > > CombineFileInputFormat.getSplits() does not take into account whether a file > is splittable. > This can lead to a problem for compressed text files - for example, > getSplits() may return more > than 1 split depending on the size of the compressed file, all the splits > recordreader will read the > complete file. > I ran into this problem while using Hive on hadoop 20. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-44) Hang JobTracker, running out of memory
[
https://issues.apache.org/jira/browse/MAPREDUCE-44?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amareshwari Sriramadasu resolved MAPREDUCE-44.
--
Assignee: (was: Amareshwari Sriramadasu)
Resolution: Not A Problem
Closing old ticket.
> Hang JobTracker, running out of memory
> --
>
> Key: MAPREDUCE-44
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-44
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Koji Noguchi
>
> This may be expected.
> Hang JobTracker with 1G heapsize, top showed 99% cpu.
> Ran about 80 jobs. Each with 2500 mappers 200 reducers. They finish quite
> fast. 3-4 mins avg per job.
> (200k tasks)
> How much memory does JobTracker use for 'completed' (but not expired) jobs ?
> jmap -heap showed
> {noformat}
> ...
> PS Old Generation
>capacity = 932118528 (888.9375MB)
>used = 932118528 (888.9375MB)
> ...
> {noformat}
> jmap -histo showed
> {noformat}
> num #instances#bytes class name
> --
> 1: 3974182 355869992 [C
> 2: 5216606 125198544 java.lang.String
> 3: 2238560 107450880 java.util.TreeMap
> 4:463206 101673488 [B
> 5: 197999563359840 java.util.TreeMap$Entry
> 6:24840035769600 org.apache.hadoop.mapred.TaskInProgress
> 7:30880330898112 [Ljava.lang.Object;
> 8:97824023477760 org.apache.hadoop.mapred.Counters$CounterRec
> 9:24987619990080 org.apache.hadoop.mapred.TaskStatus
> 10:24883619906880 java.net.URI
> 11:23033716584264 org.apache.hadoop.mapred.MapTask
> ...
> {noformat}
> Log showing many heartbeat discarded messages
> {noformat}
> 2007-10-30 22:55:46,912 WARN org.apache.hadoop.ipc.Server: IPC Server handler
> 6 on 58567, call
> heartbeat(org.apache.hadoop.mapred.tasktrackersta...@1afb9c9, false, true,
> 3942) from 99.99.99.99: discarded for being too old (2578616)
> {noformat}
> Is the solution either to increase the jobtracker heapsize or set shorter
> 'mapred.userlog.retain.hours' ?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1517) streaming should support running on background
[ https://issues.apache.org/jira/browse/MAPREDUCE-1517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915117#action_12915117 ] Amareshwari Sriramadasu commented on MAPREDUCE-1517: Can you add apache license headers for src test files, both TestStreamingBackground.java and DelayEchoApp.java? You can copy it from any other src file. Then, you won't see release audit warnings. > streaming should support running on background > -- > > Key: MAPREDUCE-1517 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1517 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/streaming >Reporter: Bochun Bai > Attachments: contrib-streaming-background-with-test.patch, > contrib-streaming-background.patch, contrib-streaming-background.patch, > hadoop-mapred-patch-logs.tar.gz > > > StreamJob submit the job and use a while loop monitor the progress. > I prefer it running on background. > Just add "&" at the end of command is a alternative solution, but it keeps a > java process on client machine. > When submit hundreds jobs at the same time, the client machine is overloaded. > Adding a -background option to StreamJob, tell it only submit and don't > monitor the progress. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1984) herriot TestCluster fails because exclusion is not there
[ https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik resolved MAPREDUCE-1984. --- Hadoop Flags: [Reviewed] Assignee: Balaji Rajagopalan Fix Version/s: 0.21.1 Resolution: Fixed +1 patch looks good. I have just committed it to the trunk and 0.21 > herriot TestCluster fails because exclusion is not there > > > Key: MAPREDUCE-1984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.1 >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan > Fix For: 0.21.1 > > Attachments: MAPREDUCE-1984.patch, testcluster.patch > > > restart is part of the test case which causes ioexceptions, and this needs to > be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1984) herriot TestCluster fails because exclusion is not there
[
https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915110#action_12915110
]
Konstantin Boudnik commented on MAPREDUCE-1984:
---
This patch has been committed to y20-security a long time ago and it works as
expected. This is impossible to verify this patch via normal {{test-patch}}
process so I'm going to commit it on the base of the above fact.
> herriot TestCluster fails because exclusion is not there
>
>
> Key: MAPREDUCE-1984
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.21.1
>Reporter: Balaji Rajagopalan
> Attachments: MAPREDUCE-1984.patch, testcluster.patch
>
>
> restart is part of the test case which causes ioexceptions, and this needs to
> be ignored. The test case should not be incorrectly failed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1984) herriot TestCluster fails because exclusion is not there
[ https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1984: -- Summary: herriot TestCluster fails because exclusion is not there (was: herriot TestCluster fails because expclusion is not there) Environment: (was: herriot) > herriot TestCluster fails because exclusion is not there > > > Key: MAPREDUCE-1984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.1 >Reporter: Balaji Rajagopalan > Attachments: MAPREDUCE-1984.patch, testcluster.patch > > > restart is part of the test case which causes ioexceptions, and this needs to > be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1984) herriot TestCluster fails because expclusion is not there
[ https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1984: -- Attachment: MAPREDUCE-1984.patch Patch is refitted for trunk. > herriot TestCluster fails because expclusion is not there > - > > Key: MAPREDUCE-1984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.1 > Environment: herriot >Reporter: Balaji Rajagopalan > Attachments: MAPREDUCE-1984.patch, testcluster.patch > > > restart is part of the test case which causes ioexceptions, and this needs to > be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1854) [herriot] Automate health script system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1854: -- Tags: herriot > [herriot] Automate health script system test > > > Key: MAPREDUCE-1854 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1854 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: test > Environment: Herriot framework >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan > Attachments: health_script_5.txt, health_script_7.txt, > health_script_trunk.txt, health_script_y20.txt, health_script_y20_1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > 1. There are three scenarios, first is induce a error from health script, > verify that task tracker is blacklisted. > 2. Make the health script timeout and verify the task tracker is blacklisted. > 3. Make an error in the health script path and make sure the task tracker > stays healthy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1883) herriot bug with getDaemonConf
[
https://issues.apache.org/jira/browse/MAPREDUCE-1883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated MAPREDUCE-1883:
--
Tags: herriot
> herriot bug with getDaemonConf
> --
>
> Key: MAPREDUCE-1883
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1883
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Environment: herriot dev
>Reporter: Balaji Rajagopalan
>
> I found this bug with getDaemonConf the interface method implemented by
> JTProtocol and TTProtocol.
> Configuration tConf = client.getProxy().getDaemonConf();
> String reportAddress = tConf.get("mapred.task.tasker.report.address");
> System.out.println(reportAddress);
> It prints it out as 0:0:0:0:12345 instead of printing it as 0.0.0.0:12345,
> and this issues needs to be fixed, somewhere is the wire the report address
> is getting mangled.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1962) [Herriot] IOException throws and it fails with token expired while running the tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1962: -- Tags: herriot > [Herriot] IOException throws and it fails with token expired while running > the tests. > - > > Key: MAPREDUCE-1962 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1962 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1962-ydist-security.patch > > > Throwing IOException and tests fails due to token is expired. I could see > this issue in a secure cluster. > This issue has been resolved by setting the following attribute in the > configuration before running the tests. > mapreduce.job.complete.cancel.delegation.tokens=false > 2. Fix the timing issue in testtaskkilling test. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1871) Create automated test scenario for "Collect information about number of tasks succeeded / total per time unit for a tasktracker"
[ https://issues.apache.org/jira/browse/MAPREDUCE-1871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1871: -- Tags: herriot > Create automated test scenario for "Collect information about number of tasks > succeeded / total per time unit for a tasktracker" > > > Key: MAPREDUCE-1871 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1871 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: 1871-ydist-security-patch.txt, > 1871-ydist-security-patch.txt, 1871-ydist-security-patch.txt, > 1871-ydist-security-patch.txt, 1871-ydist-security-patch.txt, > 1871-ydist-security-patch.txt, 1871-ydist-security-patch.txt, > 1871-ydist-security-patch.txt, 1871-ydist-security-patch.txt, > MAPREDUCE-1871.patch, MAPREDUCE-1871.patch, MAPREDUCE-1871.patch, > MAPREDUCE-1871.patch, MAPREDUCE-1871.patch > > > Create automated test scenario for "Collect information about number of tasks > succeeded / total per time unit for a tasktracker" > 1) Verification of all the above mentioned fields with the specified TTs. > Total no. of tasks and successful tasks should be equal to the corresponding > no. of tasks specified in TTs logs > 2) Fail a task on tasktracker. Node UI should update the status of tasks on > that TT accordingly. > 3) Kill a task on tasktracker. Node UI should update the status of tasks on > that TT accordingly > 4) Positive Run simultaneous jobs and check if all the fields are populated > with proper values of tasks. Node UI should have correct valiues for all the > fields mentioned above. > 5) Check the fields across one hour window Fields related to hour should be > updated after every hour > 6) Check the fields across one day window fields related to hour should be > updated after every day > 7) Restart a TT and bring it back. UI should retain the fields values. > 8) Positive Run a bunch of jobs with 0 maps and 0 reduces simultanously. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1730) Automate test scenario for successful/killed jobs' memory is properly removed from jobtracker after these jobs retire.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1730: -- Tags: herriot > Automate test scenario for successful/killed jobs' memory is properly removed > from jobtracker after these jobs retire. > -- > > Key: MAPREDUCE-1730 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1730 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.21.0 >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: MAPREDUCE-1730.patch, MAPREDUCE-1730.patch, > MAPREDUCE-1730.patch, TestJobRetired.patch, TestJobRetired.patch, > TestRetiredJobs-ydist-security-patch.txt, > TestRetiredJobs-ydist-security-patch.txt, TestRetiredJobs.patch > > > Automate using herriot framework, test scenario for successful/killed jobs' > memory is properly removed from jobtracker after these jobs retire. > This should test when successful and failed jobs are retired, their > jobInProgress object are removed properly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1957) [Herriot] Test Job cache directories cleanup after job completes.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1957: -- Tags: herriot > [Herriot] Test Job cache directories cleanup after job completes. > - > > Key: MAPREDUCE-1957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1957 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1957-ydist-security.patch, 1957-ydist-security.patch, > 1957-ydist-security.patch > > > Test the job cache directories cleanup after job completes.Test covers the > following scenarios. > 1. Submit a job and create folders and files in work folder with > non-writable permissions under task attempt id folder. Wait till the job > completes and verify whether the files and folders are cleaned up or not. > 2. Submit a job and create folders and files in work folder with > non-writable permissions under task attempt id folder. Kill the job and > verify whether the files and folders are cleaned up or not. > 3. Submit a job and create folders and files in work folder with > non-writable permissions under task attempt id folder. Fail the job and > verify whether the files and folders are cleaned up or not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1827) [Herriot] Task Killing/Failing tests for a streaming job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1827: -- Tags: herriot > [Herriot] Task Killing/Failing tests for a streaming job. > - > > Key: MAPREDUCE-1827 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1827 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1827-ydist-security.patch, 1827-ydist-security.patch, > MAPREDUCE-1827.patch > > > 1. Set the sleep time for the tasks is 3 seconds and kill the task of > streaming job using SIGKILL. After that verify whether task is killed after > 3 seconds or not and also verify whether job is succeeded or not. > 2. Set the maximum attempts for the maps and reducers are one. make the task > to fail and verify whether task is failed or not.Also verify whether the job > is failed or not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1934) Timing issue with Distributed Cache testcases for already automated (herriot ) test scenarios.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1934: -- Tags: herriot > Timing issue with Distributed Cache testcases for already automated (herriot > ) test scenarios. > -- > > Key: MAPREDUCE-1934 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1934 > Project: Hadoop Map/Reduce > Issue Type: Test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: DistributedCacheTiming.patch > > > Sometimes, Distributed cache testcases fail becuse the tasktrackers dont come > up fast enough after kill and start. This needs to be fixed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1963) [Herriot] TaskMemoryManager should log process-tree's status while killing tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1963: -- Tags: herriot > [Herriot] TaskMemoryManager should log process-tree's status while killing > tasks > > > Key: MAPREDUCE-1963 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1963 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1963-ydist-security.patch > > > 1. Execute a streaming job which will increase memory usage beyond configured > memory limits during mapping phase. TaskMemoryManager should logs a map > task's process-tree's status just before killing the task. > 2. Execute a streaming job which will increase memory usage beyond configured > memory limits during reduce phase. TaskMemoryManager should logs a > reduce task's process-tree's status just before killing the task. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1971) herriot automation system test case for verification of bug fix to jobhistory
[ https://issues.apache.org/jira/browse/MAPREDUCE-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1971: -- Tags: herriot > herriot automation system test case for verification of bug fix to jobhistory > - > > Key: MAPREDUCE-1971 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1971 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Environment: herriot >Reporter: Balaji Rajagopalan > Attachments: concurrent_exp_y20.patch, concurrent_exp_y20_1.patch, > concurrent_y20_fix.patch > > > Run a few jobs and check the job history page . Job history information > should be displayed properly . > Analyze a running job . The values shown in the page should be correct . > Concurrently access jobs in job history page . No exception should be thrown. > In the developed herriot test case accesses the job tracker tracker directly, > the jsp page access does the same. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1899) [Herriot] Test jobsummary information for different jobs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1899: -- Tags: herriot > [Herriot] Test jobsummary information for different jobs. > - > > Key: MAPREDUCE-1899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1899 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1899-ydist-security.patch, 1899-ydist-security.patch, > 1899-ydist-security.patch > > > Test the following scenarios. > 1. Verify the job summary information for killed job. > 2. Verify the job summary information for failed job. > 3. Verify the job queue information in job summary after job has successfully > completed. > 4. Verify the job summary information for high ram jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1758) Building blocks for the herriot test cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1758: -- Tags: herriot > Building blocks for the herriot test cases > > > Key: MAPREDUCE-1758 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1758 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan >Priority: Minor > Attachments: bb_patch.txt, bb_patch_1.txt, bb_patch_2.txt > > > There is so much commonality in the test cases that we are writing, so it is > pertinent to create reusable code. The common methods will be added to > herriot framework. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1984) herriot TestCluster fails because expclusion is not there
[ https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1984: -- Tags: herriot > herriot TestCluster fails because expclusion is not there > - > > Key: MAPREDUCE-1984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.1 > Environment: herriot >Reporter: Balaji Rajagopalan > Attachments: testcluster.patch > > > restart is part of the test case which causes ioexceptions, and this needs to > be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2053) [Herriot] Test Gridmix file pool for different input file sizes based on pool minimum size.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated MAPREDUCE-2053:
--
Tags: herriot
> [Herriot] Test Gridmix file pool for different input file sizes based on pool
> minimum size.
> ---
>
> Key: MAPREDUCE-2053
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2053
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: contrib/gridmix
>Reporter: Vinay Kumar Thota
>Assignee: Vinay Kumar Thota
> Attachments: 2053-ydist.patch, MAPREDUCE-2053.patch, ydist-2053.patch
>
>
> Scenario:
> 1. Generate 1.8G data with Gridmix data generator, such that the files can
> create under different folders inside the given input directory and also
> create the files directly in the given input directory with the following
> sizes {50 MB,100 MB,400 MB, 50 MB,300 MB,10 MB ,60 MB,40 MB,20 MB,10 MB,500
> MB}.
> 2.Set the FilePool minimum size is 100 MB.
> 3. Verify the files count and sizes after excluding the files that are less
> than file pool minimum size.Also make sure, whether files are collected
> recursively from the different locations under input folder or not.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1882) Use Jsch instead of Shell.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-1882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1882: -- Tags: herriot > Use Jsch instead of Shell.java > --- > > Key: MAPREDUCE-1882 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1882 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: test > Environment: herriot framework >Reporter: Balaji Rajagopalan >Assignee: Iyappan Srinivasan > Attachments: 1882-ydist-security-patch.txt, MAPREDUCE-1882.patch, > RemoteExecution.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > In herriot ( hadoop system test case dev) we often find that we are resorted > to habit of ssh to remote node execute a shell command, and come out. It is > wise to use Jsch instead of doing this through Shell.java ( hadoop code), > since Jsch provides nice Java abstraction, the JIRA will only close after we > import Jsch input hadoop build system and also fix all the existing test > cases. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1760) Herriot tests for MapReduce should support submission into a specific queue
[ https://issues.apache.org/jira/browse/MAPREDUCE-1760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1760: -- Tags: herriot > Herriot tests for MapReduce should support submission into a specific queue > --- > > Key: MAPREDUCE-1760 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1760 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: test >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik > > Hadoop might be configured in a way to allow job submission only into a > special queues. > Herriot tests have to be able to read a queue name configuration parameter > from system-test.xml file and use it for job submission. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1898) [Herriot] Implement a functionality for getting the job summary information of a job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1898: -- Tags: herriot > [Herriot] Implement a functionality for getting the job summary information > of a job. > - > > Key: MAPREDUCE-1898 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1898 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Affects Versions: 0.20.1 >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1898-ydist-security.patch, 1898-ydist-security.patch, > 1898-ydist-security.patch, 1898-ydist-security.patch, MAPREDUCE-1898.patch, > MAPREDUCE-1898.patch, MAPREDUCE-1898.patch > > > Implement a method for getting the job summary details of a job. The job > summary should be. > jobId, startTime, launchTime, finishTime, numMaps, numSlotsPerMap, > numReduces, numSlotsPerReduce, user, queue, status, mapSlotSeconds, > reduceSlotSeconds, clusterMapCapacity,clusterReduceCapacity. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2033) [Herriot] Gridmix generate data tests with various submission policies and different user resolvers.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-2033: -- Tags: herriot > [Herriot] Gridmix generate data tests with various submission policies and > different user resolvers. > > > Key: MAPREDUCE-2033 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2033 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: contrib/gridmix >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 2033-ydist.patch, 2033-ydist.patch, MAPREDUCE-2033.patch > > > Tests for submitting and verifying the gridmix generate input data in > different submission policies and various user resolver modes. It covers the > following scenarios. > 1. Generate the data in a STRESS submission policy with SubmitterUserResolver > mode and verify whether the generated data matches with given size of input > or not. > 2. Generate the data in a REPLAY submission policy with > RoundRobinUserResolver mode and verify whether the generated data matches > with the given input size or not. > 3. Generate the data in a SERIAL submission policy with EchoUserResolver mode > and specify the no.of bytes per file. Verify whether each file size is > matches with given per file size or not and also verify the overall size of > generated data. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1671) Test scenario for "Killing Task Attempt id till job fails"
[ https://issues.apache.org/jira/browse/MAPREDUCE-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1671: -- Tags: herriot > Test scenario for "Killing Task Attempt id till job fails" > -- > > Key: MAPREDUCE-1671 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1671 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: 1671-ydist-security-patch.txt, > TEST-org.apache.hadoop.mapred.TestTaskKilling.txt, TestTaskKilling.patch, > TestTaskKilling.patch, TestTaskKilling.patch, TestTaskKilling.patch, > TestTaskKilling.patch > > > 1) In a job, kill the task attemptid of one task. Whenever that task tries > to run again with another task atempt id for mapred.map.max.attempts times, > kill that task attempt id. After the mapred.map.max.attempts times, the whole > job should get killed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1616) Automate system test case for checking the file permissions in mapred.local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1616: -- Tags: herriot > Automate system test case for checking the file permissions in > mapred.local.dir > --- > > Key: MAPREDUCE-1616 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1616 > Project: Hadoop Map/Reduce > Issue Type: Test > Environment: Herriot framework is required for running the test. >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan > Attachments: patch-1616_1.txt, patch_1616.txt, patch_1616.txt, > patch_1616_2.txt, patch_1655.txt, patch_3392207_7.txt > > Original Estimate: 0.27h > Remaining Estimate: 0.27h > > The file under mapred.local.dir permission must be recursively tested when > the task is running, for this use the controllable task, so the temporary > file permission can be checked. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2055) Retired job info cache should contain more details than just JobStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-2055: -- Tags: herriot > Retired job info cache should contain more details than just JobStatus > -- > > Key: MAPREDUCE-2055 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2055 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Reporter: Krishna Ramachandran > Fix For: 0.22.0 > > Attachments: fix4_MAPREDUCE-2055.patch > > > RetireJobInfo cache currently holds only job status. Other data like profile, > counters are obtained from CompletedJobStatusStore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1889) [herriot] Ability to restart a single node for pushconfig
[ https://issues.apache.org/jira/browse/MAPREDUCE-1889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1889: -- Tags: herriot > [herriot] Ability to restart a single node for pushconfig > - > > Key: MAPREDUCE-1889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1889 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: test >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan > Attachments: restartDaemon.txt, restartDaemon_1.txt, > restartDaemon_y20.patch > > > Right now the pushconfig is supported only at a cluster level, this jira will > introduce the functionality to be supported at node level. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1919) [Herriot] Test for verification of per cache file ref count.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1919: -- Tags: herriot > [Herriot] Test for verification of per cache file ref count. > - > > Key: MAPREDUCE-1919 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1919 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1919-ydist-security.patch, 1919-ydist-security.patch, > MAPREDUCE-1919.patch > > > It covers the following scenarios. > 1. Run the job with two distributed cache files and verify whether job is > succeeded or not. > 2. Run the job with distributed cache files and remove one cache file from > the DFS when it is localized.verify whether the job is failed or not. > 3. Run the job with two distribute cache files and the size of one file > should be larger than local.cache.size.Verify whether job is succeeded or > not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1983) [Herriot] Test linux task controller with invalid information in conf.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1983: -- Tags: herriot > [Herriot] Test linux task controller with invalid information in conf. > -- > > Key: MAPREDUCE-1983 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1983 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1983-ydist-security.patch > > > This case is about scenarios where the taskcontroller file is absent, or > contains incorrect values for mapred.local.dir.Submit a job and verify > whether job fails or not. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1984) herriot TestCluster fails because exclusion is not there
[ https://issues.apache.org/jira/browse/MAPREDUCE-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1984: -- Summary: herriot TestCluster fails because exclusion is not there (was: herriot TestCluster fails because expclusion is not there) Affects Version/s: 0.21.1 Description: restart is part of the test case which causes IOexceptions, and this needs to be ignored. The test case should not be incorrectly failed. (was: restart is part of the test case which causes ioexceptions, and this needs to be ignored. The test case should not be incorrectly failed. ) > herriot TestCluster fails because exclusion is not there > > > Key: MAPREDUCE-1984 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1984 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.21.1 > Environment: herriot >Reporter: Balaji Rajagopalan > Attachments: testcluster.patch > > > restart is part of the test case which causes IOexceptions, and this needs to > be ignored. The test case should not be incorrectly failed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
