[jira] Commented: (MAPREDUCE-1933) Create automated testcase for tasktracker dealing with corrupted disk.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12893905#action_12893905
]
Konstantin Boudnik commented on MAPREDUCE-1933:
---
- I still see that string mapred.local.dir is used all over the place in the
trunk patch. First, don't use string literals where there's a named constant
for the same; Second: this is a deprecated parameter: javac will curse and
raise the level of warnings.
- Shall these two be swapped, actually?
{noformat}
+cluster.tearDown();
+cluster.restart();
{noformat}
Perhaps, cleaning the folders should also be done before the {{tearDown()}}
call?
- Is there defined constants for something like
test.randomwrite.bytes_per_map ? If not - define a private member of the test
class instead of using string literals.
Create automated testcase for tasktracker dealing with corrupted disk.
--
Key: MAPREDUCE-1933
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Reporter: Iyappan Srinivasan
Assignee: Iyappan Srinivasan
Attachments: 1933-ydist-security-patch.txt,
1933-ydist-security-patch.txt, MAPREDUCE-1933.patch, MAPREDUCE-1933.patch,
MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
After the TaskTracker has already run some tasks successfully, corrupt a
disk by making the corresponding mapred.local.dir unreadable/unwritable.
Make sure that jobs continue to succeed even though some tasks scheduled
there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for tasktracker dealing with corrupted disk.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12891725#action_12891725
]
Konstantin Boudnik commented on MAPREDUCE-1933:
---
Let's see...
- {{find src -name *java | xargs grep 'mapred.*.local.dir'}} shows that
{noformat}
src/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java:
Configuration.addDeprecation(mapred.local.dir,
{noformat}
- also it finds
{noformat}
src/java/org/apache/hadoop/mapreduce/MRConfig.java:
public static final String LOCAL_DIR = mapreduce.cluster.local.dir;
{noformat}
Hope it helps.
Create automated testcase for tasktracker dealing with corrupted disk.
--
Key: MAPREDUCE-1933
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Reporter: Iyappan Srinivasan
Assignee: Iyappan Srinivasan
Attachments: 1933-ydist-security-patch.txt, MAPREDUCE-1933.patch,
MAPREDUCE-1933.patch, TestCorruptedDiskJob.java
After the TaskTracker has already run some tasks successfully, corrupt a
disk by making the corresponding mapred.local.dir unreadable/unwritable.
Make sure that jobs continue to succeed even though some tasks scheduled
there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for tasktracker dealing with corrupted disk.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12891468#action_12891468 ] Balaji Rajagopalan commented on MAPREDUCE-1933: --- +1 Create automated testcase for tasktracker dealing with corrupted disk. -- Key: MAPREDUCE-1933 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Iyappan Srinivasan Assignee: Iyappan Srinivasan Attachments: 1933-ydist-security-patch.txt, TestCorruptedDiskJob.java After the TaskTracker has already run some tasks successfully, corrupt a disk by making the corresponding mapred.local.dir unreadable/unwritable. Make sure that jobs continue to succeed even though some tasks scheduled there fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for tasktracker dealing with corrupted disk.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12888468#action_12888468 ] Konstantin Boudnik commented on MAPREDUCE-1933: --- bq. prop.put(mapred.local.dir, /grid/0/dev/tmp/mapred/mapred-local,/grid/1/dev/tmp/mapred/mapred-local,/grid/2/dev/tmp/mapred/mapred-local,/grid/3/dev/tmp/mapred/mapred-local); Absolutely, besides this particular parameter should be set by a normal MR config already. Also, please don't use string literals for configuration parameters. There was a significant effort in 0.21 to have all configuration keys refactored to named constants. Use them instead. Create automated testcase for tasktracker dealing with corrupted disk. -- Key: MAPREDUCE-1933 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Iyappan Srinivasan Assignee: Iyappan Srinivasan Attachments: TestCorruptedDiskJob.java After the TaskTracker has already run some tasks successfully, corrupt a disk by making the corresponding mapred.local.dir unreadable/unwritable. Make sure that jobs continue to succeed even though some tasks scheduled there fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1933) Create automated testcase for tasktracker dealing with corrupted disk.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12887690#action_12887690
]
Balaji Rajagopalan commented on MAPREDUCE-1933:
---
You have attached the java file, remember to create a patch.
prop.put(mapred.local.dir,
/grid/0/dev/tmp/mapred/mapred-local,/grid/1/dev/tmp/mapred/mapred-local,/grid/2/dev/tmp/mapred/mapred-local,/grid/3/dev/tmp/mapred/mapred-local);
Please don't hard code cluster dependent values, have them come from config
file.
//Making sure that the job is complete.
while (jInfo != null !jInfo.getStatus().isJobComplete()) {
Thread.sleep(1);
jInfo = remoteJTClient.getJobInfo(id);
jStatus = jInfo.getStatus();
}
Please use the building block JTClient::isJobStopped.
Create automated testcase for tasktracker dealing with corrupted disk.
--
Key: MAPREDUCE-1933
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1933
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Reporter: Iyappan Srinivasan
Assignee: Iyappan Srinivasan
Attachments: TestCorruptedDiskJob.java
After the TaskTracker has already run some tasks successfully, corrupt a
disk by making the corresponding mapred.local.dir unreadable/unwritable.
Make sure that jobs continue to succeed even though some tasks scheduled
there fail.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
