[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test framwork for Map-Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874424#action_12874424 ] Konstantin Boudnik commented on MAPREDUCE-1154: --- I'd suggest to close this JIRA because it has been largely covered by HADOOP-6332 and coming soon MAPREDUCE-1774 Large-scale, automated test framwork for Map-Reduce --- Key: MAPREDUCE-1154 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154 Project: Hadoop Map/Reduce Issue Type: New Feature Components: test Reporter: Arun C Murthy Fix For: 0.21.0 Attachments: testing.patch HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop. This jira is meant to track relevant work to Map-Reduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test framwork for Map-Reduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12787039#action_12787039
]
Konstantin Boudnik commented on MAPREDUCE-1154:
---
bq. Has anyone done the reverse and unweaved functions from classes?
One way to do this is to create {{around}} advice for all calls of the method
you are interested in. This advice will do nothing but return immediately.
Large-scale, automated test framwork for Map-Reduce
---
Key: MAPREDUCE-1154
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: test
Reporter: Arun C Murthy
Fix For: 0.21.0
Attachments: testing.patch
HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for
Hadoop.
This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test framwork for Map-Reduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12769759#action_12769759
]
Arun C Murthy commented on MAPREDUCE-1154:
--
Some Map-Reduce specific functionality to provide a flavour for the proposal:
{noformat}
/**
* Daemon/job artifacts.
*/
public enum Artifact {
LOGS,
OUTPUTS
}
/**
* Get tasktrackers on the given rack.
* @param cluster map-reduce codeCluster/code
* @param rackId rack
* @return tasktrackers on the given rack
* @throws IOException
* @throws InterruptedException
*/
public static ListTaskTracker getTaskTrackersOnRack(Cluster cluster,
String rackId)
throws IOException, InterruptedException;
/**
* Get tasks running on the given tasktracker.
* @param cluster map-reduce codeCluster/code
* @param taskTracker task-tracker
* @return tasks running on the given tasktracker
* @throws IOException
* @throws InterruptedException
*/
public static ListTaskAttemptID
getTasksRunningOnTaskTracker(Cluster cluster, TaskTracker taskTracker)
throws IOException, InterruptedException;
/**
* Get tasktracker on the given host.
* @param cluster map-reduce codeCluster/code
* @param hostName host
* @return tasktracker on the given host
* @throws IOException
* @throws InterruptedException
*/
public static TaskTracker getTaskTracker(Cluster cluster, String hostName)
throws IOException, InterruptedException;
/**
* Get tasktrackers on which tasks of the given job executed.
* @param job map-reduce codeJob/code
* @param taskAttemptIDs optional list of task-attempts whose tasktrackers
* need to be fetched
* @return tasktrackers on which tasks of the given job executed, optionally
* for a specific set of task-attempts
* @throws IOException
* @throws InterruptedException
*/
public static ListTaskTracker
getTaskTrackersForJob(Job job, TaskAttemptID ... taskAttemptIDs)
throws IOException, InterruptedException;
/**
* Fetch artifacts of a given job (optionally a specific set of task-attempts
of
* the job).
* @param job job whose logs are to be fetched
* @param type artifact type
* @param dir directory to place the fetched logs
* @param taskAttemptIDs optional list of task-attempts of the job
* @throws IOException
*/
public static void fetchJobArtifacts(Job job,
Artifact type, Path dir,
TaskAttemptID ... taskAttemptIDs)
throws IOException;
/**
* Fetch job artifacts and check if they have the codepattern/code.
* @param job map-reduce job
* @param pattern pattern to check
* @param fetch if codetrue/code fetch the artifacts into
* codedir/code, else do not fetch
* @param dir directory to place the fetched outputs
* @param taskAttempts optional tasks of the job
* @return codetrue/code if the artifacts contain codepattern/code,
* codefalse/code otherwise
* @throws IOException
*/
public static boolean checkJobArtifacts(Job job,
Artifact type, String pattern,
boolean fetch, Path dir,
TaskAttemptID... taskAttempts)
throws IOException;
/**
* Kill the given task-tracker
* @param cluster map-reduce codeCluster/code
* @param taskTracker codeTaskTracker/code to be killed
* @throws IOException
* @throws InterruptedException
*/
public static void killTaskTracker(Cluster cluster, TaskTracker taskTracker)
throws IOException, InterruptedException;
/**
* Kill a given task-attempt running on the given tasktracker
* @param cluster map-reduce codeCluster/code
* @param taskTracker codeTaskTracker/code on which the task is running
* @param taskAttemptId task-attempt to be killed
* @throws IOException
* @throws InterruptedException
*/
public static void killTask(Cluster cluster, TaskTracker taskTracker,
TaskAttemptID taskAttemptId)
throws IOException, InterruptedException;
{noformat}
Large-scale, automated test framwork for Map-Reduce
---
Key: MAPREDUCE-1154
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: test
Reporter: Arun C Murthy
Fix For: 0.21.0
HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for
Hadoop.
This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test framwork for Map-Reduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12769762#action_12769762
]
Arun C Murthy commented on MAPREDUCE-1154:
--
ClusterTest.java:
{noformat}
public class ClusterTest extends TestCase {
Cluster cluster;
Configuration conf;
@Override
protected void setUp() throws Exception {
conf = new Configuration();
ClusterTestUtils.setupCluster(conf);
cluster = new Cluster(conf);
}
@Test
public void testCluster() throws Exception {
// Load data
DataLoader dataLoader = new DistcpDataLoader();
ListPath inputs = new ArrayListPath();
inputs.add(new Path(in));
dataLoader.load(inputs, new Path(out));
// Setup job
Job job = Job.getInstance(cluster);
job.setJobName(cluster-test-mr-job);
// Run job
long start = System.currentTimeMillis();
job.submit();
job.waitForCompletion(true);
long end = System.currentTimeMillis();
FileContext fc = FileContext.getLocalFSFileContext();
// Fetch daemon logs and validate them
Path daemonLogs = new Path(daemon-logs);
assertFalse(JobTracker logs contain FATAL warnings!,
MRClusterTestUtils.checkDaemonLogs(
cluster, LogSource.JOBTRACKER, start, end,
FATAL, true, daemonLogs) == false);
assertFalse(TaskTracker logs contain FATAL warnings!,
MRClusterTestUtils.checkDaemonLogs(
cluster, LogSource.TASKTRACKER, start, end,
FATAL, true, daemonLogs) == false);
fc.delete(daemonLogs, true);
// Fetch job logs, outputs etc. and validate them
Path jobLogs = new Path(daemon-logs);
Path jobOutputs = new Path(daemon-logs);
TaskAttemptID m0 = null; // map 0
TaskAttemptID m1 = null; // map 1
TaskAttemptID r0 = null; // reduce 0
assertFalse(Task log contains FATAL warnings!,
MRClusterTestUtils.checkJobArtifacts(job,
Artifact.LOGS, FATAL,
true, jobLogs) == false);
assertFalse(Task log contains WARN warnings!,
MRClusterTestUtils.checkJobArtifacts(
job, Artifact.LOGS, WARN,
false, jobLogs, m0, m1) == false);
assertTrue(Task outputs do not contain expected output!,
MRClusterTestUtils.checkJobArtifacts(
job, Artifact.OUTPUTS,
hello, world!, true, jobOutputs, r0));
fc.delete(jobLogs, true);
fc.delete(jobOutputs, true);
}
@Override
protected void tearDown() throws Exception {
ClusterTestUtils.tearDownCluster(conf);
}
}
{noformat}
Large-scale, automated test framwork for Map-Reduce
---
Key: MAPREDUCE-1154
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: test
Reporter: Arun C Murthy
Fix For: 0.21.0
Attachments: testing.patch
HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for
Hadoop.
This jira is meant to track relevant work to Map-Reduce.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1154) Large-scale, automated test framwork for Map-Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12769898#action_12769898 ] Philip Zeyliger commented on MAPREDUCE-1154: I'm vaguely uncomfortable with having a lot of code, even though it's test code, weaved in via AspectJ. It seems like it will make it very easy to make changes that break the testing code (because the testing code is not visible to the regular tools, and is in an unexpected place). I understand, of course, that the build system will check that the weaving can happen, but since these tests are inherently large-scale and not run at every Hudson (or are they?), it worries me a bit. Has anyone done the reverse and unweaved functions from classes? Seems like we could annotate functions with @RemoveInProduction, and then use some tool to forcibly remove methods from the resulting .class files. Still opaque, but at least it's clear where the testing code is. If you've already been working with this in AspectJ, I'm curious how the experience has been. -- Philip Large-scale, automated test framwork for Map-Reduce --- Key: MAPREDUCE-1154 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1154 Project: Hadoop Map/Reduce Issue Type: New Feature Components: test Reporter: Arun C Murthy Fix For: 0.21.0 Attachments: testing.patch HADOOP-6332 proposes a large-scale, automated, junit-based test-framework for Hadoop. This jira is meant to track relevant work to Map-Reduce. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
