[jira] Created: (HADOOP-6956) compile-core-test
compile-core-test
-
Key: HADOOP-6956
URL: https://issues.apache.org/jira/browse/HADOOP-6956
Project: Hadoop Common
Issue Type: Task
Components: build
Affects Versions: 0.21.0
Reporter: Vivek Mishra
Currently compile-core-test only compiles test/core classes. classes defined in
test/system are required by Mapreduce project's test classes. It requires to
make a minor change in common build.xml as given below:
Changed
javac
encoding=${build.encoding}
srcdir=${test.src.dir}
includes=**/org/apache/hadoop/**/*.java
destdir=${test.core.build.classes}
debug=${javac.debug}
optimize=${javac.optimize}
target=${javac.version}
source=${javac.version}
deprecation=${javac.deprecation}
compilerarg line=${javac.args} ${javac.args.warnings} /
classpath refid=test.classpath/
/javac
Original:
javac
encoding=${build.encoding}
srcdir=${test.src.dir}/core
includes=org/apache/hadoop/**/*.java
destdir=${test.core.build.classes}
debug=${javac.debug}
optimize=${javac.optimize}
target=${javac.version}
source=${javac.version}
deprecation=${javac.deprecation}
compilerarg line=${javac.args} ${javac.args.warnings} /
classpath refid=test.classpath/
/javac
change is to
srcdir=${test.src.dir}
includes=**/org/apache/hadoop/**/*.java
from original
srcdir=${test.src.dir}/core
includes=org/apache/hadoop/**/*.java
Please suggest.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
how to view the result of kmeans
hi, I install the mahout and hadoop on SUSE Linux, and it works. I run the kmeans sample use command: hadoop fs -put synthetic_control.data testdata mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job it run successfully here is the output files drwxr-xr-x - root supergroup 0 2010-09-17 10:59 /user/root/output/canopies drwxr-xr-x - root supergroup 0 2010-09-17 10:59 /user/root/output/canopies/_logs drwxr-xr-x - root supergroup 0 2010-09-17 10:59 /user/root/output/canopies/_logs/history -rw-r--r-- 1 root supergroup 17229 2010-09-17 10:59 /user/root/output/canopies/_logs/history/localhost_1284692116007_job_2010091 71055_0002_conf.xml -rw-r--r-- 1 root supergroup 9165 2010-09-17 10:59 /user/root/output/canopies/_logs/history/localhost_1284692116007_job_2010091 71055_0002_root_mahout-examples-0.3.job -rw-r--r-- 1 root supergroup 5749 2010-09-17 10:59 /user/root/output/canopies/part-0 drwxr-xr-x - root supergroup 0 2010-09-17 11:00 /user/root/output/clusters-0 drwxr-xr-x - root supergroup 0 2010-09-17 10:59 /user/root/output/clusters-0/_logs drwxr-xr-x - root supergroup 0 2010-09-17 10:59 /user/root/output/clusters-0/_logs/history -rw-r--r-- 1 root supergroup 17479 2010-09-17 10:59 /user/root/output/clusters-0/_logs/history/localhost_1284692116007_job_20100 9171055_0003_conf.xml -rw-r--r-- 1 root supergroup 9283 2010-09-17 10:59 /user/root/output/clusters-0/_logs/history/localhost_1284692116007_job_20100 9171055_0003_root_mahout-examples-0.3.job -rw-r--r-- 1 root supergroup 5757 2010-09-17 11:00 /user/root/output/clusters-0/part-0 drwxr-xr-x - root supergroup 0 2010-09-17 11:00 /user/root/output/clusters-1 drwxr-xr-x - root supergroup 0 2010-09-17 11:00 /user/root/output/clusters-1/_logs drwxr-xr-x - root supergroup 0 2010-09-17 11:00 /user/root/output/clusters-1/_logs/history -rw-r--r-- 1 root supergroup 17481 2010-09-17 11:00 /user/root/output/clusters-1/_logs/history/localhost_1284692116007_job_20100 9171055_0004_conf.xml -rw-r--r-- 1 root supergroup 9283 2010-09-17 11:00 /user/root/output/clusters-1/_logs/history/localhost_1284692116007_job_20100 9171055_0004_root_mahout-examples-0.3.job -rw-r--r-- 1 root supergroup 5757 2010-09-17 11:00 /user/root/output/clusters-1/part-0 drwxr-xr-x - root supergroup 0 2010-09-17 11:01 /user/root/output/clusters-2 drwxr-xr-x - root supergroup 0 2010-09-17 11:00 /user/root/output/clusters-2/_logs drwxr-xr-x - root supergroup 0 2010-09-17 11:00 /user/root/output/clusters-2/_logs/history -rw-r--r-- 1 root supergroup 17481 2010-09-17 11:00 /user/root/output/clusters-2/_logs/history/localhost_1284692116007_job_20100 9171055_0005_conf.xml -rw-r--r-- 1 root supergroup 9496 2010-09-17 11:00 /user/root/output/clusters-2/_logs/history/localhost_1284692116007_job_20100 9171055_0005_root_mahout-examples-0.3.job -rw-r--r-- 1 root supergroup 5757 2010-09-17 11:01 /user/root/output/clusters-2/part-0 drwxr-xr-x - root supergroup 0 2010-09-17 11:01 /user/root/output/clusters-3 drwxr-xr-x - root supergroup 0 2010-09-17 11:01 /user/root/output/clusters-3/_logs drwxr-xr-x - root supergroup 0 2010-09-17 11:01 /user/root/output/clusters-3/_logs/history -rw-r--r-- 1 root supergroup 17481 2010-09-17 11:01 /user/root/output/clusters-3/_logs/history/localhost_1284692116007_job_20100 9171055_0006_conf.xml -rw-r--r-- 1 root supergroup 9496 2010-09-17 11:01 /user/root/output/clusters-3/_logs/history/localhost_1284692116007_job_20100 9171055_0006_root_mahout-examples-0.3.job -rw-r--r-- 1 root supergroup 5757 2010-09-17 11:01 /user/root/output/clusters-3/part-0 drwxr-xr-x - root supergroup 0 2010-09-17 11:02 /user/root/output/clusters-4 drwxr-xr-x - root supergroup 0 2010-09-17 11:01 /user/root/output/clusters-4/_logs drwxr-xr-x - root supergroup 0 2010-09-17 11:01 /user/root/output/clusters-4/_logs/history -rw-r--r-- 1 root supergroup 17481 2010-09-17 11:01 /user/root/output/clusters-4/_logs/history/localhost_1284692116007_job_20100 9171055_0007_conf.xml -rw-r--r-- 1 root supergroup 9496 2010-09-17 11:01 /user/root/output/clusters-4/_logs/history/localhost_1284692116007_job_20100 9171055_0007_root_mahout-examples-0.3.job -rw-r--r-- 1 root supergroup 5757 2010-09-17 11:02 /user/root/output/clusters-4/part-0 drwxr-xr-x - root supergroup 0 2010-09-17 11:02 /user/root/output/clusters-5 drwxr-xr-x - root supergroup 0 2010-09-17 11:02 /user/root/output/clusters-5/_logs drwxr-xr-x - root supergroup 0 2010-09-17 11:02 /user/root/output/clusters-5/_logs/history -rw-r--r-- 1 root supergroup 17481 2010-09-17 11:02
Re: Launching Tasks From JobTracker
hi, Thanks for the reply. The problem i want to solve is, I am writing a scheduler plugin in which, i want to divide a job into tasks and assign to TaskTrackers(Obvious). Then, i want to assign the tasks on to which ever TaskTracker i want and assign it whenever i want. I am using org.apache.hadoop.mapred.TaskScheduler.assignTasks(TaskTrackerStatus taskTrackerStatus) which would return list of Tasks, on every Heart Beat. In the above method, i am calling org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask method to get Task for a job. But everytime it is called, i would only get a null as return. Initially i thought that JobStatus.status is not set to RUNNING. But the problem is not that. Is there any other way that i could assign tasks on to a TaskTracker apart from above method. hoping a solution from you. Thank you Radheshyam Nanduri On Fri, Sep 10, 2010 at 6:46 PM, Hemanth Yamijala yhema...@gmail.comwrote: [Moving to mapreduce-dev, copying common-dev] Hi, On Thu, Sep 9, 2010 at 11:30 AM, radheshyam nanduri radheshyam.nand...@gmail.com wrote: Hi, I am working on writing a scheduler plugin for Hadoop. Currently, the model supported to plug-in schedulers to Hadoop is to extend the TaskScheduler class in o.a.h.mapred package. Primarily what a 'plug-in' scheduler can do is that given a set of jobs and a tasktracker, it can assign one or more suitable tasks to the tasktracker. The scheduler will have flexibility in choosing the job and the tasks it wants to schedule. You can take a look at some of the existing schedulers like CapacityTaskScheduler or FairScheduler to see what they do and how. I have divided the job received into number of tasks. This is already done in the framework when a job is submitted. Are you overriding this ? Can you explain what you are doing in some more detail ? My task now is to assign a task on to a particular TaskTracker. I want to start the Task right away with a method which accepts the Task and TaskTracker as arguments. I am not sure I am following this. A task needs to be sent via Hadoop's RPC mechanisms to a tasktracker where it should be executed. So, conceptually, it is the tasktracker that has an RPC method which accepts tasks to launch. The task is launched typically straight-away, but in case of certain scheduling choices, it could have to wait for a short while to get a free slot to execute. Could you guide me on doing this. It may be better if you can describe what you want the plug-in scheduler to achieve. Thanks Hemanth Thanks in advance. -- Radheshyam Nanduri -- Radheshyam Nanduri
Re: FW: [jira] Moved: (MAPREDUCE-2008) write Junit test cases that run the Hadoop examples
On Sat, Sep 4, 2010 at 10:27 PM, Saikat Kanjilal sxk1...@hotmail.com wrote: HI Folks,I am working on this jira task and had some additional questions:1) It seems that whenever I go to run the test or the test-core target on the mapreduce codebase they both fail, I did a fresh sync prior to running these targets today. Am I missing some other files that are necessary? If it's individual test failures there might already be jiras for those failures, sometimes tests are flaky. 2) If I want to add a target for compiling and running a junit test suite that runs all of the current mapreduce examples should I create a brand new target or try to hack the test target to also run this set of tests, is there a standard when introducing a new set of tests?Looking forward to getting some more help on the above.Regards Looks like this was handled on MR-2008. Thanks, Eli Date: Fri, 13 Aug 2010 01:19:15 -0400 From: j...@apache.org To: sxk1...@hotmail.com Subject: [jira] Moved: (MAPREDUCE-2008) write Junit test cases that run the Hadoop examples [ https://issues.apache.org/jira/browse/MAPREDUCE-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu moved HADOOP-6891 to MAPREDUCE-2008: Project: Hadoop Map/Reduce (was: Hadoop Common) Key: MAPREDUCE-2008 (was: HADOOP-6891) Affects Version/s: 0.20.2 (was: 0.20.2) Fix Version/s: 0.22.0 (was: 0.20.3) Component/s: examples test (was: test) write Junit test cases that run the Hadoop examples --- Key: MAPREDUCE-2008 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2008 Project: Hadoop Map/Reduce Issue Type: New Feature Components: examples, test Affects Versions: 0.20.2 Environment: N/A Reporter: Saikat Kanjilal Priority: Minor Fix For: 0.22.0 Original Estimate: 20h Remaining Estimate: 20h I will be adding junit test cases that run the hadoop examples, specifically I'll be adding a test suite that tests everything under the src/examples/org/apache/hadoop/examples package. Since this is my first contribution to the project I'll need some help to figure out how to submit this -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Page BayAreaHadoopUserGroup deleted from Hadoop Wiki
Does anyone know who SomeOtherAccount is? They've been making good changes to the wiki. It'd be nice to know who this is, rather than keeping it anonymous. Thanks, Doug On 09/17/2010 09:59 AM, Apache Wiki wrote: Dear wiki user, You have subscribed to a wiki page Hadoop Wiki for change notification. The page BayAreaHadoopUserGroup has been deleted by SomeOtherAccount. The comment on this change is: It is wrong, completely out of date, never updated, and replaced with a link to http://www.meetup.com/hadoop/ anyway. http://wiki.apache.org/hadoop/BayAreaHadoopUserGroup
Re: Page BayAreaHadoopUserGroup deleted from Hadoop Wiki
On Sep 17, 2010, at 1:08 PM, Doug Cutting wrote: Does anyone know who SomeOtherAccount is? They've been making good changes to the wiki. It'd be nice to know who this is, rather than keeping it anonymous. *Laugh* I know who it is, but clearly they didn't feel like putting a name in. It isn't that hard to figure out if you look at some of the changes. -- Owen
Re: Paths of evolution for Hadoop metrics
From the HBase perspective, life is complicated by the need to support running on multiple Hadoop versions. Current HBase trunk is really targeted at 0.20-append, but I'm also working to add 0.20-security (or 0.20-append-security) to that. And there have been questions (though I'm not sure a clear decision) on support for 0.21 now that it's released. And obviously I can't predict where things will stand in the 0.22 timeframe. Sounds like the new framework will have some great stuff allowing us to strip out some of the HBase code we've added for metrics (built-on JMX support), which would be great to make use of. But until HBase is targeting only 0.22+ I don't think we'll fully be able to convert to it. So I'd vote for option #2 to allow us to make a phased transition. --gh
RE: [jira] Moved: (MAPREDUCE-2008) write Junit test cases that run the Hadoop examples
Hey Eli,Thanks for responding, I've got all my new junit version 4 tests compiling, however I am observing that some other tests are failing and are flaky, on your point number 2 I am working on MR2008 and hoping to submit a patch soon. Will keep you posted on more details. Date: Fri, 17 Sep 2010 11:26:40 -0700 Subject: Re: FW: [jira] Moved: (MAPREDUCE-2008) write Junit test cases that run the Hadoop examples From: e...@cloudera.com To: common-dev@hadoop.apache.org On Sat, Sep 4, 2010 at 10:27 PM, Saikat Kanjilal sxk1...@hotmail.com wrote: HI Folks,I am working on this jira task and had some additional questions:1) It seems that whenever I go to run the test or the test-core target on the mapreduce codebase they both fail, I did a fresh sync prior to running these targets today. Am I missing some other files that are necessary? If it's individual test failures there might already be jiras for those failures, sometimes tests are flaky. 2) If I want to add a target for compiling and running a junit test suite that runs all of the current mapreduce examples should I create a brand new target or try to hack the test target to also run this set of tests, is there a standard when introducing a new set of tests?Looking forward to getting some more help on the above.Regards Looks like this was handled on MR-2008. Thanks, Eli Date: Fri, 13 Aug 2010 01:19:15 -0400 From: j...@apache.org To: sxk1...@hotmail.com Subject: [jira] Moved: (MAPREDUCE-2008) write Junit test cases that run the Hadoop examples [ https://issues.apache.org/jira/browse/MAPREDUCE-2008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu moved HADOOP-6891 to MAPREDUCE-2008: Project: Hadoop Map/Reduce (was: Hadoop Common) Key: MAPREDUCE-2008 (was: HADOOP-6891) Affects Version/s: 0.20.2 (was: 0.20.2) Fix Version/s: 0.22.0 (was: 0.20.3) Component/s: examples test (was: test) write Junit test cases that run the Hadoop examples --- Key: MAPREDUCE-2008 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2008 Project: Hadoop Map/Reduce Issue Type: New Feature Components: examples, test Affects Versions: 0.20.2 Environment: N/A Reporter: Saikat Kanjilal Priority: Minor Fix For: 0.22.0 Original Estimate: 20h Remaining Estimate: 20h I will be adding junit test cases that run the hadoop examples, specifically I'll be adding a test suite that tests everything under the src/examples/org/apache/hadoop/examples package. Since this is my first contribution to the project I'll need some help to figure out how to submit this -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
