Hadoop-Mapreduce-trunk-Commit - Build # 1647 - Failure
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1647/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14372 lines...] [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat run-commit-test: [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/data [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/logs [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/extraconf [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/extraconf [junit] Running org.apache.hadoop.mapred.TestJobHistoryParsing [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.117 sec [junit] Running org.apache.hadoop.mapred.TestJobInProgress [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.248 sec [junit] Running org.apache.hadoop.mapred.TestJobQueueTaskScheduler [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.216 sec [junit] Running org.apache.hadoop.mapred.TestKillCompletedJob [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.519 sec [junit] Running org.apache.hadoop.mapred.TestLostTracker [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.376 sec [junit] Running org.apache.hadoop.mapred.TestQueueAclsForCurrentUser [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.673 sec [junit] Running org.apache.hadoop.mapred.TestRackAwareTaskPlacement [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.834 sec [junit] Running org.apache.hadoop.mapred.TestSeveral [junit] Tests run: 0, Failures: 0, Errors: 0, Time elapsed: 0.032 sec [junit] Running org.apache.hadoop.mapred.TestSpeculativeExecution [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 4.368 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 4.859 sec [junit] Running org.apache.hadoop.mapred.TestTaskTrackerBlacklisting [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 2.066 sec [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 117.369 sec [junit] Test org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs FAILED [junit] Running org.apache.hadoop.mapreduce.TestMapReduceLocal [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 102.511 sec [junit] Test org.apache.hadoop.mapreduce.TestMapReduceLocal FAILED checkfailure: [touch] Creating /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/testsfailed BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:778: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:741: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:802: Tests failed! Total time: 5
Re: Building Hadoop 0.21.0 on Eclipse
Pavlos, 0.21 was abandoned and you'd hardly find support for it from the major part of the community. What are you looking to build it for? Perhaps you wanted to use the trunk or the 0.23 branch? On Fri, Jan 20, 2012 at 5:15 AM, Pavlos Mitsoulis - Ntompos wrote: > Hello, > > I am trying to compile hadoop 0.21.0 on eclipse but I get some errors. In > particular, the Java classes in package > org.apache.hadoop.mapreduce.jobhistory complain that some classes does not > exist. For instance, in the class EventReader.java I get the following error: > "Event cannot be resolved". > > Thanks, > Pavlos -- Harsh J Customer Ops. Engineer Cloudera | http://tiny.cloudera.com/about
Re: regarding to sort and reducer
Samaneh, Sorry for the late response. Inline, some of what I can offer. On Sat, Jan 21, 2012 at 10:51 PM, Samaneh Shokuhi wrote: > Hi All, Welcome! > I am very new to hadoop and going to do some research on it regarding to my > master thesis. First of all What i want to do is to know the fuctionality > of sort and shuffle and to run an applition while hadoop included and not > included sort part. > I need to know which class in hadoop is taking care of sort ? Are you looking for the sort mechanism or the algorithm? This is an excellent presentation on the MR sort/shuffle/merge layers that I recommend reading: http://www.slideshare.net/hadoopusergroup/ordered-record-collection > Another thing i need to know is the functianlity of reducer and to find out > the possibility of sending message from one reducer to another one and > doing kind of work stealing between reducers. You probably want to read ReduceTask class, but this functionality is not present today. Perhaps easier to do with the new MR2 framework, detailed in http://developer.yahoo.com/blogs/hadoop/posts/2011/03/mapreduce-nextgen-scheduler/ > Since i am very new to hadoop and it has alot of modules ,i need to know > which project should i look at it. > Also i ll appriciate you to let me know if you have any comment on this > idea. You need to look at the hadoop-mapreduce-project in trunk for all things MR today. It also uses some generic components from the hadoop-common project. See http://wiki.apache.org/hadoop/HowToContribute for more details. Please feel free to mail the lists with any specific questions you have as you go ahead! -- Harsh J Customer Ops. Engineer Cloudera | http://tiny.cloudera.com/about
Hadoop-Mapreduce-0.23-Build - Build # 178 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/178/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 27529 lines...] [INFO] Executed tasks [INFO] [INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ hadoop-mapreduce --- [INFO] [INFO] --- maven-antrun-plugin:1.6:run (tar) @ hadoop-mapreduce --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-install-plugin:2.3.1:install (default-install) @ hadoop-mapreduce --- [INFO] Installing /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/pom.xml to /home/jenkins/.m2/repository/org/apache/hadoop/hadoop-mapreduce/0.23.1-SNAPSHOT/hadoop-mapreduce-0.23.1-SNAPSHOT.pom [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-yarn ... SUCCESS [1.898s] [INFO] hadoop-yarn-api ... SUCCESS [8.653s] [INFO] hadoop-yarn-common SUCCESS [22.728s] [INFO] hadoop-yarn-server SUCCESS [0.108s] [INFO] hadoop-yarn-server-common . SUCCESS [4.072s] [INFO] hadoop-yarn-server-nodemanager SUCCESS [2:01.659s] [INFO] hadoop-yarn-server-web-proxy .. SUCCESS [1.255s] [INFO] hadoop-yarn-server-resourcemanager SUCCESS [2:51.747s] [INFO] hadoop-yarn-server-tests .. SUCCESS [39.707s] [INFO] hadoop-mapreduce-client ... SUCCESS [0.060s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [7.137s] [INFO] hadoop-yarn-applications .. SUCCESS [0.140s] [INFO] hadoop-yarn-applications-distributedshell . SUCCESS [23.546s] [INFO] hadoop-yarn-site .. SUCCESS [0.148s] [INFO] hadoop-mapreduce-client-common SUCCESS [29.139s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [1.043s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [3:48.027s] [INFO] hadoop-mapreduce-client-hs SUCCESS [1:04.282s] [INFO] hadoop-mapreduce-client-jobclient . SUCCESS [42:09.671s] [INFO] Apache Hadoop MapReduce Examples .. SUCCESS [43.237s] [INFO] hadoop-mapreduce .. SUCCESS [0.082s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 54:59.019s [INFO] Finished at: Wed Feb 01 14:09:12 UTC 2012 [INFO] Final Memory: 72M/294M [INFO] + [[ 2 -ne 0 ]] + echo 'Build failed' Build failed + exit 1 Build step 'Execute shell' marked build as failure Archiving artifacts Updating MAPREDUCE-3749 Updating HDFS-2827 Updating MAPREDUCE-3756 Updating HDFS-2864 Updating HDFS-2835 Updating MAPREDUCE-3754 Updating HADOOP-8006 Updating MAPREDUCE-3762 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
Re: Building Hadoop 0.21.0 on Eclipse
Hi Harsh, So by trunk, do you mean 1.0.0 version? Thanks, Ethan On Wed, Feb 1, 2012 at 7:34 AM, Harsh J wrote: > Pavlos, > > 0.21 was abandoned and you'd hardly find support for it from the major > part of the community. What are you looking to build it for? Perhaps > you wanted to use the trunk or the 0.23 branch? > > On Fri, Jan 20, 2012 at 5:15 AM, Pavlos Mitsoulis - Ntompos > wrote: > > Hello, > > > > I am trying to compile hadoop 0.21.0 on eclipse but I get some errors. > In particular, the Java classes in package > org.apache.hadoop.mapreduce.jobhistory complain that some classes does not > exist. For instance, in the class EventReader.java I get the following > error: "Event cannot be resolved". > > > > Thanks, > > Pavlos > > > > -- > Harsh J > Customer Ops. Engineer > Cloudera | http://tiny.cloudera.com/about >
Re: Building Hadoop 0.21.0 on Eclipse
Brisk, No, trunk is currently numbered 0.24. The branch-1 is a number revisioning over the earlier branch-0.20-security stable line, and is not trunk. Check out this blog post on the Bigtop project: https://blogs.apache.org/bigtop/entry/all_you_wanted_to_know On Wed, Feb 1, 2012 at 11:31 PM, brisk wrote: > Hi Harsh, > > So by trunk, do you mean 1.0.0 version? > > Thanks, > Ethan > > On Wed, Feb 1, 2012 at 7:34 AM, Harsh J wrote: > >> Pavlos, >> >> 0.21 was abandoned and you'd hardly find support for it from the major >> part of the community. What are you looking to build it for? Perhaps >> you wanted to use the trunk or the 0.23 branch? >> >> On Fri, Jan 20, 2012 at 5:15 AM, Pavlos Mitsoulis - Ntompos >> wrote: >> > Hello, >> > >> > I am trying to compile hadoop 0.21.0 on eclipse but I get some errors. >> In particular, the Java classes in package >> org.apache.hadoop.mapreduce.jobhistory complain that some classes does not >> exist. For instance, in the class EventReader.java I get the following >> error: "Event cannot be resolved". >> > >> > Thanks, >> > Pavlos >> >> >> >> -- >> Harsh J >> Customer Ops. Engineer >> Cloudera | http://tiny.cloudera.com/about >> -- Harsh J Customer Ops. Engineer Cloudera | http://tiny.cloudera.com/about
Hadoop-Mapreduce-trunk-Commit - Build # 1649 - Failure
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1649/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14483 lines...] [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat run-commit-test: [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/data [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/logs [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/extraconf [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/extraconf [junit] Running org.apache.hadoop.mapred.TestJobHistoryParsing [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.75 sec [junit] Running org.apache.hadoop.mapred.TestJobInProgress [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.318 sec [junit] Running org.apache.hadoop.mapred.TestJobQueueTaskScheduler [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.203 sec [junit] Running org.apache.hadoop.mapred.TestKillCompletedJob [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.419 sec [junit] Running org.apache.hadoop.mapred.TestLostTracker [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.411 sec [junit] Running org.apache.hadoop.mapred.TestQueueAclsForCurrentUser [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.661 sec [junit] Running org.apache.hadoop.mapred.TestRackAwareTaskPlacement [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.843 sec [junit] Running org.apache.hadoop.mapred.TestSeveral [junit] Tests run: 0, Failures: 0, Errors: 0, Time elapsed: 0.033 sec [junit] Running org.apache.hadoop.mapred.TestSpeculativeExecution [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 4.255 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 5.004 sec [junit] Running org.apache.hadoop.mapred.TestTaskTrackerBlacklisting [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 2.18 sec [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 117.369 sec [junit] Test org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs FAILED [junit] Running org.apache.hadoop.mapreduce.TestMapReduceLocal [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 97.41 sec [junit] Test org.apache.hadoop.mapreduce.TestMapReduceLocal FAILED checkfailure: [touch] Creating /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/testsfailed BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:778: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:741: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:802: Tests failed! Total time: 5 min
[jira] [Created] (MAPREDUCE-3779) Create hard and soft limits for job counters
Create hard and soft limits for job counters Key: MAPREDUCE-3779 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3779 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker, tasktracker Affects Versions: 1.0.0, 0.23.0 Reporter: Dave Shine Priority: Minor The mapreduce.job.counters.limit is not overridable at the job level. While it is necessary to limit the number of counters to reduce overhead, there are times when exceeding the limit is required. Currently, the only solution is to increase the limit cluster wide. I would like to see a soft limit set in the mapred-site.xml that can be overridden at the job level, in addition to the hard limit that exists today. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3780) RM assigns containers to killed applications
RM assigns containers to killed applications Key: MAPREDUCE-3780 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3780 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 RM attempts to assign containers to killed applications. The applications were killed when they were inactive and waiting for AM allocation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3781) Fix history for apps which were terminated before the AM launch
Fix history for apps which were terminated before the AM launch --- Key: MAPREDUCE-3781 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3781 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Fix For: 0.23.1 Currently the history for applications which were terminated/killed/failed before the AM was launched redirects to a page that does not exist. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3782) teragen terasort jobs fail when using webhdfs://
teragen terasort jobs fail when using webhdfs:// - Key: MAPREDUCE-3782 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta When running a teragen job with a webhdfs:// url the delegation token that is retrieved is an hdfs delegation token. And the subsequent terasort job on the output fails with java io exception -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3783) Fix "queue -list" to display the correct capacity of queues
Fix "queue -list" to display the correct capacity of queues --- Key: MAPREDUCE-3783 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3783 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor Occasionally, the capacity of the queue as displayed by "queue -list" has incorrect values. For e.g: yarn.scheduler.capacity.root.queues=a,b yarn.scheduler.capacity.root.b.queues=b1,b2,b3 yarn.scheduler.capacity.root.b.b1.capacity=30 yarn.scheduler.capacity.root.b.b2.capacity=30 yarn.scheduler.capacity.root.b.b3.capacity=40 $ mapred queue -list Queue Name : b Queue State : running Scheduling Info : Capacity: 40.0, MaximumCapacity: 1.0, CurrentCapacity: 0.0 == Queue Name : b1 Queue State : running Scheduling Info : Capacity: *30.02*, MaximumCapacity: 1.0, CurrentCapacity: 0.0 == Queue Name : b2 Queue State : running Scheduling Info : Capacity: *30.02*, MaximumCapacity: 1.0, CurrentCapacity: 0.0 ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3784) maxActiveApplications(|PerUser) per queue is too low for small clusters
maxActiveApplications(|PerUser) per queue is too low for small clusters --- Key: MAPREDUCE-3784 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3784 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil We ran into this issue while testing on small clusters. On a 7node cluster with 8G per node, for a queue with absolute capacity 30%, user limit 100%, maxActiveApplications and maxActiveApplicationsPerUser is calculated to be 1. This means that even though the queue has 17GB(0.3*8*7), only 1 user can run 1 app at a given time queuing up rest of the apps/users. This hurts performance on small clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3427) streaming tests fail with MR2
[
https://issues.apache.org/jira/browse/MAPREDUCE-3427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy resolved MAPREDUCE-3427.
--
Resolution: Fixed
Fix Version/s: (was: 0.24.0)
I just committed this. Thanks Hitesh!
> streaming tests fail with MR2
> -
>
> Key: MAPREDUCE-3427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3427
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Alejandro Abdelnur
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MR-3427.1.patch
>
>
> After Mavenizing streaming and getting its testcases to use the MiniMRCluster
> wrapper (MAPREDUCE-3169), 4 testcases fail to pass.
> Following is an assessment of those failures. Note that the testcases have
> been tweaked only to set the streaming JAR and yarn as the framework.
>
> (If these issues are unrelated we should create sub-tasks for each one of
> them).
> *TestStreamingCombiner*, fails because returned counters don't match
> assertion. However, counters printed in the test output indicate values that
> would satisfy the assertion. As Tom has indicated it seems MR/YARN are not
> passing back counter information to the client API.
> *TestStreamingBadRecords*, the job is failing with the following exception
> {code}
> Application application_1321575850006_0001 failed 1 times due to AM Container
> for
> appattempt_1321575850006_0001_01 exited with exitCode: 127 due to:
> .Failing this attempt.. Failing the application.
> {code}
> Difficult to troubleshoot because there are not task logs from Mini MR/YARN
> run.
> *TestStreamingStatus* fails in validateTaskStatus() in the following assertion
> {code}
> expected:<[before consuming input > sort]> but was:<[SUCCEEDED]>
> {code}
> *TestUlimit* fails with
> {code}
> org.junit.ComparisonFailure: output is wrong expected:<[786432]> but
> was:<[unlimited]>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop-Mapreduce-0.23-Commit - Build # 478 - Failure
See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/478/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14265 lines...] [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/concat run-commit-test: [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/data [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/logs [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/extraconf [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/extraconf [junit] Running org.apache.hadoop.mapred.TestJobHistoryParsing [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 3.675 sec [junit] Running org.apache.hadoop.mapred.TestJobInProgress [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.188 sec [junit] Running org.apache.hadoop.mapred.TestJobQueueTaskScheduler [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.291 sec [junit] Running org.apache.hadoop.mapred.TestKillCompletedJob [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.491 sec [junit] Running org.apache.hadoop.mapred.TestLostTracker [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.361 sec [junit] Running org.apache.hadoop.mapred.TestQueueAclsForCurrentUser [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.677 sec [junit] Running org.apache.hadoop.mapred.TestRackAwareTaskPlacement [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.825 sec [junit] Running org.apache.hadoop.mapred.TestSeveral [junit] Tests run: 0, Failures: 0, Errors: 0, Time elapsed: 0.033 sec [junit] Running org.apache.hadoop.mapred.TestSpeculativeExecution [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 4.418 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 4.981 sec [junit] Running org.apache.hadoop.mapred.TestTaskTrackerBlacklisting [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 2.059 sec [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 117.394 sec [junit] Test org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs FAILED [junit] Running org.apache.hadoop.mapreduce.TestMapReduceLocal [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 102.525 sec [junit] Test org.apache.hadoop.mapreduce.TestMapReduceLocal FAILED checkfailure: [touch] Creating /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build/test/testsfailed BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build.xml:778: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build.xml:741: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Commit/trunk/hadoop-mapreduce-project/build.xml:802: Tests failed! Total time: 5 minutes 53 seconds Build step 'Execute shell' marked build as failure Recording test results Updating MAPREDUCE-3427 Updating HADOOP-7470 Email was triggered for: Fa
[jira] [Created] (MAPREDUCE-3785) TestBlockRecovery does not compile
TestBlockRecovery does not compile -- Key: MAPREDUCE-3785 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3785 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.1 Reporter: Eli Collins Priority: Blocker Looks like HDFS-2563 introduced a compilation error in TestBlockRecovery. We didn't catch this because of HDFS-2876. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Hadoop-Mapreduce-trunk-Commit - Build # 1653 - Failure
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1653/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 14453 lines...] [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/cache [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/concat run-commit-test: [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/data [mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/logs [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/extraconf [copy] Copying 1 file to /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/extraconf [junit] Running org.apache.hadoop.mapred.TestJobHistoryParsing [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 2.741 sec [junit] Running org.apache.hadoop.mapred.TestJobInProgress [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.243 sec [junit] Running org.apache.hadoop.mapred.TestJobQueueTaskScheduler [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1.264 sec [junit] Running org.apache.hadoop.mapred.TestKillCompletedJob [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.498 sec [junit] Running org.apache.hadoop.mapred.TestLostTracker [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 2.274 sec [junit] Running org.apache.hadoop.mapred.TestQueueAclsForCurrentUser [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 0.665 sec [junit] Running org.apache.hadoop.mapred.TestRackAwareTaskPlacement [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 1.737 sec [junit] Running org.apache.hadoop.mapred.TestSeveral [junit] Tests run: 0, Failures: 0, Errors: 0, Time elapsed: 0.033 sec [junit] Running org.apache.hadoop.mapred.TestSpeculativeExecution [junit] Tests run: 5, Failures: 0, Errors: 0, Time elapsed: 4.31 sec [junit] Running org.apache.hadoop.mapred.TestTaskLimits [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 4.965 sec [junit] Running org.apache.hadoop.mapred.TestTaskTrackerBlacklisting [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 2.06 sec [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Tests run: 1, Failures: 0, Errors: 1, Time elapsed: 117.379 sec [junit] Test org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs FAILED [junit] Running org.apache.hadoop.mapreduce.TestMapReduceLocal [junit] Tests run: 1, Failures: 1, Errors: 0, Time elapsed: 102.543 sec [junit] Test org.apache.hadoop.mapreduce.TestMapReduceLocal FAILED checkfailure: [touch] Creating /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build/test/testsfailed BUILD FAILED /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:778: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:741: The following error occurred while executing this line: /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk-Commit/trunk/hadoop-mapreduce-project/build.xml:802: Tests failed! Total time: 5 minutes 49 seconds Build step 'Execute shell' marked build as failure Recording test results Updating MAPREDUCE-3427 Updating MAPREDUCE-3640 Email w
[jira] [Resolved] (MAPREDUCE-3725) Hadoop 22 hadoop job -list returns user name as NULL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved MAPREDUCE-3725. Resolution: Fixed Fix Version/s: 0.22.1 Hadoop Flags: Reviewed I just committed this. Thank you Mayank. > Hadoop 22 hadoop job -list returns user name as NULL > > > Key: MAPREDUCE-3725 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3725 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.22.1 >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Fix For: 0.22.1 > > Attachments: patch-MAPREDUCE-3725.patch > > > Hadoop 22 hadoop job -list returns user name as NULL -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3786) No checks for misconfigured userlimit
No checks for misconfigured userlimit - Key: MAPREDUCE-3786 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3786 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor Currently, there are no checks being made for misconfigured userLimit (such as negative values/values >100) This can potentially be a problem if the RM comes up with incorrect userLimit values. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3787) [Gridmix] Improve STRESS mode
[Gridmix] Improve STRESS mode - Key: MAPREDUCE-3787 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3787 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Amar Kamat Fix For: 0.23.1, 0.24.0 Gridmix STRESS mode can be improved as follows: 1. The sleep time in JobMonitor can be reduced and/or made configurable 2. Map and reduce load calculation in StressJobFactory can be done in one loop 3. Updating the overload status from the job submitter thread (inline) 4. Optimizations to avoid un-necessary progress check (which inturn would result into delay) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3788) [Gridmix] Investigate if Gridmix can be made YARN aware
[Gridmix] Investigate if Gridmix can be made YARN aware --- Key: MAPREDUCE-3788 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3788 Project: Hadoop Map/Reduce Issue Type: Improvement Components: contrib/gridmix Affects Versions: 0.24.0 Reporter: Amar Kamat Assignee: Ravi Gummadi Fix For: 0.23.1, 0.24.0 Gridmix was written keeping in mind the monolithic JobTracker. Calls to the single JobTracker were throttled to avoid excess load. Also, polling was faster in JobTracker as the job statuses were cached even if the job was complete. In the YARN world, the situation is slightly different. To make Gridmix scalable and really a YARN scale-benchmarking tool, Gridmix should be enhanced. Some directions worth investigating are: 1. Investigate if Gridmix can cache the AM handles and poll the AM directly for map/reduce task progress. 2. Can the job monitor be made multi-threaded? Each thread can poll a bunch of AMs. 3. Check if there are better ways for getting job progress updates and get away with the busy-waiting logic in Gridmix. 4. Can Gridmix be made container aware. The definition of cluster load should be container aware. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
