[jira] Commented: (MAPREDUCE-1954) Automate the Test scenario of Execution of jobs by original user and other users with Linux Task Controller
[ https://issues.apache.org/jira/browse/MAPREDUCE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891013#action_12891013 ] Balaji Rajagopalan commented on MAPREDUCE-1954: --- +1 from me for y20 > Automate the Test scenario of Execution of jobs by original user and other > users with Linux Task Controller > --- > > Key: MAPREDUCE-1954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1954 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: TestLinuxTaskControllerOtherUser.patch > > > Execution of streaming script "other user" Permission denied message . Same > user it should pass. > Execution of sleep job by the other user on the cluster. Job should not pass. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1955) Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs to change.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12891012#action_12891012 ] Balaji Rajagopalan commented on MAPREDUCE-1955: --- +1 > Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs > to change. > -- > > Key: MAPREDUCE-1955 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1955 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: JobInProgressAspectaj.patch, MAPREDUCE-1955.patch > > > Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs > to change. > A variable taskInited is changed from Boolean to boolean in > JobInProgress.java. So JobInProgressAspect.aj also needs to change too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1957) [Herriot] Test Job cache directories cleanup after job completes.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1957: - Attachment: 1957-ydist-security.patch > [Herriot] Test Job cache directories cleanup after job completes. > - > > Key: MAPREDUCE-1957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1957 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1957-ydist-security.patch, 1957-ydist-security.patch > > > Test the job cache directories cleanup after job completes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1898) [Herriot] Implement a functionality for getting the job summary information of a job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890976#action_12890976
]
Konstantin Boudnik commented on MAPREDUCE-1898:
---
Patch looks good. The only nit: would it be better to replace string
concatenation in {{JobTracker.getJobSummaryInfo}} with StringBuffer.append()
operations ?
> [Herriot] Implement a functionality for getting the job summary information
> of a job.
> -
>
> Key: MAPREDUCE-1898
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1898
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: test
>Reporter: Vinay Kumar Thota
>Assignee: Vinay Kumar Thota
> Attachments: 1898-ydist-security.patch, 1898-ydist-security.patch,
> 1898-ydist-security.patch, MAPREDUCE-1898.patch
>
>
> Implement a method for getting the job summary details of a job. The job
> summary should be.
> jobId, startTime, launchTime, finishTime, numMaps, numSlotsPerMap,
> numReduces, numSlotsPerReduce, user, queue, status, mapSlotSeconds,
> reduceSlotSeconds, clusterMapCapacity,clusterReduceCapacity.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1854) [herriot] Automate health script system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890972#action_12890972 ] Konstantin Boudnik commented on MAPREDUCE-1854: --- bq. Well, in the trunk tests are separated from the rest of the framework. Thus, their scripts should be with them. So, what about the scripts location? Any comments? > [herriot] Automate health script system test > > > Key: MAPREDUCE-1854 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1854 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: test > Environment: Herriot framework >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan > Attachments: health_script_5.txt, health_script_7.txt, > health_script_trunk.txt, health_script_y20.txt, health_script_y20_1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > 1. There are three scenarios, first is induce a error from health script, > verify that task tracker is blacklisted. > 2. Make the health script timeout and verify the task tracker is blacklisted. > 3. Make an error in the health script path and make sure the task tracker > stays healthy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1899) [Herriot] Test jobsummary information for different jobs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890973#action_12890973 ] Konstantin Boudnik commented on MAPREDUCE-1899: --- Patch's looking good. Make sure it compiles and runs in y20 env. Please generate a version for the trunk. > [Herriot] Test jobsummary information for different jobs. > - > > Key: MAPREDUCE-1899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1899 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1899-ydist-security.patch, 1899-ydist-security.patch > > > Test the following scenarios. > 1. Verify the job summary information for killed job. > 2. Verify the job summary information for failed job. > 3. Verify the job queue information in job summary after job has successfully > completed. > 4. Verify the job summary information for high ram jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1958) using delegation token over hftp for long running clients (part of hdfs 1296)
using delegation token over hftp for long running clients (part of hdfs 1296) - Key: MAPREDUCE-1958 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1958 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1566) Need to add a mechanism to import tokens and secrets into a submitted job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated MAPREDUCE-1566: Attachment: MR-1566.3.patch Added a new test which uses the binary token file. > Need to add a mechanism to import tokens and secrets into a submitted job. > -- > > Key: MAPREDUCE-1566 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1566 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Reporter: Owen O'Malley >Assignee: Owen O'Malley > Fix For: 0.22.0 > > Attachments: mr-1566-1.1.patch, mr-1566-1.patch, MR-1566.1.patch, > MR-1566.2.patch, MR-1566.3.patch > > > We need to include tokens and secrets into a submitted job. I propose adding > a configuration attribute that when pointed at a token storage file will > include the tokens and secrets from that token storage file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1897) trunk build broken on compile-mapred-test
[
https://issues.apache.org/jira/browse/MAPREDUCE-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890938#action_12890938
]
Greg Roelofs commented on MAPREDUCE-1897:
-
bq. Perhaps {{ant -v mvn-install}} will help?
It might. :-) I'll be sure to try that next time around. Thanks.
> trunk build broken on compile-mapred-test
> -
>
> Key: MAPREDUCE-1897
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1897
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.22.0
> Environment: RHEL4 Linux, Java 1.6.0_15-b03
>Reporter: Greg Roelofs
>Assignee: Konstantin Boudnik
>
> ...apparently. Fresh checkout of trunk (all three hadoop-*),
> build.properties project.version fix, ant veryclean mvn-install of common,
> hdfs, and then mapreduce:
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:52:
> cannot access org.apache.hadoop.test.system.DaemonProtocol
> [javac] class file for org.apache.hadoop.test.system.DaemonProtocol not
> found
> [javac] static class FakeJobTracker extends JobTracker {
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
> non-static variable this cannot be referenced from a static context
> [javac] this.trackers = tts;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
> cannot find symbol
> [javac] symbol : variable trackers
> [javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
> [javac] this.trackers = tts;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
> cannot find symbol
> [javac] symbol : method taskTrackers()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] taskTrackers().size() - getBlacklistedTrackerCount(),
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
> cannot find symbol
> [javac] symbol : method getBlacklistedTrackerCount()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] taskTrackers().size() - getBlacklistedTrackerCount(),
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:68:
> cannot find symbol
> [javac] symbol : method getBlacklistedTrackerCount()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] getBlacklistedTrackerCount(), 0, 0, 0, totalSlots/2,
> totalSlots/2,
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:64:
> method does not override or implement a method from a supertype
> [javac] @Override
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
> non-static variable this cannot be referenced from a static context
> [javac] this.totalSlots = totalSlots;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
> cannot find symbol
> [javac] symbol : variable totalSlots
> [javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
> [javac] this.totalSlots = totalSlots;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:91:
> establishFirstContact(org.apache.hadoop.mapred.JobTracker,java.lang.String)
> in org.apache.hadoop.mapred.FakeObjectUtilities cannot be applied to
> (org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker,java.lang.String)
> [javac] FakeObjectUtilities.establishFirstContact(jobTracker,
> s);
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:170:
> cannot find symbol
> [javac] symbol : constructor
> MyFakeJobInProgress(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker)
> [javac] location: class
> org.apache.hadoop.mapred.TestJobInProgress.MyFakeJobInProgress
> [
[jira] Commented: (MAPREDUCE-1897) trunk build broken on compile-mapred-test
[
https://issues.apache.org/jira/browse/MAPREDUCE-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890934#action_12890934
]
Konstantin Boudnik commented on MAPREDUCE-1897:
---
bq. What I think would be stupendously useful is if there were a way to make
ant mvn-install log everything it's doing--not just "I'm about to...
Perhaps {{ant -v mvn-install}} will help?
> trunk build broken on compile-mapred-test
> -
>
> Key: MAPREDUCE-1897
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1897
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.22.0
> Environment: RHEL4 Linux, Java 1.6.0_15-b03
>Reporter: Greg Roelofs
>Assignee: Konstantin Boudnik
>
> ...apparently. Fresh checkout of trunk (all three hadoop-*),
> build.properties project.version fix, ant veryclean mvn-install of common,
> hdfs, and then mapreduce:
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:52:
> cannot access org.apache.hadoop.test.system.DaemonProtocol
> [javac] class file for org.apache.hadoop.test.system.DaemonProtocol not
> found
> [javac] static class FakeJobTracker extends JobTracker {
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
> non-static variable this cannot be referenced from a static context
> [javac] this.trackers = tts;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
> cannot find symbol
> [javac] symbol : variable trackers
> [javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
> [javac] this.trackers = tts;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
> cannot find symbol
> [javac] symbol : method taskTrackers()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] taskTrackers().size() - getBlacklistedTrackerCount(),
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
> cannot find symbol
> [javac] symbol : method getBlacklistedTrackerCount()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] taskTrackers().size() - getBlacklistedTrackerCount(),
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:68:
> cannot find symbol
> [javac] symbol : method getBlacklistedTrackerCount()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] getBlacklistedTrackerCount(), 0, 0, 0, totalSlots/2,
> totalSlots/2,
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:64:
> method does not override or implement a method from a supertype
> [javac] @Override
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
> non-static variable this cannot be referenced from a static context
> [javac] this.totalSlots = totalSlots;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
> cannot find symbol
> [javac] symbol : variable totalSlots
> [javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
> [javac] this.totalSlots = totalSlots;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:91:
> establishFirstContact(org.apache.hadoop.mapred.JobTracker,java.lang.String)
> in org.apache.hadoop.mapred.FakeObjectUtilities cannot be applied to
> (org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker,java.lang.String)
> [javac] FakeObjectUtilities.establishFirstContact(jobTracker,
> s);
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:170:
> cannot find symbol
> [javac] symbol : constructor
> MyFakeJobInProgress(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker)
> [javac] l
[jira] Resolved: (MAPREDUCE-1956) allow reducer to initialize lazily
[ https://issues.apache.org/jira/browse/MAPREDUCE-1956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy resolved MAPREDUCE-1956. -- Resolution: Invalid The part about 'loading data in reduce method controlled by an initialize flag variable which assures that it is loaded only once' makes no sense to me. However, you can 'slowstart' reduces by ensuring sufficient maps are complete before _any_ reduces are launched... from mapred-default.xml: mapred.reduce.slowstart.completed.maps 0.05 Fraction of the number of maps in the job which should be complete before reduces are scheduled for the job. > allow reducer to initialize lazily > -- > > Key: MAPREDUCE-1956 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1956 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tasktracker >Affects Versions: 0.20.2 >Reporter: Ted Yu > > From http://www.scribd.com/doc/23046928/Hadoop-Performance-Tuning: > "In M/R job Reducers are initialized with Mappers at the job initialization, > but the reduce method is called in reduce phase when all the maps had been > finished. So in large jobs where Reducer loads data (>100 MB for business > logic) in-memory on initialization, the performance can be increased by > lazily initializing Reducers i.e. loading data in reduce method controlled by > an initialize flag variable which assures that it is loaded only once. By > lazily initializing Reducers which require memory (for business logic) on > initialization, number of maps can be increased." > Introducing a parameter for this purpose would allow more people to utilize > the above pattern. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1897) trunk build broken on compile-mapred-test
[
https://issues.apache.org/jira/browse/MAPREDUCE-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890932#action_12890932
]
Greg Roelofs commented on MAPREDUCE-1897:
-
I've been working on the internal Yahoo branch for the last couple of weeks,
but I'll be switching back to trunk next week and will retry things then. I
don't have any further info at the moment.
What I think would be stupendously useful is if there were a way to make ant
mvn-install log everything it's doing--not just "I'm about to compile foo" but
literally "I'm cd'ing into this directory", "this is the exact command I'm
running", "here's the complete output from that command", etc. (i.e., what real
makefiles do). Then we could compare your verbose output with mine and maybe
figure out which dependency is broken.
(There are definitely broken dependencies, at least in the Y20 branch; editing
a mapreduce .java file, for example, causes it to be recompiled but not all of
its downstream artifacts to be rebuilt, which can result in unit tests
continuing to use the previous code. I don't understand this build system well
enough to figure out where the breakdown is. Current workaround is to blow
away the entire build directory each time. On trunk I've usually been able to
avoid this problem by manually copying jars from build to ~/.ivy2/cache/...,
but that's annoying, too.)
> trunk build broken on compile-mapred-test
> -
>
> Key: MAPREDUCE-1897
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1897
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.22.0
> Environment: RHEL4 Linux, Java 1.6.0_15-b03
>Reporter: Greg Roelofs
>Assignee: Konstantin Boudnik
>
> ...apparently. Fresh checkout of trunk (all three hadoop-*),
> build.properties project.version fix, ant veryclean mvn-install of common,
> hdfs, and then mapreduce:
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:52:
> cannot access org.apache.hadoop.test.system.DaemonProtocol
> [javac] class file for org.apache.hadoop.test.system.DaemonProtocol not
> found
> [javac] static class FakeJobTracker extends JobTracker {
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
> non-static variable this cannot be referenced from a static context
> [javac] this.trackers = tts;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
> cannot find symbol
> [javac] symbol : variable trackers
> [javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
> [javac] this.trackers = tts;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
> cannot find symbol
> [javac] symbol : method taskTrackers()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] taskTrackers().size() - getBlacklistedTrackerCount(),
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
> cannot find symbol
> [javac] symbol : method getBlacklistedTrackerCount()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] taskTrackers().size() - getBlacklistedTrackerCount(),
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:68:
> cannot find symbol
> [javac] symbol : method getBlacklistedTrackerCount()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] getBlacklistedTrackerCount(), 0, 0, 0, totalSlots/2,
> totalSlots/2,
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:64:
> method does not override or implement a method from a supertype
> [javac] @Override
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
> non-static variable this cannot be referenced from a static context
> [javac] this.totalSlots = totalSlots;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
> cannot find symbo
[jira] Commented: (MAPREDUCE-1733) Authentication between pipes processes and java counterparts.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890902#action_12890902 ] Jitendra Nath Pandey commented on MAPREDUCE-1733: - ant test, javadoc, javac warnings and findbugs were run manually. TestRumenJobTraces failed which is unrelated to this patch. javadoc generates warning in Counter.java which is also unrelated. > Authentication between pipes processes and java counterparts. > - > > Key: MAPREDUCE-1733 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1733 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: 0.22.0 > > Attachments: MR-1733-y20.1.patch, MR-1733-y20.2.patch, > MR-1733-y20.3.patch, MR-1733.5.patch > > > The connection between a pipe process and its parent java process should be > authenticated. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more customizable.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-1936: - Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Fix Version/s: 0.22.0 Resolution: Fixed +1 I committed this. Thanks, Hong! > [gridmix3] Make Gridmix3 more customizable. > --- > > Key: MAPREDUCE-1936 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/gridmix >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.22.0 > > Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, > mr-1936-yhadoop-20.1xx.patch > > > I'd like to make gridmix3 more customizable. Specifically, the proposed > customizations include: > - add (random) location information for each sleep map task. > - make the parameters used in stress submission load throttling configurable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more customizable.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890823#action_12890823
]
Chris Douglas commented on MAPREDUCE-1936:
--
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 5 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{noformat}
> [gridmix3] Make Gridmix3 more customizable.
> ---
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
>Reporter: Hong Tang
>Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch,
> mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed
> customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1856) Extract a subset of tests for smoke (DOA) validation
[ https://issues.apache.org/jira/browse/MAPREDUCE-1856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated MAPREDUCE-1856: --- Hadoop Flags: [Reviewed] +1 > Extract a subset of tests for smoke (DOA) validation > > > Key: MAPREDUCE-1856 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1856 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Affects Versions: 0.21.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: MAPREDUCE-1856.patch, MAPREDUCE-1856.patch, > MAPREDUCE-1856.patch, MAPREDUCE-1856.patch, MAPREDUCE-1856.patch > > > Similar to that of HDFS-1199 for MapReduce. > Adds an ability to run up to 30 minutes of the tests to 'smoke' MapReduce > build i.e. find possible issues faster than the full test cycle does). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1957) [Herriot] Test Job cache directories cleanup after job completes.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-1957: - Attachment: 1957-ydist-security.patch Patch for yahoo dist security branch. > [Herriot] Test Job cache directories cleanup after job completes. > - > > Key: MAPREDUCE-1957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1957 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1957-ydist-security.patch > > > Test the job cache directories cleanup after job completes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1957) [Herriot] Test Job cache directories cleanup after job completes.
[Herriot] Test Job cache directories cleanup after job completes. - Key: MAPREDUCE-1957 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1957 Project: Hadoop Map/Reduce Issue Type: Task Components: test Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Test the job cache directories cleanup after job completes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1912) [Rumen] Add a driver for Rumen tool
[
https://issues.apache.org/jira/browse/MAPREDUCE-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890816#action_12890816
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1912:
---
{noformat}
Distch: A tool to perform distributed chmod
Distcp: A tool to copy data from one hdfs installation to another
{noformat}
The capitalization are incorrect. Please use "distcp" and "distch".
> [Rumen] Add a driver for Rumen tool
>
>
> Key: MAPREDUCE-1912
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1912
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: mapreduce-1912-v1.1.patch
>
>
> Rumen, as a tool, has 2 entry points :
> - Trace builder
> - Folder
> It would be nice to have a single driver program and have 'trace-builder' and
> 'folder' as its options.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1897) trunk build broken on compile-mapred-test
[
https://issues.apache.org/jira/browse/MAPREDUCE-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890810#action_12890810
]
Konstantin Boudnik commented on MAPREDUCE-1897:
---
I haven't seen any issues similar to the reported since last month neither I
heard from anyone about this. I'm going to close it as 'Not reproducible'
unless I hear otherwise.
> trunk build broken on compile-mapred-test
> -
>
> Key: MAPREDUCE-1897
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1897
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.22.0
> Environment: RHEL4 Linux, Java 1.6.0_15-b03
>Reporter: Greg Roelofs
>Assignee: Konstantin Boudnik
>
> ...apparently. Fresh checkout of trunk (all three hadoop-*),
> build.properties project.version fix, ant veryclean mvn-install of common,
> hdfs, and then mapreduce:
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:52:
> cannot access org.apache.hadoop.test.system.DaemonProtocol
> [javac] class file for org.apache.hadoop.test.system.DaemonProtocol not
> found
> [javac] static class FakeJobTracker extends JobTracker {
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
> non-static variable this cannot be referenced from a static context
> [javac] this.trackers = tts;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:60:
> cannot find symbol
> [javac] symbol : variable trackers
> [javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
> [javac] this.trackers = tts;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
> cannot find symbol
> [javac] symbol : method taskTrackers()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] taskTrackers().size() - getBlacklistedTrackerCount(),
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:67:
> cannot find symbol
> [javac] symbol : method getBlacklistedTrackerCount()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] taskTrackers().size() - getBlacklistedTrackerCount(),
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:68:
> cannot find symbol
> [javac] symbol : method getBlacklistedTrackerCount()
> [javac] location: class
> org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker
> [javac] getBlacklistedTrackerCount(), 0, 0, 0, totalSlots/2,
> totalSlots/2,
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:64:
> method does not override or implement a method from a supertype
> [javac] @Override
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
> non-static variable this cannot be referenced from a static context
> [javac] this.totalSlots = totalSlots;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/FakeObjectUtilities.java:73:
> cannot find symbol
> [javac] symbol : variable totalSlots
> [javac] location: class org.apache.hadoop.mapred.FakeObjectUtilities
> [javac] this.totalSlots = totalSlots;
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:91:
> establishFirstContact(org.apache.hadoop.mapred.JobTracker,java.lang.String)
> in org.apache.hadoop.mapred.FakeObjectUtilities cannot be applied to
> (org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker,java.lang.String)
> [javac] FakeObjectUtilities.establishFirstContact(jobTracker,
> s);
> [javac] ^
> [javac]
> /home/roelofs/grid/trunk2/hadoop-mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobInProgress.java:170:
> cannot find symbol
> [javac] symbol : constructor
> MyFakeJobInProgress(org.apache.hadoop.mapred.JobConf,org.apache.hadoop.mapred.FakeObjectUtilities.FakeJobTracker)
> [javac] location: cla
[jira] Updated: (MAPREDUCE-1830) Ivy2.0 has bugs: let's upgrate to 2.1.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-1830: -- Status: Resolved (was: Patch Available) Resolution: Duplicate Fixed already as a part of MAPREDUCE-1870 > Ivy2.0 has bugs: let's upgrate to 2.1.0 > --- > > Key: MAPREDUCE-1830 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1830 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Affects Versions: 0.21.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: mapreduce-1830.patch > > > Similar to HDFS-1177 Ivy needs to be upgraded up to 2.1 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1918) Add documentation to Rumen
[
https://issues.apache.org/jira/browse/MAPREDUCE-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890795#action_12890795
]
Hong Tang commented on MAPREDUCE-1918:
--
I think we should also describe (1) the Json objects are created through
Jackson ObjectMapper from LoggedXXX classes; (2) the API interface how to
build LoggedXXX objects, and how to read them.
The basic API flow for creating parsed rumen object is as follows (user's
responsibility of creating input streams from job conf xml and job history
logs):
- JobConfigurationParser: parser that parses job conf xml. One instance can be
reused to parse many job conf xml files.
{code}
JobConfigurationParser jcp = new
JobConfigurationParser(interestedProperties); // interestedProperties is a a
list of keys to be extracted from the job conf xml file.
Properties parsedProperties = jcp.parse(inputStream); // inputStream is
the file input stream for the job conf xml file.
{code}
- JobHistoryParser: parser that parses job history files. It is an interface
and actual implementations are defined as enums in JobHistoryParserFactory. One
can directly use the version matching the the version of job history logs. Or
she can also use method "canParse()" to detect which parser is suitable for
parsing the job history logs (following the pattern in TraceBuilder). Create
one instance to parse a job history log and close it after use.
{code}
JobHistoryParser parser = new Hadoop20JHParser(inputStream); //
inputStream is the file input stream for the job history file.
// JobHistoryParser APIs will be used later when being fed into
JobBuilder (below).
parser.close();
{code}
- JobBuilder: builder for LoggedJobs. Create one instance to parse the pairing
job history log and job conf. The order of parsing conf file or job history
file is not important.
{code}
JobBuilder jb = new JobBuilder(jobID); // you will need to extract the
job ID from the file name: _job__
jb.process(jcp.parse(jobConfInputStream));
JobHistoryParser parser = new Hadoop20JHParser(jobHistoryInputStream);
try {
HistoryEvent e;
while ((e = parser.nextEvent()) != null) {
jobBuilder.process(e);
}
} finally {
parser.close();
}
LoggedJob job = jb.build();
{code}
>From the reading side, the output produced by TraceBuilder or Folder can be
>read through JobTraceReader or ClusterTopologyReader. One can also use
>Jackson's ObjectMapper to parse the json formatted data into LoggedJob or
>LoggedTopology objects.
> Add documentation to Rumen
> --
>
> Key: MAPREDUCE-1918
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1918
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tools/rumen
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
> Attachments: mapreduce-1918-v1.3.patch, mapreduce-1918-v1.4.patch,
> rumen.pdf, rumen.pdf
>
>
> Add forrest documentation to Rumen tool.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1956) allow reducer to initialize lazily
allow reducer to initialize lazily -- Key: MAPREDUCE-1956 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1956 Project: Hadoop Map/Reduce Issue Type: Improvement Components: tasktracker Affects Versions: 0.20.2 Reporter: Ted Yu >From http://www.scribd.com/doc/23046928/Hadoop-Performance-Tuning: "In M/R job Reducers are initialized with Mappers at the job initialization, but the reduce method is called in reduce phase when all the maps had been finished. So in large jobs where Reducer loads data (>100 MB for business logic) in-memory on initialization, the performance can be increased by lazily initializing Reducers i.e. loading data in reduce method controlled by an initialize flag variable which assures that it is loaded only once. By lazily initializing Reducers which require memory (for business logic) on initialization, number of maps can be increased." Introducing a parameter for this purpose would allow more people to utilize the above pattern. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1925) TestRumenJobTraces fails in trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-1925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890786#action_12890786 ] Hong Tang commented on MAPREDUCE-1925: -- One minor nit: Can we avoid involving InputDemuxer in getRewindableInputStream? It probably suffice to create a FSDataInputStream and create a RewindableInputStream on top of it. Other than that, the patch looks good. > TestRumenJobTraces fails in trunk > - > > Key: MAPREDUCE-1925 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1925 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tools/rumen >Affects Versions: 0.22.0 >Reporter: Amareshwari Sriramadasu >Assignee: Ravi Gummadi > Fix For: 0.22.0 > > Attachments: 1925.patch, 1925.v1.1.patch, 1925.v1.patch, 1925.v2.patch > > > TestRumenJobTraces failed with following error: > Error Message > the gold file contains more text at line 1 expected:<56> but was:<0> > Stacktrace > at > org.apache.hadoop.tools.rumen.TestRumenJobTraces.testHadoop20JHParser(TestRumenJobTraces.java:294) > Full log of the failure is available at > http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/292/testReport/org.apache.hadoop.tools.rumen/TestRumenJobTraces/testHadoop20JHParser/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1854) [herriot] Automate health script system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890721#action_12890721 ] Balaji Rajagopalan commented on MAPREDUCE-1854: --- The info statement I added intentionally it gives me an idea of the status of the task tracker, there is a delay in the while loop for 3 secs, to the log statement does not swamp the test output. > [herriot] Automate health script system test > > > Key: MAPREDUCE-1854 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1854 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: test > Environment: Herriot framework >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan > Attachments: health_script_5.txt, health_script_7.txt, > health_script_trunk.txt, health_script_y20.txt, health_script_y20_1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > 1. There are three scenarios, first is induce a error from health script, > verify that task tracker is blacklisted. > 2. Make the health script timeout and verify the task tracker is blacklisted. > 3. Make an error in the health script path and make sure the task tracker > stays healthy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1854) [herriot] Automate health script system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890719#action_12890719 ] Iyappan Srinivasan commented on MAPREDUCE-1854: --- Remove the LOG.info statement from nodeHealthStatus method, as it is inside a while loop. Rest of the code is Ok for 20.1.xxx. > [herriot] Automate health script system test > > > Key: MAPREDUCE-1854 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1854 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: test > Environment: Herriot framework >Reporter: Balaji Rajagopalan >Assignee: Balaji Rajagopalan > Attachments: health_script_5.txt, health_script_7.txt, > health_script_trunk.txt, health_script_y20.txt, health_script_y20_1.patch > > Original Estimate: 120h > Remaining Estimate: 120h > > 1. There are three scenarios, first is induce a error from health script, > verify that task tracker is blacklisted. > 2. Make the health script timeout and verify the task tracker is blacklisted. > 3. Make an error in the health script path and make sure the task tracker > stays healthy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1955) Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs to change.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890704#action_12890704 ] Iyappan Srinivasan commented on MAPREDUCE-1955: --- Disregard my trunk patch. Consider only the patch for 20.1.xxx. This change is not in trunk. > Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs > to change. > -- > > Key: MAPREDUCE-1955 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1955 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: JobInProgressAspectaj.patch, MAPREDUCE-1955.patch > > > Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs > to change. > A variable taskInited is changed from Boolean to boolean in > JobInProgress.java. So JobInProgressAspect.aj also needs to change too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1955) Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs to change.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iyappan Srinivasan updated MAPREDUCE-1955: -- Attachment: MAPREDUCE-1955.patch patch for trunk > Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs > to change. > -- > > Key: MAPREDUCE-1955 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1955 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: JobInProgressAspectaj.patch, MAPREDUCE-1955.patch > > > Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs > to change. > A variable taskInited is changed from Boolean to boolean in > JobInProgress.java. So JobInProgressAspect.aj also needs to change too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1955) Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs to change.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iyappan Srinivasan updated MAPREDUCE-1955: -- Attachment: JobInProgressAspectaj.patch patch for 20.1.xxx > Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs > to change. > -- > > Key: MAPREDUCE-1955 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1955 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: JobInProgressAspectaj.patch > > > Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs > to change. > A variable taskInited is changed from Boolean to boolean in > JobInProgress.java. So JobInProgressAspect.aj also needs to change too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1955) Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs to change.
Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs to change. -- Key: MAPREDUCE-1955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1955 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Iyappan Srinivasan Assignee: Iyappan Srinivasan Attachments: JobInProgressAspectaj.patch Because of changes in JobInProgress.java, JobInProgressAspect.aj also needs to change. A variable taskInited is changed from Boolean to boolean in JobInProgress.java. So JobInProgressAspect.aj also needs to change too. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1954) Automate the Test scenario of Execution of jobs by original user and other users with Linux Task Controller
[ https://issues.apache.org/jira/browse/MAPREDUCE-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iyappan Srinivasan updated MAPREDUCE-1954: -- Attachment: TestLinuxTaskControllerOtherUser.patch patch for 20.1.xxx > Automate the Test scenario of Execution of jobs by original user and other > users with Linux Task Controller > --- > > Key: MAPREDUCE-1954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1954 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Reporter: Iyappan Srinivasan >Assignee: Iyappan Srinivasan > Attachments: TestLinuxTaskControllerOtherUser.patch > > > Execution of streaming script "other user" Permission denied message . Same > user it should pass. > Execution of sleep job by the other user on the cluster. Job should not pass. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1954) Automate the Test scenario of Execution of jobs by original user and other users with Linux Task Controller
Automate the Test scenario of Execution of jobs by original user and other users with Linux Task Controller --- Key: MAPREDUCE-1954 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1954 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Iyappan Srinivasan Assignee: Iyappan Srinivasan Execution of streaming script "other user" Permission denied message . Same user it should pass. Execution of sleep job by the other user on the cluster. Job should not pass. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1867) Remove unused methods in org.apache.hadoop.streaming.StreamUtil
[ https://issues.apache.org/jira/browse/MAPREDUCE-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1867: --- Attachment: patch-1867-1.txt Patch is updated to trunk. Also moved test related utility functions from StreamUtil to UtilTest.java > Remove unused methods in org.apache.hadoop.streaming.StreamUtil > --- > > Key: MAPREDUCE-1867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 0.22.0 > > Attachments: patch-1867-1.txt, patch-1867.txt > > > There are many unused methods in org.apache.hadoop.streaming.StreamUtil. They > should be removed from the class for maintainability. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1867) Remove unused methods in org.apache.hadoop.streaming.StreamUtil
[ https://issues.apache.org/jira/browse/MAPREDUCE-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1867: --- Status: Patch Available (was: Open) > Remove unused methods in org.apache.hadoop.streaming.StreamUtil > --- > > Key: MAPREDUCE-1867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 0.22.0 > > Attachments: patch-1867-1.txt, patch-1867.txt > > > There are many unused methods in org.apache.hadoop.streaming.StreamUtil. They > should be removed from the class for maintainability. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1918) Add documentation to Rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1918: -- Attachment: rumen.pdf Attaching a modified document incorporating changes from Dick. > Add documentation to Rumen > -- > > Key: MAPREDUCE-1918 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1918 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tools/rumen >Affects Versions: 0.22.0 >Reporter: Amar Kamat >Assignee: Amar Kamat > Fix For: 0.22.0 > > Attachments: mapreduce-1918-v1.3.patch, rumen.pdf, rumen.pdf > > > Add forrest documentation to Rumen tool. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1918) Add documentation to Rumen
[ https://issues.apache.org/jira/browse/MAPREDUCE-1918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1918: -- Attachment: mapreduce-1918-v1.4.patch Attaching a patch for the same. test-patch passed on my box. > Add documentation to Rumen > -- > > Key: MAPREDUCE-1918 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1918 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tools/rumen >Affects Versions: 0.22.0 >Reporter: Amar Kamat >Assignee: Amar Kamat > Fix For: 0.22.0 > > Attachments: mapreduce-1918-v1.3.patch, mapreduce-1918-v1.4.patch, > rumen.pdf, rumen.pdf > > > Add forrest documentation to Rumen tool. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more customizable.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1936: - Status: Open (was: Patch Available) > [gridmix3] Make Gridmix3 more customizable. > --- > > Key: MAPREDUCE-1936 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/gridmix >Reporter: Hong Tang >Assignee: Hong Tang > Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, > mr-1936-yhadoop-20.1xx.patch > > > I'd like to make gridmix3 more customizable. Specifically, the proposed > customizations include: > - add (random) location information for each sleep map task. > - make the parameters used in stress submission load throttling configurable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1899) [Herriot] Test jobsummary information for different jobs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12890620#action_12890620 ] Iyappan Srinivasan commented on MAPREDUCE-1899: --- +1 from me. > [Herriot] Test jobsummary information for different jobs. > - > > Key: MAPREDUCE-1899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1899 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: test >Reporter: Vinay Kumar Thota >Assignee: Vinay Kumar Thota > Attachments: 1899-ydist-security.patch, 1899-ydist-security.patch > > > Test the following scenarios. > 1. Verify the job summary information for killed job. > 2. Verify the job summary information for failed job. > 3. Verify the job queue information in job summary after job has successfully > completed. > 4. Verify the job summary information for high ram jobs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more customizable.
[
https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hong Tang updated MAPREDUCE-1936:
-
Status: Patch Available (was: Open)
New patch that addresses Chris's comments. The usage output now looks like the
following:
{noformat}
Usage: gridmix [-generate ] [-users URI] [-Dname=value ...]
e.g. gridmix -generate 100m foo -
Configuration parameters:
General parameters:
gridmix.output.directory : Output directory
gridmix.client.submit.threads: Submitting threads
gridmix.client.pending.queue.depth : Queued job desc
gridmix.user.resolve.class : User resolution class
gridmix.job.type : Job types
(LOADJOB|SLEEPJOB)
Parameters related to job submission:
gridmix.job-submission.default-queue : Default queue
gridmix.job-submission.use-queue-in-trace: Enable/disable using
queues in trace
gridmix.job-submission.policy: Job submission policy
(REPLAY|STRESS|SERIAL)
Parameters specific for LOADJOB:
gridmix.key.fraction : Key fraction of rec
Parameters specific for SLEEPJOB:
gridmix.sleep.maptask-only : Whether to ignore
reduce tasks
gridmix.sleep.fake-locations : Number of fake
locations for map tasks
gridmix.sleep.max-map-time : Maximum map task
runtime in mili-sec
gridmix.sleep.max-reduce-time: Maximum reduce task
runtime in mili-sec (merge+reduce)
Parameters specific for STRESS submission throttling policy:
gridmix.throttle.jobs-to-tracker-ratio : jobs vs task-tracker
ratio
gridmix.throttle.maps.task-to-slot-ratio : maps vs map-slot ratio
gridmix.throttle.reduces.task-to-slot-ratio : reduces vs
reduce-slot ratio
gridmix.throttle.maps.max-slot-share-per-job : map-slot share per job
gridmix.throttle.reducess.max-slot-share-per-job : reduce-slot share per
job
{noformat}
> [gridmix3] Make Gridmix3 more customizable.
> ---
>
> Key: MAPREDUCE-1936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/gridmix
>Reporter: Hong Tang
>Assignee: Hong Tang
> Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch,
> mr-1936-yhadoop-20.1xx.patch
>
>
> I'd like to make gridmix3 more customizable. Specifically, the proposed
> customizations include:
> - add (random) location information for each sleep map task.
> - make the parameters used in stress submission load throttling configurable.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (MAPREDUCE-1948) [Rumen] Use org.apache.commons.cli.CommandLineParser for parsing Rumen's cli arguments
[ https://issues.apache.org/jira/browse/MAPREDUCE-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi reassigned MAPREDUCE-1948: --- Assignee: Ravi Gummadi > [Rumen] Use org.apache.commons.cli.CommandLineParser for parsing Rumen's cli > arguments > -- > > Key: MAPREDUCE-1948 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1948 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: tools/rumen >Reporter: Amar Kamat >Assignee: Ravi Gummadi > > Instead of writing a custom cli parser, its better that we use > org.apache.commons.cli.CommandLineParser for parsing Rumen's arguments. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1936) [gridmix3] Make Gridmix3 more customizable.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1936: - Attachment: mr-1936-20100720.patch > [gridmix3] Make Gridmix3 more customizable. > --- > > Key: MAPREDUCE-1936 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1936 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/gridmix >Reporter: Hong Tang >Assignee: Hong Tang > Attachments: mr-1936-20100715.patch, mr-1936-20100720.patch, > mr-1936-yhadoop-20.1xx.patch > > > I'd like to make gridmix3 more customizable. Specifically, the proposed > customizations include: > - add (random) location information for each sleep map task. > - make the parameters used in stress submission load throttling configurable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1867) Remove unused methods in org.apache.hadoop.streaming.StreamUtil
[ https://issues.apache.org/jira/browse/MAPREDUCE-1867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1867: --- Status: Open (was: Patch Available) Assignee: Amareshwari Sriramadasu > Remove unused methods in org.apache.hadoop.streaming.StreamUtil > --- > > Key: MAPREDUCE-1867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Reporter: Amareshwari Sriramadasu >Assignee: Amareshwari Sriramadasu >Priority: Minor > Fix For: 0.22.0 > > Attachments: patch-1867.txt > > > There are many unused methods in org.apache.hadoop.streaming.StreamUtil. They > should be removed from the class for maintainability. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
