[jira] Commented: (MAPREDUCE-1342) Potential JT deadlock in faulty TT tracking
[ https://issues.apache.org/jira/browse/MAPREDUCE-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796571#action_12796571 ] Jothi Padmanabhan commented on MAPREDUCE-1342: -- Prior to the patch, all accesses to FaultInfo objects were synchronized on potentiallyFaultyTrackers and so sort of internally synchronized themselves. However, we are relaxing that restriction with this patch, atleast for isBlackListed and the reasonForBlacklisting. I think we should guard the accesses to these variables or make them volatile, no? > Potential JT deadlock in faulty TT tracking > --- > > Key: MAPREDUCE-1342 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1342 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > Attachments: cycle0.png, mapreduce-1342-1.patch, > mapreduce-1342-2.patch > > > JT$FaultyTrackersInfo.incrementFaults first locks potentiallyFaultyTrackers, > and then calls blackListTracker, which calls removeHostCapacity, which locks > JT.taskTrackers > On the other hand, JT.blacklistedTaskTrackers() locks taskTrackers, then > calls faultyTrackers.isBlacklisted() which goes on to lock > potentiallyFaultyTrackers. > I haven't produced such a deadlock, but the lock ordering here is inverted > and therefore could deadlock. > Not sure if this goes back to 0.21 or just in trunk. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1333) Parallel running tasks on one single node may slow down the performance
[
https://issues.apache.org/jira/browse/MAPREDUCE-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796570#action_12796570
]
Zhaoning Zhang commented on MAPREDUCE-1333:
---
I think it's a general fall back though I run sort benchmark on my cluster
with 20 nodes.
I set the mapred.tasktracker.{map|reduce}.tasks.maximum = 1 individually. In
a single job, map task running simultaneously with a reduce task will be
more slower than the solo one. And for the inter-dependency of the tasks,
shuffles in the reduce tasks will waiting for the maps and the response time
of the job will increase.
2010/1/5 Hong Tang (JIRA)
--
Thank you!
谢谢!
张钊宁
zzningxp
> Parallel running tasks on one single node may slow down the performance
> ---
>
> Key: MAPREDUCE-1333
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1333
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, task, tasktracker
>Affects Versions: 0.20.1
>Reporter: Zhaoning Zhang
>
> When I analysis running tasks performance, I found that parallel running
> tasks on one single node will not be better performance than the serialized
> ones.
> We can set mapred.tasktracker.{map|reduce}.tasks.maximum = 1 individually,
> but there will be parallel map AND reduce tasks.
> And I wonder it's true in the real commercial clusters?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796556#action_12796556 ] Kay Kay commented on MAPREDUCE-1352: HDFS-869 is in now and republished. @Giri: Can we help get this patch applied on branch-0.21 too . Also the most recently build on hudson for 0.21 - http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-21-Build/51/testReport/org.apache.hadoop.cli/TestMRCLI/testAll/ fails with the same dynamic linking error that we are facing in hbase currently. Can you help commit the patch and get a new build to see if this still persists. Thanks. > 0.21.0 - snapshot incorrect dependency published in .pom files > --- > > Key: MAPREDUCE-1352 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.21.0 >Reporter: Kay Kay >Assignee: Giridharan Kesavan >Priority: Critical > Fix For: 0.21.0 > > Attachments: mapreduce-1352.patch > > > The snapshot available here at - > https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ > , has an incorrect dependency specified for hadoop-core ( in the .pom file > ). > The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in > ivy/libaries.properties but pom.xml published in the repository refers to > 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Attachment: patch-1186-4.txt Patch with suggested proposal and incorporating review comments. > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1186-1.txt, patch-1186-2.txt, > patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, > patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-1186: --- Status: Patch Available (was: Open) > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1186-1.txt, patch-1186-2.txt, > patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, > patch-1186-4.txt, patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1155) Streaming tests swallow exceptions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1155: - Fix Version/s: 0.22.0 > Streaming tests swallow exceptions > -- > > Key: MAPREDUCE-1155 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1155 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Fix For: 0.22.0 > > Attachments: mapreduce-1155.patch, mapreduce-1155.patch, > mapreduce-1155.txt, mapreduce-1155.txt > > > Many of the streaming tests (including TestMultipleArchiveFiles) catch > exceptions and print their stack trace rather than failing the job. This > means that tests do not fail even when the job fails. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1155) Streaming tests swallow exceptions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1155: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Todd! > Streaming tests swallow exceptions > -- > > Key: MAPREDUCE-1155 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1155 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: mapreduce-1155.patch, mapreduce-1155.patch, > mapreduce-1155.txt, mapreduce-1155.txt > > > Many of the streaming tests (including TestMultipleArchiveFiles) catch > exceptions and print their stack trace rather than failing the job. This > means that tests do not fail even when the job fails. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796531#action_12796531 ] Hemanth Yamijala commented on MAPREDUCE-1186: - I originally thought if 2a and 2b should be the same, except that ownership would be different if the DefaultTaskController is used. However, since we do not enforce exclusive group ownership for the TT in the DefaultTaskController case, we might end up opening the group permissions on localized files and make it less private than Amarsri's proposal. So, I am +1 for the proposal above. > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1186-1.txt, patch-1186-2.txt, > patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, > patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-899) When using LinuxTaskController, localized files may become accessible to unintended users if permissions are misconfigured.
[ https://issues.apache.org/jira/browse/MAPREDUCE-899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amareshwari Sriramadasu updated MAPREDUCE-899: -- Status: Open (was: Patch Available) > When using LinuxTaskController, localized files may become accessible to > unintended users if permissions are misconfigured. > --- > > Key: MAPREDUCE-899 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-899 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Vinod K V >Assignee: Vinod K V > Attachments: MAPREDUCE-899-20090828.txt, patch-899.txt > > > To enforce the accessibility of job files to only the job-owner and the > TaskTracker, as per MAPREDUCE-842, it is _trusted_ that the setuid/setgid > linux TaskController binary is group owned by a _special group_ to which only > TaskTracker belongs and not just any group to which TT belongs. If the trust > is broken, possibly due to misconfiguration by admins, the local files become > accessible to unintended users, yet giving false sense of security to the > admins. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1218) Collecting cpu and memory usage for TaskTrackers
[ https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796501#action_12796501 ] Scott Chen commented on MAPREDUCE-1218: --- Zheng provides some suggestions on this patch. He said we should move the calculation of the CPU usage inside LinuxCalculationPlugin. I will change the patch and uploaded again. > Collecting cpu and memory usage for TaskTrackers > > > Key: MAPREDUCE-1218 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 0.22.0 > Environment: linux >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1218-rename.sh, MAPREDUCE-1218-v2.patch, > MAPREDUCE-1218-v3.patch, MAPREDUCE-1218.patch > > > The information can be used for resource aware scheduling. > Note that this is related to MAPREDUCE-220. There the per task resource > information is collected. > This one collects the per machine information. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1354) Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
[ https://issues.apache.org/jira/browse/MAPREDUCE-1354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796483#action_12796483 ] Devaraj Das commented on MAPREDUCE-1354: bq. We should see if we can avoid taking the lock altogether. This may require major changes.. Maybe we should just make the locking more granular within the method for the time being. > Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS > accesses > - > > Key: MAPREDUCE-1354 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1354 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Devaraj Das > > It'd be nice to have the JobTracker object not be locked while accessing the > HDFS for reading the jobconf file and while writing the jobinfo file in the > submitJob method. We should see if we can avoid taking the lock altogether. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1354) Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses
Refactor JobTracker.submitJob to not lock the JobTracker during the HDFS accesses - Key: MAPREDUCE-1354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1354 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Devaraj Das It'd be nice to have the JobTracker object not be locked while accessing the HDFS for reading the jobconf file and while writing the jobinfo file in the submitJob method. We should see if we can avoid taking the lock altogether. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1166) SerialUtils.cc: dynamic allocation of arrays based on runtime variable is not portable
[
https://issues.apache.org/jira/browse/MAPREDUCE-1166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796473#action_12796473
]
Allen Wittenauer commented on MAPREDUCE-1166:
-
So I spent some time looking at this today and trying to get this to work with
VLAs in SunStudio. Doing research, I ran across this:
http://forums.sun.com/thread.jspa?threadID=5348093
"C99 VLAs were discussed in the C++ Committee for possible inclusion in the
next standard. The unanimous view was that they did not have nice properties
(the word "suck" was bandied about), and that std::vector or other containers
were a superior solution in C++ programming."
Thus, while __func__ works with CC -features=extensions, it doesn't appear that
Sun Studio supports all of the C99 features in their C++ compiler. This makes
me wonder if using __func__ was the right thing to do now.
They also have an interesting discussion about alloca vs. malloc.
> SerialUtils.cc: dynamic allocation of arrays based on runtime variable is not
> portable
> --
>
> Key: MAPREDUCE-1166
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1166
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Allen Wittenauer
>Assignee: Allen Wittenauer
> Attachments: MAPREREDUCE-1166.patch
>
>
> In SerialUtils.cc, the following code appears:
> int len;
> if (b < -120) {
> negative = true;
> len = -120 - b;
> } else {
> negative = false;
> len = -112 - b;
> }
> uint8_t barr[len];
> as far as I'm aware, this is not legal in ANSI C and will be rejected by ANSI
> compliant compilers. Instead, this should be malloc()'d based upon the size
> of len and free()'d later.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796461#action_12796461
]
Devaraj Das commented on MAPREDUCE-1322:
bq. Isnt MAPREDUCE-1139 the right fix?
That fix would address this issue as well but I'd like the staging directory to
be defined in mapred-default.xml anyway.
bq. Error Launching job : chmod: cannot access
`/tmp/hadoop/mapred/staging/dummy44913967/.staging/job_local_0001': No such
file or directory
The attached patch should fix this problem (this happens due to the fact that
the same /tmp/hadoop directory is being read/written by multiple users).
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1333) Parallel running tasks on one single node may slow down the performance
[
https://issues.apache.org/jira/browse/MAPREDUCE-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796446#action_12796446
]
Hong Tang commented on MAPREDUCE-1333:
--
Could you provide more details? hardware spec, your benchmark program, and what
do you observe as the bottleneck resource?
> Parallel running tasks on one single node may slow down the performance
> ---
>
> Key: MAPREDUCE-1333
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1333
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker, task, tasktracker
>Affects Versions: 0.20.1
>Reporter: Zhaoning Zhang
>
> When I analysis running tasks performance, I found that parallel running
> tasks on one single node will not be better performance than the serialized
> ones.
> We can set mapred.tasktracker.{map|reduce}.tasks.maximum = 1 individually,
> but there will be parallel map AND reduce tasks.
> And I wonder it's true in the real commercial clusters?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1212) Mapreduce contrib project ivy dependencies are not included in binary target
[
https://issues.apache.org/jira/browse/MAPREDUCE-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom White updated MAPREDUCE-1212:
-
Status: Open (was: Patch Available)
Moving to open, pending discussion on HADOOP-6370.
> Mapreduce contrib project ivy dependencies are not included in binary target
>
>
> Key: MAPREDUCE-1212
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1212
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Reporter: Aaron Kimball
>Assignee: Aaron Kimball
>Priority: Critical
> Attachments: MAPREDUCE-1212.patch
>
>
> As in HADOOP-6370, only Hadoop's own library dependencies are promoted to
> ${build.dir}/lib; any libraries required by contribs are not redistributed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1268) Update streaming tests to JUnit 4 style
[ https://issues.apache.org/jira/browse/MAPREDUCE-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved MAPREDUCE-1268. Resolution: Duplicate Went ahead and did this within the original JIRA MAPREDUCE-1155 > Update streaming tests to JUnit 4 style > --- > > Key: MAPREDUCE-1268 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1268 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/streaming, test >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > > Suggested by Chris in MAPREDUCE-1155 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1311) TestStreamingExitStatus fails on hudson patch builds
[
https://issues.apache.org/jira/browse/MAPREDUCE-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved MAPREDUCE-1311.
Resolution: Duplicate
MAPREDUCE-1155 fixes this, resolving dup
> TestStreamingExitStatus fails on hudson patch builds
>
>
> Key: MAPREDUCE-1311
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1311
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Amareshwari Sriramadasu
>
> TestStreamingExitStatus fails on hudson patch builds. The logs have the
> following error :
> {noformat}
> 09/12/16 20:30:58 INFO fs.FSInputChecker: Found checksum error: b[0,
> 6]=68656c6c6f0a
> org.apache.hadoop.fs.ChecksumException: Checksum error:
> file:/grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h3.grid.sp2.yahoo.net/trunk/build/contrib/streaming/test/data/input.txt
> at 0
> at
> org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:278)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:242)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:190)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> at java.io.DataInputStream.read(DataInputStream.java:83)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:180)
> at
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:206)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:191)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:376)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:257)
> 09/12/16 20:30:58 INFO streaming.PipeMapRed: MRErrorThread done
> {noformat}
> The same passes on my local machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (MAPREDUCE-1312) TestStreamingKeyValue fails on hudson patch builds
[ https://issues.apache.org/jira/browse/MAPREDUCE-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved MAPREDUCE-1312. Resolution: Duplicate This is fixed in MAPREDUCE-1155 > TestStreamingKeyValue fails on hudson patch builds > -- > > Key: MAPREDUCE-1312 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1312 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Reporter: Amareshwari Sriramadasu > > TestStreamingKeyValue fails on hudson patch builds with FileNotFoundException. > The failure log from one of the builds is @ > http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/203/testReport/org.apache.hadoop.streaming/TestStreamingKeyValue/testCommandLine/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1155) Streaming tests swallow exceptions
[ https://issues.apache.org/jira/browse/MAPREDUCE-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796400#action_12796400 ] Kay Kay commented on MAPREDUCE-1155: Are we waiting for more code review before this goes into the tree. If this makes the build green- when can we get this committed. thanks. > Streaming tests swallow exceptions > -- > > Key: MAPREDUCE-1155 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1155 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 0.20.1, 0.21.0, 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Minor > Attachments: mapreduce-1155.patch, mapreduce-1155.patch, > mapreduce-1155.txt, mapreduce-1155.txt > > > Many of the streaming tests (including TestMultipleArchiveFiles) catch > exceptions and print their stack trace rather than failing the job. This > means that tests do not fail even when the job fails. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796401#action_12796401 ] Hadoop QA commented on MAPREDUCE-1317: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12428801/mapreduce-1317-20091223.patch against trunk revision 894964. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/354/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/354/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/354/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/354/console This message is automatically generated. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796329#action_12796329 ] Kay Kay commented on MAPREDUCE-1352: Are we looking at the trunk ( 0.22, I guess ) or 0.21 branch. Since the 0.21 branch seems to have a quite a lot of failures - http://hudson.zones.apache.org/hudson/view/Hadoop/job/Hadoop-Mapreduce-21-Build/ . That might be because - mapreduce-0.21 , depends on hdfs-0.21 , that in turn depends on hadoop-core-0.22 by mistake and hence causing strange dynamic link issues. So - we can have this as a dependency on HDFS-869 , get a green build on hdfs trunk with revised .pom and then retry this again. > 0.21.0 - snapshot incorrect dependency published in .pom files > --- > > Key: MAPREDUCE-1352 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.21.0 >Reporter: Kay Kay >Assignee: Giridharan Kesavan >Priority: Critical > Fix For: 0.21.0 > > Attachments: mapreduce-1352.patch > > > The snapshot available here at - > https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ > , has an incorrect dependency specified for hadoop-core ( in the .pom file > ). > The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in > ivy/libaries.properties but pom.xml published in the repository refers to > 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796327#action_12796327 ] stack commented on MAPREDUCE-1352: -- Now I see that there were test failures too. 3 look to have been present before application of this patch. This failure might be related to this change: org.apache.hadoop.mapreduce.TestMRJobClient.testMissingProfileOutput . > 0.21.0 - snapshot incorrect dependency published in .pom files > --- > > Key: MAPREDUCE-1352 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.21.0 >Reporter: Kay Kay >Assignee: Giridharan Kesavan >Priority: Critical > Fix For: 0.21.0 > > Attachments: mapreduce-1352.patch > > > The snapshot available here at - > https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ > , has an incorrect dependency specified for hadoop-core ( in the .pom file > ). > The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in > ivy/libaries.properties but pom.xml published in the repository refers to > 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796318#action_12796318 ] stack commented on MAPREDUCE-1352: -- Unless objection, I'll commit this patch in next day or so (A change in pom can't have associated unit tests so ignoring the -1s above). > 0.21.0 - snapshot incorrect dependency published in .pom files > --- > > Key: MAPREDUCE-1352 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.21.0 >Reporter: Kay Kay >Assignee: Giridharan Kesavan >Priority: Critical > Fix For: 0.21.0 > > Attachments: mapreduce-1352.patch > > > The snapshot available here at - > https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ > , has an incorrect dependency specified for hadoop-core ( in the .pom file > ). > The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in > ivy/libaries.properties but pom.xml published in the repository refers to > 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796320#action_12796320 ] stack commented on MAPREDUCE-1352: -- Or, in case some one else wants to commit it: +1. > 0.21.0 - snapshot incorrect dependency published in .pom files > --- > > Key: MAPREDUCE-1352 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.21.0 >Reporter: Kay Kay >Assignee: Giridharan Kesavan >Priority: Critical > Fix For: 0.21.0 > > Attachments: mapreduce-1352.patch > > > The snapshot available here at - > https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ > , has an incorrect dependency specified for hadoop-core ( in the .pom file > ). > The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in > ivy/libaries.properties but pom.xml published in the repository refers to > 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1312) TestStreamingKeyValue fails on hudson patch builds
[ https://issues.apache.org/jira/browse/MAPREDUCE-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796313#action_12796313 ] Kay Kay commented on MAPREDUCE-1312: MAPREDUCE-1346 addresses the same issue . The failure is due to invalid test fixtures. > TestStreamingKeyValue fails on hudson patch builds > -- > > Key: MAPREDUCE-1312 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1312 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, test >Reporter: Amareshwari Sriramadasu > > TestStreamingKeyValue fails on hudson patch builds with FileNotFoundException. > The failure log from one of the builds is @ > http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/203/testReport/org.apache.hadoop.streaming/TestStreamingKeyValue/testCommandLine/ -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1311) TestStreamingExitStatus fails on hudson patch builds
[
https://issues.apache.org/jira/browse/MAPREDUCE-1311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796314#action_12796314
]
Kay Kay commented on MAPREDUCE-1311:
MAPREDUCE-1346 addresses the same issue but due to invalid test fixtures of
contrib/streaming of not deleting files properly.
> TestStreamingExitStatus fails on hudson patch builds
>
>
> Key: MAPREDUCE-1311
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1311
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Amareshwari Sriramadasu
>
> TestStreamingExitStatus fails on hudson patch builds. The logs have the
> following error :
> {noformat}
> 09/12/16 20:30:58 INFO fs.FSInputChecker: Found checksum error: b[0,
> 6]=68656c6c6f0a
> org.apache.hadoop.fs.ChecksumException: Checksum error:
> file:/grid/0/hudson/hudson-slave/workspace/Mapreduce-Patch-h3.grid.sp2.yahoo.net/trunk/build/contrib/streaming/test/data/input.txt
> at 0
> at
> org.apache.hadoop.fs.FSInputChecker.verifySum(FSInputChecker.java:278)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:242)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:190)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
> at java.io.DataInputStream.read(DataInputStream.java:83)
> at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
> at
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:180)
> at
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:206)
> at
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:191)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
> at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:376)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:257)
> 09/12/16 20:30:58 INFO streaming.PipeMapRed: MRErrorThread done
> {noformat}
> The same passes on my local machine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1317: - Status: Patch Available (was: Open) > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hong Tang updated MAPREDUCE-1317: - Status: Open (was: Patch Available) > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1317) Reducing memory consumption of rumen objects
[ https://issues.apache.org/jira/browse/MAPREDUCE-1317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796288#action_12796288 ] Hong Tang commented on MAPREDUCE-1317: -- The failed tests are not related to the patch. > Reducing memory consumption of rumen objects > > > Key: MAPREDUCE-1317 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1317 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.21.0, 0.22.0 >Reporter: Hong Tang >Assignee: Hong Tang > Fix For: 0.21.0, 0.22.0 > > Attachments: mapreduce-1317-20091218.patch, > mapreduce-1317-20091222-2.patch, mapreduce-1317-20091222.patch, > mapreduce-1317-20091223.patch > > > We have encountered OutOfMemoryErrors in mumak and gridmix when dealing with > very large jobs. The purpose of this jira is to optimze memory consumption of > rumen produced job objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1342) Potential JT deadlock in faulty TT tracking
[ https://issues.apache.org/jira/browse/MAPREDUCE-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796269#action_12796269 ] Konstantin Boudnik commented on MAPREDUCE-1342: --- I wonder if [JSure|http://wiki.apache.org/hadoop/HowToUseConcurrencyAnalysisTools] was able to find the issue? M.b. Chris wants to comment on this? > Potential JT deadlock in faulty TT tracking > --- > > Key: MAPREDUCE-1342 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1342 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > Attachments: cycle0.png, mapreduce-1342-1.patch, > mapreduce-1342-2.patch > > > JT$FaultyTrackersInfo.incrementFaults first locks potentiallyFaultyTrackers, > and then calls blackListTracker, which calls removeHostCapacity, which locks > JT.taskTrackers > On the other hand, JT.blacklistedTaskTrackers() locks taskTrackers, then > calls faultyTrackers.isBlacklisted() which goes on to lock > potentiallyFaultyTrackers. > I haven't produced such a deadlock, but the lock ordering here is inverted > and therefore could deadlock. > Not sure if this goes back to 0.21 or just in trunk. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1346) TestStreamingExitStatus / TestStreamingKeyValue - correct text fixtures in place
[ https://issues.apache.org/jira/browse/MAPREDUCE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796233#action_12796233 ] Kay Kay commented on MAPREDUCE-1346: oops.yes - i meant 'test' fixtures > TestStreamingExitStatus / TestStreamingKeyValue - correct text fixtures in > place > - > > Key: MAPREDUCE-1346 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1346 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: Kay Kay > Fix For: 0.20.2 > > Attachments: MAPREDUCE-1346.patch > > > TestStreamingExitStatus does not have the correct text fixtures , of deleting > the input / output files after the last test run. > Cleanup methods as part of setUp refactored to tearDown to accomplish that. > Because of incorrect text fixtures - subsequent test cases fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1346) TestStreamingExitStatus / TestStreamingKeyValue - correct text fixtures in place
[ https://issues.apache.org/jira/browse/MAPREDUCE-1346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796224#action_12796224 ] Konstantin Boudnik commented on MAPREDUCE-1346: --- I believe it should read 'test fixtures'? > TestStreamingExitStatus / TestStreamingKeyValue - correct text fixtures in > place > - > > Key: MAPREDUCE-1346 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1346 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: Kay Kay > Fix For: 0.20.2 > > Attachments: MAPREDUCE-1346.patch > > > TestStreamingExitStatus does not have the correct text fixtures , of deleting > the input / output files after the last test run. > Cleanup methods as part of setUp refactored to tearDown to accomplish that. > Because of incorrect text fixtures - subsequent test cases fail. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1353) Remove JobInProgress (back?) reference from TaskInProgress
[
https://issues.apache.org/jira/browse/MAPREDUCE-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796205#action_12796205
]
Amar Kamat commented on MAPREDUCE-1353:
---
TaskInProgress requires few getter-like information from JobInProgress which
can be passed as (method) arguments. By doing this we can get rid of the
JobInProgress back-reference in TaskInProgress and lower the effect of a TIP
reference leak atleast (see MAPREDUCE-1316).
Following are the methods where the JobInProgress object is used by
TaskInProgress
- _shouldClose()_ : Calls job.getStatus().getRunState() which can be passed as
a parameter.
- _updateStatus()_ : Requires job.hasSpeculativeMaps()/job.hasSpeculativeMaps()
which can be passed as arguments.
- _updateStatus()_ : Called from JobInProgress. Calls
job.updateStatistics(oldProgRate, currProgRate, isMapTask()) which can be
called by JobInProgress after the call to _updateStatus()_ something like
{code}
float oldProgRate = tip.getOldProgress();
tip.updateStatus(taskstatus, hasSpeculativeMaps(), hasSpeculativeReduces());
float currProgRate = tip.getOldProgress();
updateStatistics(oldProgRate, currProgRate, isMapTask()); // internal call
{code}
- _canBeSpeculated()_ : Requires job.getRunningTaskStatistics(isMapTask()) and
job.getSlowTaskThreshold() which can be passed as an argument.
- _getTaskToRun()_ : Requires job.getNumRestarts() which can be passed as an
argument.
Also TaskInProgress has a _getJob()_ api used only by the JobTracker either
for JobID or for JobInProgress. JobTracker can very well use its own _getJob()_
i.e
{code}
JobID id = tip.getJob().getJobID()
{code}
can be replaced by
{code}
JobID id = tip. getTIPId().getJobID();
{code}
and
{code}
JobInProgress jip = tip.getJob()
{code}
can be replaced by
{code}
JobInProgress jip = getJob(tip.getTIPId().getJobID());
{code}
Thoughts?
> Remove JobInProgress (back?) reference from TaskInProgress
> --
>
> Key: MAPREDUCE-1353
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1353
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
>Affects Versions: 0.22.0
>Reporter: Amar Kamat
>Assignee: Amar Kamat
> Fix For: 0.22.0
>
>
> Looks like TaskInProgress can get rid of JobInProgress back-reference if the
> values it requires from the JobInProgress are passed as parameters.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1353) Remove JobInProgress (back?) reference from TaskInProgress
[ https://issues.apache.org/jira/browse/MAPREDUCE-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat updated MAPREDUCE-1353: -- Affects Version/s: 0.22.0 Fix Version/s: 0.22.0 Assignee: Amar Kamat > Remove JobInProgress (back?) reference from TaskInProgress > -- > > Key: MAPREDUCE-1353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1353 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.22.0 >Reporter: Amar Kamat >Assignee: Amar Kamat > Fix For: 0.22.0 > > > Looks like TaskInProgress can get rid of JobInProgress back-reference if the > values it requires from the JobInProgress are passed as parameters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (MAPREDUCE-1353) Remove JobInProgress (back?) reference from TaskInProgress
Remove JobInProgress (back?) reference from TaskInProgress -- Key: MAPREDUCE-1353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1353 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Reporter: Amar Kamat Looks like TaskInProgress can get rid of JobInProgress back-reference if the values it requires from the JobInProgress are passed as parameters. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796162#action_12796162 ] Hadoop QA commented on MAPREDUCE-1352: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12429322/mapreduce-1352.patch against trunk revision 894964. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/353/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/353/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/353/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/353/console This message is automatically generated. > 0.21.0 - snapshot incorrect dependency published in .pom files > --- > > Key: MAPREDUCE-1352 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.21.0 >Reporter: Kay Kay >Assignee: Giridharan Kesavan >Priority: Critical > Fix For: 0.21.0 > > Attachments: mapreduce-1352.patch > > > The snapshot available here at - > https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ > , has an incorrect dependency specified for hadoop-core ( in the .pom file > ). > The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in > ivy/libaries.properties but pom.xml published in the repository refers to > 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1322) TestStreamingAsDifferentUser fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-1322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796156#action_12796156
]
Amar Kamat commented on MAPREDUCE-1322:
---
Isnt MAPREDUCE-1139 the right fix? Also I see the following error message due
to staging area sharing in testcases
{code}
09/12/22 19:18:10 ERROR streaming.StreamJob:
Error Launching job : chmod: cannot access
`/tmp/hadoop/mapred/staging/dummy44913967/.staging/job_local_0001': No such
file or directory
{code}
Should we fix that here?
> TestStreamingAsDifferentUser fails on trunk
> ---
>
> Key: MAPREDUCE-1322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/streaming, test
>Affects Versions: 0.22.0
>Reporter: Amareshwari Sriramadasu
>Assignee: Devaraj Das
> Fix For: 0.22.0
>
> Attachments: 1322.patch
>
>
> TestStreamingAsDifferentUser fails on trunk with following exception :
> Can not create a Path from a null string
> java.lang.IllegalArgumentException: Can not create a Path from a null string
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:78)
> at org.apache.hadoop.fs.Path.(Path.java:90)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.createHomeAndStagingDirectory(ClusterWithLinuxTaskController.java:158)
> at
> org.apache.hadoop.mapred.ClusterWithLinuxTaskController.startCluster(ClusterWithLinuxTaskController.java:147)
> at
> org.apache.hadoop.streaming.TestStreamingAsDifferentUser.testStreaming(TestStreamingAsDifferentUser.java:49)
> The corresponding line for the exception :
> {code}
> Path stagingArea = new Path(conf.get(JTConfig.JT_STAGING_AREA_ROOT));
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1186) While localizing a DistributedCache file, TT sets permissions recursively on the whole base-dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12796129#action_12796129 ] Amareshwari Sriramadasu commented on MAPREDUCE-1186: With public and private visibilities introduced for distributed cache files(through MAPREDUCE-744), the implementation for setting permissions for localized files changes in the following way: 1. Localized public files can have read and execute permissions for all the users, recursively on localized dir (Current DefaultTaskController's code) 2. Localized private files, a. With DefaultTaskController, can have recursive execute permission on the localized dir (Pre HADOOP-4490 code). b. With LinuxTaskController, owner is the user, group owner is TT, and the permissions are r_xrws___ on the localized dir(Current LinuxTaskController's code). 1 and 2(a) are different, because if the user has not give permissions for others(i.e. private files), I think we should not give permissions for all. Thoughts > While localizing a DistributedCache file, TT sets permissions recursively on > the whole base-dir > --- > > Key: MAPREDUCE-1186 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1186 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.21.0 >Reporter: Vinod K V >Assignee: Amareshwari Sriramadasu > Fix For: 0.22.0 > > Attachments: patch-1186-1.txt, patch-1186-2.txt, > patch-1186-3-ydist.txt, patch-1186-3-ydist.txt, patch-1186-3.txt, > patch-1186-ydist.txt, patch-1186-ydist.txt, patch-1186.txt > > > This is a performance problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1352: -- Status: Patch Available (was: Open) > 0.21.0 - snapshot incorrect dependency published in .pom files > --- > > Key: MAPREDUCE-1352 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.21.0 >Reporter: Kay Kay >Assignee: Giridharan Kesavan >Priority: Critical > Fix For: 0.21.0 > > Attachments: mapreduce-1352.patch > > > The snapshot available here at - > https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ > , has an incorrect dependency specified for hadoop-core ( in the .pom file > ). > The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in > ivy/libaries.properties but pom.xml published in the repository refers to > 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1352) 0.21.0 - snapshot incorrect dependency published in .pom files
[ https://issues.apache.org/jira/browse/MAPREDUCE-1352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated MAPREDUCE-1352: -- Attachment: mapreduce-1352.patch this patch updates the version in the template pom file > 0.21.0 - snapshot incorrect dependency published in .pom files > --- > > Key: MAPREDUCE-1352 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1352 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.21.0 >Reporter: Kay Kay >Assignee: Giridharan Kesavan >Priority: Critical > Fix For: 0.21.0 > > Attachments: mapreduce-1352.patch > > > The snapshot available here at - > https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-mapred/0.21.0-SNAPSHOT/ > , has an incorrect dependency specified for hadoop-core ( in the .pom file > ). > The source code ( in branch-0.21 ) refers to hadoop-core-0.21 in > ivy/libaries.properties but pom.xml published in the repository refers to > 0.22 . Please fix the same by republishing a .pom again. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
