[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221344#comment-13221344 ] Konstantin Shvachko commented on MAPREDUCE-3837: I've been reviewing this patch, and have a couple of cosmetic comments below. I agree with Alejandro. This is not introducing new feature, it is just enabling already existing feature. There is low risk, since the feature is enabled in a restricted context, that is restarting failed jobs from scratch rather than trying to continue from the point they were terminated. The patch seems to be larger than it actually is, because it is removing the [troubled] logic responsible for resurrecting the job from its history. Besides that it is simple. Take a look, Arun. Cosmetic comments - Several lines are too long - See several tabs - should be spaces - indentation is wrong in couple of places recoveryManager.addJobForRecovery(JobID.forName(fileName)); shouldRecover = true; // enable actual recovery if num-files > 1 - Add spaces after commas in method calls and parameters Otherwise it looks good. > Hadoop 22 Job tracker is not able to recover job in case of crash and after > that no user can submit job. > > > Key: MAPREDUCE-3837 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Fix For: 0.24.0, 0.22.1, 0.23.2 > > Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch, > PATCH-HADOOP-1-MAPREDUCE-3837.patch, PATCH-MAPREDUCE-3837.patch, > PATCH-TRUNK-MAPREDUCE-3837.patch > > > If job tracker is crashed while running , and there were some jobs are > running , so if job tracker's property mapreduce.jobtracker.restart.recover > is true then it should recover the job. > However the current behavior is as follows > jobtracker try to restore the jobs but it can not . And after that jobtracker > closes its handle to hdfs and nobody else can submit job. > Thanks, > Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3875) S-live: Rate operation count for delete is worse than 0.20.204 by 28.8%
[ https://issues.apache.org/jira/browse/MAPREDUCE-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210424#comment-13210424 ] Konstantin Shvachko commented on MAPREDUCE-3875: Harsh> See HDFS-708 on Slive, and org.apache.hadoop.fs.slive.SliveTest. Todd> Which bug? > S-live: Rate operation count for delete is worse than 0.20.204 by 28.8% > --- > > Key: MAPREDUCE-3875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3875 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: benchmarks, mrv2 >Affects Versions: 0.23.1 >Reporter: Vinay Kumar Thota >Priority: Critical > > Rate operation count for delete is worse than 0.20.204.xx by 28.8% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13207198#comment-13207198 ] Konstantin Shvachko commented on MAPREDUCE-3837: +1 The patch looks good. It enables an important feature of automatic job recovery on JT startup. > Hadoop 22 Job tracker is not able to recover job in case of crash and after > that no user can submit job. > > > Key: MAPREDUCE-3837 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Attachments: PATCH-MAPREDUCE-3837.patch, > PATCH-TRUNK-MAPREDUCE-3837.patch > > > If job tracker is crashed while running , and there were some jobs are > running , so if job tracker's property mapreduce.jobtracker.restart.recover > is true then it should recover the job. > However the current behavior is as follows > jobtracker try to restore the jobs but it can not . And after that jobtracker > closes its handle to hdfs and nobody else can submit job. > Thanks, > Mayank -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2539) NPE when calling JobClient.getMapTaskReports for retired job
[ https://issues.apache.org/jira/browse/MAPREDUCE-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195357#comment-13195357 ] Konstantin Shvachko commented on MAPREDUCE-2539: This is needed for Pig unit test to run over 0.22. I ran MR test target. All pass. Will commit it to the branch. > NPE when calling JobClient.getMapTaskReports for retired job > > > Key: MAPREDUCE-2539 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2539 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 0.22.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Attachments: MR-2539-trunk-v1.patch, MR-2539-yarn-v1.patch > > > When calling JobClient.getMapTaskReports for a retired job this results in a > NPE. In the 0.20.* version an empty TaskReport array was returned instead. > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:588) > at > org.apache.pig.tools.pigstats.JobStats.addMapReduceStatistics(JobStats.java:388) > .. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3593) MAPREDUCE Impersonation is not working in 22
[ https://issues.apache.org/jira/browse/MAPREDUCE-3593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183011#comment-13183011 ] Konstantin Shvachko commented on MAPREDUCE-3593: +1 The patch looks good to me. Mayank could you please comment on the status of test run. I will commit it if everything passed. > MAPREDUCE Impersonation is not working in 22 > > > Key: MAPREDUCE-3593 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3593 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.22.0 >Reporter: Mayank Bansal >Assignee: Mayank Bansal > Fix For: 0.22.1 > > Attachments: MAPREDUCE-3593.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3592) DistCp should not setTimes on directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174316#comment-13174316
]
Konstantin Shvachko commented on MAPREDUCE-3592:
This happened in -update mode, when the source and destination turned out to be
in sync.
I think DistCp does not need to update anything at all. But it does and gets
this exception. The exception is confusing, but it means setTimes() is not
supported for directories.
{code}
11/12/20 15:55:49 INFO tools.DistCp: sourcePathsCount(files+directories)=11602
11/12/20 15:55:49 INFO tools.DistCp: filesToCopyCount=0
11/12/20 15:55:49 INFO tools.DistCp: bytesToCopyCount=0
11/12/20 15:55:49 INFO tools.DistCp: filesToSkipCopyCount=5676
11/12/20 15:55:49 INFO tools.DistCp: bytesToSkipCopyCount=5339.3g
11/12/20 15:56:01 INFO tools.DistCp:
deletedPathsFromDestCount(files+directories)=0
11/12/20 15:56:01 INFO tools.DistCp: sourcePathsCount=11602
11/12/20 15:56:01 INFO tools.DistCp: filesToCopyCount=0
11/12/20 15:56:01 INFO tools.DistCp: bytesToCopyCount=0
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.FileNotFoundException: File /my-backup/my-data-input does
not exist.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimes(FSNamesystem.java:1096)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.setTimes(NameNode.java:1269)
at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:349)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1482)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1476)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
at org.apache.hadoop.hdfs.DFSClient.setTimes(DFSClient.java:1347)
at
org.apache.hadoop.hdfs.DistributedFileSystem.setTimes(DistributedFileSystem.java:777)
at org.apache.hadoop.tools.DistCp.updateDestStatus(DistCp.java:827)
at org.apache.hadoop.tools.DistCp.finalize(DistCp.java:853)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:794)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:1028)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:1055)
{code}
> DistCp should not setTimes on directories
> -
>
> Key: MAPREDUCE-3592
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3592
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
>Affects Versions: 0.22.0
>Reporter: Konstantin Shvachko
>
> On finalize() stage DistCp attempts to updateDestStatus(), which includes
> calling setTimes(). In HDFS setTimes() is not supported for directories,
> therefore DistCp fails if there are directories in the tree.
> It looks the failure occurs only when there are no files to copy.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3570) SleepJob is missing from hadoop 0.23 examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-3570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171190#comment-13171190 ] Konstantin Shvachko commented on MAPREDUCE-3570: It has been moved to test. See MAPREDUCE-973. No need to have it in 2 places. > SleepJob is missing from hadoop 0.23 examples > - > > Key: MAPREDUCE-3570 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3570 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: examples, mrv2 >Affects Versions: 0.23.0, 0.23.1 >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan >Priority: Minor > Attachments: MAPREDUCE-3570.patch > > > I have noticed that Sleepjob is missing from the examples in 0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3438) TestRaidNode fails because of "Too many open files"
[ https://issues.apache.org/jira/browse/MAPREDUCE-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158987#comment-13158987 ] Konstantin Shvachko commented on MAPREDUCE-3438: I committed this to branch 0.22. Let's see if it helps. > TestRaidNode fails because of "Too many open files" > --- > > Key: MAPREDUCE-3438 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3438 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/raid >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko > Fix For: 0.22.0 > > Attachments: MAPREDUCE-3438.patch > > > TestRaidNode fails because it opens many connections. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3438) TestRaidNode fails because of "Too many open files"
[
https://issues.apache.org/jira/browse/MAPREDUCE-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158624#comment-13158624
]
Konstantin Shvachko commented on MAPREDUCE-3438:
Thanks, Ram. Couple of questions.
# Does this mean that Raid does not close files / sockets? Do we need to create
a separate jira for that?
# Will it be possible to prevent socket leak in the test by just closing the
file system {{fileSys}} instead of restarting the entire cluster many times,
which increases running time of the test substantially, which is already one of
the longest running?
> TestRaidNode fails because of "Too many open files"
> ---
>
> Key: MAPREDUCE-3438
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3438
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Affects Versions: 0.22.0
>Reporter: Konstantin Shvachko
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-3438.patch
>
>
> TestRaidNode fails because it opens many connections.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3438) TestRaidNode fails because of "Too many open files"
[
https://issues.apache.org/jira/browse/MAPREDUCE-3438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153953#comment-13153953
]
Konstantin Shvachko commented on MAPREDUCE-3438:
This is the last failing test for 0.22. See last several builds for
Hadoop-Mapreduce-22-branch.
The failure is because of the following exception:
{code}
11/11/21 01:05:26 INFO hdfs.DFSClient: Failed to connect to /127.0.0.1:45905,
add to deadNodes and continue
java.net.SocketException: Too many open files
at sun.nio.ch.Net.socket0(Native Method)
at sun.nio.ch.Net.socket(Net.java:97)
at sun.nio.ch.SocketChannelImpl.(SocketChannelImpl.java:84)
at
sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:37)
at java.nio.channels.SocketChannel.open(SocketChannel.java:105)
at
org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:63)
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:702)
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:390)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:514)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:122)
at org.apache.hadoop.raid.RaidUtils.copyBytes(RaidUtils.java:93)
at org.apache.hadoop.raid.Decoder.decodeFile(Decoder.java:133)
at org.apache.hadoop.raid.RaidNode.unRaid(RaidNode.java:867)
at org.apache.hadoop.raid.RaidNode.recoverFile(RaidNode.java:333)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:349)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1482)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1476)
{code}
Which leads to {{BlockMissingException}} and failure of
{{TestRaidNode.testPathFilter}} in the end.
The fix is either
# to increase ulimit on Jenkins machines, which I did on my box and everything
passed, or
# to scale down the test itself.
> TestRaidNode fails because of "Too many open files"
> ---
>
> Key: MAPREDUCE-3438
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3438
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/raid
>Affects Versions: 0.22.0
>Reporter: Konstantin Shvachko
> Fix For: 0.22.0
>
>
> TestRaidNode fails because it opens many connections.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2059) RecoveryManager attempts to add jobtracker.info
[ https://issues.apache.org/jira/browse/MAPREDUCE-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153890#comment-13153890 ] Konstantin Shvachko commented on MAPREDUCE-2059: I just committed this to 0.22 branch. Thanks Subroto. Keeping it open until the inclusion to 0.20.security is decided. > RecoveryManager attempts to add jobtracker.info > --- > > Key: MAPREDUCE-2059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.203.0, 0.22.0 > Environment: > https://svn.apache.org/repos/asf/hadoop/mapreduce/trunk@994941 >Reporter: Dan Adkins > Labels: hadoop > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2059.patch, MAPREDUCE-2059.patch > > > The jobtracker is treating the file 'jobtracker.info' in the system data > directory as a job to be recovered, resulting in the following: > 10/09/09 18:06:02 WARN mapred.JobTracker: Failed to add the job > jobtracker.info > java.lang.IllegalArgumentException: JobId string : jobtracker.info is not > properly formed > at org.apache.hadoop.mapreduce.JobID.forName(JobID.java:158) > at org.apache.hadoop.mapred.JobID.forName(JobID.java:84) > at > org.apache.hadoop.mapred.JobTracker$RecoveryManager.addJobForRecovery(JobTracker.java:1057) > at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1565) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:275) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:267) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:262) > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4256) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2059) RecoveryManager attempts to add jobtracker.info
[ https://issues.apache.org/jira/browse/MAPREDUCE-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153882#comment-13153882 ] Konstantin Shvachko commented on MAPREDUCE-2059: Otherwise code looks good +1. > RecoveryManager attempts to add jobtracker.info > --- > > Key: MAPREDUCE-2059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.203.0, 0.22.0 > Environment: > https://svn.apache.org/repos/asf/hadoop/mapreduce/trunk@994941 >Reporter: Dan Adkins > Labels: hadoop > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2059.patch, MAPREDUCE-2059.patch > > > The jobtracker is treating the file 'jobtracker.info' in the system data > directory as a job to be recovered, resulting in the following: > 10/09/09 18:06:02 WARN mapred.JobTracker: Failed to add the job > jobtracker.info > java.lang.IllegalArgumentException: JobId string : jobtracker.info is not > properly formed > at org.apache.hadoop.mapreduce.JobID.forName(JobID.java:158) > at org.apache.hadoop.mapred.JobID.forName(JobID.java:84) > at > org.apache.hadoop.mapred.JobTracker$RecoveryManager.addJobForRecovery(JobTracker.java:1057) > at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1565) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:275) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:267) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:262) > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4256) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2059) RecoveryManager attempts to add jobtracker.info
[ https://issues.apache.org/jira/browse/MAPREDUCE-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13153336#comment-13153336 ] Konstantin Shvachko commented on MAPREDUCE-2059: I see this problem in 0.22 and I think the fix is right. Unfortunately, the test does not succeed. It loops forever waiting for the job to reach 50% completion which it never does. I would like to commit it to 0.22 if the test is fixed. I see that 0.20.security has the same problem. > RecoveryManager attempts to add jobtracker.info > --- > > Key: MAPREDUCE-2059 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2059 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.20.203.0, 0.22.0 > Environment: > https://svn.apache.org/repos/asf/hadoop/mapreduce/trunk@994941 >Reporter: Dan Adkins > Labels: hadoop > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2059.patch > > > The jobtracker is treating the file 'jobtracker.info' in the system data > directory as a job to be recovered, resulting in the following: > 10/09/09 18:06:02 WARN mapred.JobTracker: Failed to add the job > jobtracker.info > java.lang.IllegalArgumentException: JobId string : jobtracker.info is not > properly formed > at org.apache.hadoop.mapreduce.JobID.forName(JobID.java:158) > at org.apache.hadoop.mapred.JobID.forName(JobID.java:84) > at > org.apache.hadoop.mapred.JobTracker$RecoveryManager.addJobForRecovery(JobTracker.java:1057) > at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1565) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:275) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:267) > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:262) > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4256) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3429) Few contrib tests are failing because of the missing commons-lang dependency
[
https://issues.apache.org/jira/browse/MAPREDUCE-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152951#comment-13152951
]
Konstantin Shvachko commented on MAPREDUCE-3429:
+1
> Few contrib tests are failing because of the missing commons-lang dependency
>
>
> Key: MAPREDUCE-3429
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3429
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/capacity-sched, contrib/gridmix
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-3429.patch
>
>
> As the result of MAPREDUCE-3311 fix a transient {{commons-lang}} isn't
> available anymore to contrib tests. This causing silent failure with timeout.
> The problem is only seeing if tests are ran with {{-Dtest.output=yes}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3311) Bump jetty to 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151537#comment-13151537 ] Konstantin Shvachko commented on MAPREDUCE-3311: I checked TestRaidNode is fixed by adding commons-lang to raid/ivy.xml It was failing in the beginning, but turned out it was due to the ulimit. Increased ulimit - it passes now. +1 on latest patch. > Bump jetty to 6.1.26 > > > Key: MAPREDUCE-3311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3311 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.22.0 > > Attachments: MAPREDUCE-3311.patch, MAPREDUCE-3311.patch, > MAPREDUCE-3311.patch, MAPREDUCE-3311.patch, MAPREDUCE-3311.patch > > > MapReduce part of HADOOP-7450 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3311) Bump jetty to 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150997#comment-13150997 ] Konstantin Shvachko commented on MAPREDUCE-3311: I checked fairscheduler. It also needs common-lang. At least this fixed TestFairSchedulerSystem for me. We should add common-lang on the higher level, may be directly into mapred/ivy.xml > Bump jetty to 6.1.26 > > > Key: MAPREDUCE-3311 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3311 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: build >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Fix For: 0.22.0 > > Attachments: MAPREDUCE-3311.patch, MAPREDUCE-3311.patch, > MAPREDUCE-3311.patch > > > MapReduce part of HADOOP-7450 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3311) Bump jetty to 6.1.26
[
https://issues.apache.org/jira/browse/MAPREDUCE-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150843#comment-13150843
]
Konstantin Shvachko commented on MAPREDUCE-3311:
The following three tests failed. But they also are failing without this patch.
{code}
[junit] Test org.apache.hadoop.streaming.TestUlimit FAILED
[junit] Test org.apache.hadoop.mapred.TestFairSchedulerSystem FAILED
[junit] Test org.apache.hadoop.raid.TestRaidNode FAILED
{code}
So I am going to commit the Jetty upgrade patches.
> Bump jetty to 6.1.26
>
>
> Key: MAPREDUCE-3311
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3311
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: build
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-3311.patch, MAPREDUCE-3311.patch
>
>
> MapReduce part of HADOOP-7450
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1118) Capacity Scheduler scheduling information is hard to read / should be tabular format
[ https://issues.apache.org/jira/browse/MAPREDUCE-1118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13144320#comment-13144320 ] Konstantin Shvachko commented on MAPREDUCE-1118: The test is failing. Run ant -Dtestcase=TestCapacitySchedulerServlet -Dtest.output=yes test-contrib you will see. > Capacity Scheduler scheduling information is hard to read / should be tabular > format > > > Key: MAPREDUCE-1118 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1118 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: Allen Wittenauer > Fix For: 0.20.203.0, 0.22.0 > > Attachments: MR-1118-22.patch, mapred-1118-1.patch, > mapred-1118-2.patch, mapred-1118-3.patch, mapred-1118.20S.patch, > mapred-1118.patch > > > The scheduling information provided by the capacity scheduler is extremely > hard to read on the job tracker web page. Instead of just flat text, it > should be presenting the information in a tabular format, similar to what the > fair share scheduler provides. This makes it much easier to compare what > different queues are doing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3311) Bump jetty to 6.1.26
[
https://issues.apache.org/jira/browse/MAPREDUCE-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142730#comment-13142730
]
Konstantin Shvachko commented on MAPREDUCE-3311:
TestMiniMRChildTask passed for me. Actually all MR tests passed. But
test-contrib is failing with
{code}
[ivy:resolve] ::
[ivy:resolve] :: UNRESOLVED DEPENDENCIES ::
[ivy:resolve] ::
[ivy:resolve] :: org.mortbay.jetty#jsp-api-2.1;6.1.26: not found
[ivy:resolve] :: org.mortbay.jetty#jsp-2.1;6.1.26: not found
[ivy:resolve] ::
{code}
> Bump jetty to 6.1.26
>
>
> Key: MAPREDUCE-3311
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3311
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: build
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-3311.patch
>
>
> MapReduce part of HADOOP-7450
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3286) Unit tests for MAPREDUCE-3186 - User jobs are getting hanged if the Resource manager process goes down and comes up while job is getting executed.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142502#comment-13142502
]
Konstantin Shvachko commented on MAPREDUCE-3286:
This is an example. They are hanging out there since September 15
{code}
jenkins 32743 0.0 0.8 1803048 144544 ? Sl Oct20 3:38
/home/jenkins/tools/java/latest/bin/java -Dlog4j.config
uration=container-log4j.properties
-Dyarn.app.mapreduce.container.log.dir=/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/target/org.apache.hadoop.mapreduce.v2.TestMRJobs/org.apache.hadoop.mapreduce.v2.TestMRJobs-logDir/application_1319147575527_0003/container_1319147575527_0003_01_01
-Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
-Xmx1536m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
{code}
I'll be killing them manually now.
> Unit tests for MAPREDUCE-3186 - User jobs are getting hanged if the Resource
> manager process goes down and comes up while job is getting executed.
> --
>
> Key: MAPREDUCE-3286
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3286
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: mrv2
>Affects Versions: 0.23.0
> Environment: linux
>Reporter: Eric Payne
>Assignee: Eric Payne
> Labels: test
>
> If the resource manager is restarted while the job execution is in progress,
> the job is getting hanged.
> UI shows the job as running.
> In the RM log, it is throwing an error "ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> AppAttemptId doesnt exist in cache appattempt_1318579738195_0004_01"
> In the console MRAppMaster and Runjar processes are not getting killed
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3286) Unit tests for MAPREDUCE-3186 - User jobs are getting hanged if the Resource manager process goes down and comes up while job is getting executed.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142495#comment-13142495 ] Konstantin Shvachko commented on MAPREDUCE-3286: I see many > 70 of those hanging on hadoop7 and breaking other builds. > Unit tests for MAPREDUCE-3186 - User jobs are getting hanged if the Resource > manager process goes down and comes up while job is getting executed. > -- > > Key: MAPREDUCE-3286 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3286 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: mrv2 >Affects Versions: 0.23.0 > Environment: linux >Reporter: Eric Payne >Assignee: Eric Payne > Labels: test > > If the resource manager is restarted while the job execution is in progress, > the job is getting hanged. > UI shows the job as running. > In the RM log, it is throwing an error "ERROR > org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: > AppAttemptId doesnt exist in cache appattempt_1318579738195_0004_01" > In the console MRAppMaster and Runjar processes are not getting killed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3139) SlivePartitioner generates negative partitions
[
https://issues.apache.org/jira/browse/MAPREDUCE-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13141848#comment-13141848
]
Konstantin Shvachko commented on MAPREDUCE-3139:
+1 This solves the problem.
> SlivePartitioner generates negative partitions
> --
>
> Key: MAPREDUCE-3139
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3139
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.22.0
>Reporter: Konstantin Shvachko
>Assignee: Jakob Homan
> Fix For: 0.24.0
>
> Attachments: MR-3139-0.patch
>
>
> {{SlivePartitioner.getPartition()}} returns negative partition numbers on
> some occasions, which is illegal.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2328) memory-related configurations missing from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138108#comment-13138108 ] Konstantin Shvachko commented on MAPREDUCE-2328: I committed this to 0.22 branch. Thank you Harsh. Leaving it open for the action on trunk / 0.23 > memory-related configurations missing from mapred-default.xml > - > > Key: MAPREDUCE-2328 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2328 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Harsh J > Labels: newbie > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2328.r1.diff, MAPREDUCE-2328.r2.diff, > MAPREDUCE-2328.r3.diff > > > HADOOP-5881 added new configuration parameters for memory-based scheduling, > but they weren't added to mapred-default.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2328) memory-related configurations missing from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13135502#comment-13135502 ] Konstantin Shvachko commented on MAPREDUCE-2328: Sorry to come even later. This does apply to 0.22 and would be good to have it in the release. But a patch for trunk is needed. > memory-related configurations missing from mapred-default.xml > - > > Key: MAPREDUCE-2328 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2328 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Harsh J > Labels: newbie > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2328.r1.diff, MAPREDUCE-2328.r2.diff, > MAPREDUCE-2328.r3.diff > > > HADOOP-5881 added new configuration parameters for memory-based scheduling, > but they weren't added to mapred-default.xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3156) Allow TestMRCLI to be run against a cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124727#comment-13124727 ] Konstantin Shvachko commented on MAPREDUCE-3156: +1 > Allow TestMRCLI to be run against a cluster > --- > > Key: MAPREDUCE-3156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3156 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: MAPREDUCE-3156.patch > > > Mapreduce part of HDFS-1762 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120468#comment-13120468 ] Konstantin Shvachko commented on MAPREDUCE-3138: Oh, you are distinguishing between 0.20.*** and any version from 0.21 and up. Will try Pig on 0.22 with this patch now. > Allow for applications to deal with MAPREDUCE-954 > - > > Key: MAPREDUCE-3138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3138.0.22.patch, MAPREDUCE-3138.patch, > MAPREDUCE-3138.patch > > > MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We > need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[
https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13120376#comment-13120376
]
Konstantin Shvachko commented on MAPREDUCE-3138:
Arun, thanks for the tip.
Do you plan to remove {{org.apache.hadoop.mapreduce.task.JobContextImpl}} from
trunk and 0.23. Or is it some Pig build magic that excludes the package from
the classpath?
Otherwise I don't understand how this patch distinguishes between 0.21 and 0.23.
> Allow for applications to deal with MAPREDUCE-954
> -
>
> Key: MAPREDUCE-3138
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, mrv2
>Affects Versions: 0.23.0
>Reporter: Arun C Murthy
>Assignee: Owen O'Malley
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3138.0.22.patch, MAPREDUCE-3138.patch,
> MAPREDUCE-3138.patch
>
>
> MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We
> need a bridge for them to move to 0.23.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2266) JvmManager sleeps between SIGTERM and SIGKILL while holding many TT locks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118576#comment-13118576 ] Konstantin Shvachko commented on MAPREDUCE-2266: Todd. What do I include? If there was a patch I would be happy to. > JvmManager sleeps between SIGTERM and SIGKILL while holding many TT locks > - > > Key: MAPREDUCE-2266 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2266 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller, tasktracker >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > Fix For: 0.22.0 > > > Between sending a task SIGTERM and SIGKILL, the JvmManager will sleep for > sleepTimeBeforeSigKill millis. But in many call heirarchies this is done > while holding important locks like the TT lock and the JvmManagerForType > lock. With the default 5 second sleep, this prevents other tasks from getting > scheduled and reduces scheduling throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2779) JobSplitWriter.java can't handle large job.split file
[ https://issues.apache.org/jira/browse/MAPREDUCE-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117838#comment-13117838 ] Konstantin Shvachko commented on MAPREDUCE-2779: Adjusted the patch for the new trunk. > JobSplitWriter.java can't handle large job.split file > - > > Key: MAPREDUCE-2779 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2779 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Ming Ma >Assignee: Ming Ma > Fix For: 0.22.0 > > Attachments: MAPREDUCE-2779-0.22.patch, MAPREDUCE-2779-trunk.patch, > MAPREDUCE-2779-trunk.patch > > > We use cascading MultiInputFormat. MultiInputFormat sometimes generates big > job.split used internally by hadoop, sometimes it can go beyond 2GB. > In JobSplitWriter.java, the function that generates such file uses 32bit > signed integer to compute offset into job.split. > writeNewSplits > ... > int prevCount = out.size(); > ... > int currCount = out.size(); > writeOldSplits > ... > long offset = out.size(); > ... > int currLen = out.size(); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
