[jira] [Updated] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4371: Status: Patch Available (was: Open) > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371-2.patch, > MAPREDUCE-4371-2.patch, MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4371: Attachment: MAPREDUCE-4371-2.patch Reattaching the patch. > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371-2.patch, > MAPREDUCE-4371-2.patch, MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403791#comment-13403791 ] madhukara phatak commented on MAPREDUCE-4371: - Seems like the earlier commit has to reverted in order to apply the patch. Cancelling patch as of now. > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371-2.patch, > MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4371: Status: Open (was: Patch Available) > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371-2.patch, > MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4371: Status: Patch Available (was: Reopened) > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371-2.patch, > MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak reopened MAPREDUCE-4371: - Reopened to address NPE issues in TestLocalJobControl and TestMapReduceJobControl > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371-2.patch, > MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4371: Attachment: MAPREDUCE-4371-2.patch Updated the patch to address NPE issues in TestLocalJobControl and TestMapReduceJobControl > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Fix For: 3.0.0 > > Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371-2.patch, > MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4371: Attachment: MAPREDUCE-4371-1.patch Updated the patch to fix test case and style issues. > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
[ https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4371: Attachment: MAPREDUCE-4371.patch > Check for cyclic dependencies in Jobcontrol job DAG > --- > > Key: MAPREDUCE-4371 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: 3.0.0 >Reporter: madhukara phatak > Attachments: MAPREDUCE-4371.patch > > > In current implementation of JobControl, whenever there is a cyclic > dependency between the jobs it throws a Stack overflow exception. This jira > adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG
madhukara phatak created MAPREDUCE-4371: --- Summary: Check for cyclic dependencies in Jobcontrol job DAG Key: MAPREDUCE-4371 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1 Affects Versions: 3.0.0 Reporter: madhukara phatak In current implementation of JobControl, whenever there is a cyclic dependency between the jobs it throws a Stack overflow exception. This jira adds a cyclic check to jobcontrol. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3047) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission
[ https://issues.apache.org/jira/browse/MAPREDUCE-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-3047: Attachment: MAPREDUCE-3047-2.patch updated the patch with space and granted the licence for inclusion > FileOutputCommitter throws wrong type of exception when calling abortTask() > to handle a directory without permission > > > Key: MAPREDUCE-3047 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3047 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.24.0 >Reporter: JiangKai >Priority: Trivial > Fix For: 0.24.0 > > Attachments: MAPREDUCE-3047-1.patch, MAPREDUCE-3047-2.patch, > MAPREDUCE-3047.patch > > > When FileOutputCommitter calls abortTask() to create a temp directory, if the > user has no permission to access the directory, or a file with the same name > has existed, of course it will fail, however the system will output the error > information into the log file instead of throwing an exception.As a result, > when the temp directory is needed later, since the temp directory hasn't been > created yet, system will throw an exception to tell user that the temp > directory doesn't exist.In my opinion, the exception is not exact and the > error infomation will confuse users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3968) add support for getNumMapTasks() into mapreduce JobContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-3968: Affects Version/s: (was: 0.22.0) trunk Status: Patch Available (was: Open) > add support for getNumMapTasks() into mapreduce JobContext > -- > > Key: MAPREDUCE-3968 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3968 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: trunk > Environment: hadoop 0.22 >Reporter: Radim Kolar >Priority: Minor > Attachments: MAPREDUCE-3968.patch > > > In old mapred api there was way to query number of mappers: > job.getNumMapTasks()) > No such function exists in new mapreduce api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3968) add support for getNumMapTasks() into mapreduce JobContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-3968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-3968: Attachment: MAPREDUCE-3968.patch added new property. Doesn't need the test case since it just gives nicer interface for the existing configuration property > add support for getNumMapTasks() into mapreduce JobContext > -- > > Key: MAPREDUCE-3968 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3968 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1 >Affects Versions: trunk > Environment: hadoop 0.22 >Reporter: Radim Kolar >Priority: Minor > Attachments: MAPREDUCE-3968.patch > > > In old mapred api there was way to query number of mappers: > job.getNumMapTasks()) > No such function exists in new mapreduce api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4234) SortValidator.java is incompatible with multi-user or parallel use (due to a /tmp file with static name)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4234: Affects Version/s: trunk Status: Patch Available (was: Open) > SortValidator.java is incompatible with multi-user or parallel use (due to a > /tmp file with static name) > > > Key: MAPREDUCE-4234 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4234 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: examples >Affects Versions: 0.23.3, trunk >Reporter: Randy Clayton >Priority: Minor > Attachments: MAPREDUCE-4234.patch > > > The SortValidator.java file checkRecords method creates a file in the > /tmp/sortvalidator directory using a static filename. This can result in > failures due to name collisions when the > hadoop-mapreduce-client-jobclient-*-tests jar is used by more than one task > or one user simultaneously. We use this jar when testing compression codecs > and after we started running tests in parallel (four at a time to reduce > overall test time) we started experiencing random test failures due to name > collisions. Creating a random or unique per thread filename may resolve this > issue. We have developed a change to introduce per use unique file names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4234) SortValidator.java is incompatible with multi-user or parallel use (due to a /tmp file with static name)
[ https://issues.apache.org/jira/browse/MAPREDUCE-4234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-4234: Attachment: MAPREDUCE-4234.patch added UUID to get an unique output name > SortValidator.java is incompatible with multi-user or parallel use (due to a > /tmp file with static name) > > > Key: MAPREDUCE-4234 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4234 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: examples >Affects Versions: 0.23.3, trunk >Reporter: Randy Clayton >Priority: Minor > Attachments: MAPREDUCE-4234.patch > > > The SortValidator.java file checkRecords method creates a file in the > /tmp/sortvalidator directory using a static filename. This can result in > failures due to name collisions when the > hadoop-mapreduce-client-jobclient-*-tests jar is used by more than one task > or one user simultaneously. We use this jar when testing compression codecs > and after we started running tests in parallel (four at a time to reduce > overall test time) we started experiencing random test failures due to name > collisions. Creating a random or unique per thread filename may resolve this > issue. We have developed a change to introduce per use unique file names. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3191) docs for map output compression incorrectly reference SequenceFile
[ https://issues.apache.org/jira/browse/MAPREDUCE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277616#comment-13277616 ] madhukara phatak commented on MAPREDUCE-3191: - Its just a update in javadoc comment which don't require testcases. > docs for map output compression incorrectly reference SequenceFile > -- > > Key: MAPREDUCE-3191 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3191 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Priority: Trivial > Labels: noob > Attachments: MAPREDUCE-3191.patch > > > The documentation currently says that map output compression uses > SequenceFile compression. This hasn't been true in several years, since we > use IFile for intermediate data now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3191) docs for map output compression incorrectly reference SequenceFile
[ https://issues.apache.org/jira/browse/MAPREDUCE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-3191: Status: Patch Available (was: Open) > docs for map output compression incorrectly reference SequenceFile > -- > > Key: MAPREDUCE-3191 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3191 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Priority: Trivial > Labels: noob > Attachments: MAPREDUCE-3191.patch > > > The documentation currently says that map output compression uses > SequenceFile compression. This hasn't been true in several years, since we > use IFile for intermediate data now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3191) docs for map output compression incorrectly reference SequenceFile
[ https://issues.apache.org/jira/browse/MAPREDUCE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated MAPREDUCE-3191: Attachment: MAPREDUCE-3191.patch update the javadoc. > docs for map output compression incorrectly reference SequenceFile > -- > > Key: MAPREDUCE-3191 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3191 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Priority: Trivial > Labels: noob > Attachments: MAPREDUCE-3191.patch > > > The documentation currently says that map output compression uses > SequenceFile compression. This hasn't been true in several years, since we > use IFile for intermediate data now. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
