[jira] [Commented] (MAPREDUCE-4495) Workflow Application Master in YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13427817#comment-13427817 ] Santhosh Srinivasan commented on MAPREDUCE-4495: [~eric14] bq Look at the OpenMPI work... They are proposing to add their AM to MPI. By that logic since this pertains to MapReduce, it should be part of MapReduce as Tom White indicates [~chrismattmann] I don't see how this jira is any different from MAPREDUCE-4393 (PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS) which received a +1 from [~acmurthy] with clear directions to make it a peer of distributed shell (https://issues.apache.org/jira/browse/MAPREDUCE-4393?focusedCommentId=13406813page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13406813) I rest my case. Workflow Application Master in YARN --- Key: MAPREDUCE-4495 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4495 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.0.0-alpha Reporter: Bo Wang Assignee: Bo Wang It is useful to have a workflow application master, which will be capable of running a DAG of jobs. The workflow client submits a DAG request to the AM and then the AM will manage the life cycle of this application in terms of requesting the needed resources from the RM, and starting, monitoring and retrying the application's individual tasks. Compared to running Oozie with the current MapReduce Application Master, these are some of the advantages: - Less number of consumed resources, since only one application master will be spawned for the whole workflow. - Reuse of resources, since the same resources can be used by multiple consecutive jobs in the workflow (no need to request/wait for resources for every individual job from the central RM). - More optimization opportunities in terms of collective resource requests. - Optimization opportunities in terms of rewriting and composing jobs in the workflow (e.g. pushing down Mappers). - This Application Master can be reused/extended by higher systems like Pig and hive to provide an optimized way of running their workflows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-76) JobControl should handle exceptions
[ https://issues.apache.org/jira/browse/MAPREDUCE-76?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13411121#comment-13411121 ] Santhosh Srinivasan commented on MAPREDUCE-76: -- Its been many years since I filed this ticket. From what I can recollect, I had problems when JobControl failed due to non IOExceptions and this resulted in the Pig script failure. This might no longer be relevant. Also, I tried different mechanisms to reproduce this problem and I was not able to. Probably, it was a transient error. Nevertheless, I filed this JIRA to track this issue. If this is no longer relevant, please feel free to resolve this as won't fix or mark it as the duplicate of the JIRA that fixed this issue. JobControl should handle exceptions --- Key: MAPREDUCE-76 URL: https://issues.apache.org/jira/browse/MAPREDUCE-76 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Santhosh Srinivasan If the JobControl encounters non IOExceptions, then JobControl fails without reporting failures. JobControl should handle all exceptions and report the failure to launch jobs. In addition, an API to support the querying of failure to launch jobs should be supported. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-1782) GlobPath support for har
GlobPath support for har Key: MAPREDUCE-1782 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1782 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Affects Versions: 0.20.1 Reporter: Santhosh Srinivasan When a fully qualified path for a har file is used, the FileSystem.globStatus() returns null. Please see the attached test case. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1782) GlobPath support for har
[ https://issues.apache.org/jira/browse/MAPREDUCE-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Santhosh Srinivasan updated MAPREDUCE-1782: --- Attachment: Test.java Test case that returns null while using FileSystem.globStatus() with a fully qualified har file name. GlobPath support for har Key: MAPREDUCE-1782 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1782 Project: Hadoop Map/Reduce Issue Type: Bug Components: harchive Affects Versions: 0.20.1 Reporter: Santhosh Srinivasan Attachments: Test.java When a fully qualified path for a har file is used, the FileSystem.globStatus() returns null. Please see the attached test case. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
