[ https://issues.apache.org/jira/browse/MAPREDUCE-1802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869490#action_12869490 ]
Joydeep Sen Sarma commented on MAPREDUCE-1802: ---------------------------------------------- thanks. 463 it is. > allow outputcommitters to skip setup/cleanup > -------------------------------------------- > > Key: MAPREDUCE-1802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1802 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Joydeep Sen Sarma > Assignee: Joydeep Sen Sarma > > Job setup and cleanup overheads in our (larger) clusters are very significant > and add to latency for small jobs. It turns out that Hive does not require > job setup and cleanup at all - since all management of output/temporary files > and such is done by the hive client side. So it would be a big win for our > environment (and Hive users in general) if we could skip job cleanup/setup > altogether. > The proposal is to add new calls to OutputCommitter interface (along the > lines of needsTaskCommit()) to optionally allow skipping of setup/cleanup and > for the JT to take these into account while scheduling setup/cleanup. > NullOutputFormat should not need setup/cleanup for example. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.