[ https://issues.apache.org/jira/browse/TEZ-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16496968#comment-16496968 ]
Rohini Palaniswamy commented on TEZ-3391: ----------------------------------------- bq. 1) Moving the validation checks to AM Yo can skip this. Looking at the code, it is not a easy thing as the AM will have to deconstruct it to MRInput and then perform the check. It would be wasteful to do that just for this purpose. The validation check is better done by clients like Pig which create the file. They can do it even before submitting the DAG which is even more better. > MR split file validation should be done in the AM > ------------------------------------------------- > > Key: TEZ-3391 > URL: https://issues.apache.org/jira/browse/TEZ-3391 > Project: Apache Tez > Issue Type: Bug > Reporter: Rohini Palaniswamy > Assignee: Nishant Dash > Priority: Major > > We had a case where Split metadata size exceeded 10000000. Instead of job > failing from validation during initialization in AM like mapreduce, each of > the tasks failed doing that validation during initialization. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)