[ https://issues.apache.org/jira/browse/TEZ-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14312960#comment-14312960 ]
Gopal V commented on TEZ-1993: ------------------------------ The FIleSplit::getLen() along with offset is used to read/locate the blocks on HDFS - that is an MRv1 contract for that API. This patch is playing it safe by adding a new API contract to get a weighted size for the grouping heuristic, instead of modifying an existing (MRv1 is a forced compatibility one) contract. > Implement a pluggable InputSizeEstimator for grouping fairly > ------------------------------------------------------------ > > Key: TEZ-1993 > URL: https://issues.apache.org/jira/browse/TEZ-1993 > Project: Apache Tez > Issue Type: Bug > Affects Versions: 0.7.0 > Reporter: Gopal V > Assignee: Gopal V > Attachments: TEZ-1993.1.patch, TEZ-1993.2.patch > > > Split grouping is currently done using a file size measurement which is the > exact size of the split as it stays at rest on HDFS. > This is not valid for columnar formats and especially suffers from highly > compressible data skews. -- This message was sent by Atlassian JIRA (v6.3.4#6332)