[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16183639#comment-16183639 ]
Ted Yu commented on HBASE-18885: -------------------------------- Please go ahead with the commit. > HFileOutputFormat2 hardcodes default FileOutputCommitter > -------------------------------------------------------- > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce > Reporter: Shaofeng SHI > Assignee: Shaofeng SHI > Fix For: 1.4.0, 1.5.0, 2.0.0-alpha-4 > > Attachments: HBASE-18885.branch-1.001.patch, > HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v6.4.14#64029)