[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gavin updated HBASE-18885: -------------------------- Comment: was deleted (was: A comment with security level 'jira-users' was removed.) > HFileOutputFormat2 hardcodes default FileOutputCommitter > -------------------------------------------------------- > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce > Reporter: Shaofeng SHI > Assignee: Shaofeng SHI > Priority: Major > Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-4, 2.0.0 > > Attachments: HBASE-18885.branch-1.001.patch, > HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v7.6.3#76005)