[jira] [Updated] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nihal Jain updated HBASE-18885: --- Labels: s3 (was: ) > HFileOutputFormat2 hardcodes default FileOutputCommitter > > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Labels: s3 > Fix For: 1.4.0, 1.3.2, 2.0.0-alpha-4, 2.0.0, 1.2.7 > > Attachments: HBASE-18885.branch-1.001.patch, > HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-18885: Fix Version/s: 1.2.7 1.3.2 > HFileOutputFormat2 hardcodes default FileOutputCommitter > > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: 1.4.0, 1.3.2, 1.2.7, 2.0.0-alpha-4 > > Attachments: HBASE-18885.branch-1.001.patch, > HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18885: --- Fix Version/s: (was: 1.5.0) > HFileOutputFormat2 hardcodes default FileOutputCommitter > > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: 1.4.0, 2.0.0-alpha-4 > > Attachments: HBASE-18885.branch-1.001.patch, > HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18885: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0-alpha-4 1.5.0 1.4.0 Status: Resolved (was: Patch Available) Thanks for the patch, shaofeng > HFileOutputFormat2 hardcodes default FileOutputCommitter > > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: 1.4.0, 1.5.0, 2.0.0-alpha-4 > > Attachments: HBASE-18885.branch-1.001.patch, > HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated HBASE-18885: - Attachment: HBASE-18885.branch-1.001.patch > HFileOutputFormat2 hardcodes default FileOutputCommitter > > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Attachments: HBASE-18885.branch-1.001.patch, > HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-18885: --- Status: Patch Available (was: Open) > HFileOutputFormat2 hardcodes default FileOutputCommitter > > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Shaofeng SHI > Attachments: HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-18885) HFileOutputFormat2 hardcodes default FileOutputCommitter
[ https://issues.apache.org/jira/browse/HBASE-18885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated HBASE-18885: - Attachment: HBASE-18885.master.001.patch > HFileOutputFormat2 hardcodes default FileOutputCommitter > > > Key: HBASE-18885 > URL: https://issues.apache.org/jira/browse/HBASE-18885 > Project: HBase > Issue Type: Bug > Components: mapreduce >Reporter: Shaofeng SHI > Attachments: HBASE-18885.master.001.patch > > > Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. > The original reporting is in KYLIN-2788[1]. After some investigation, we > found this class always uses the default "FileOutputCommitter", see [2], > regardless of the job's configuration; so it always writing to "_temporary" > folder. Since AWS EMR configured to use DirectOutputCommitter for S3, then > this problem occurs: Hadoop expects to see the file directly under output > path, while the RecordWriter generates them in "_temporary" folder. This > caused no data be loaded to HTable. > Seems this problem exists in all versions so far. > [1] https://issues.apache.org/jira/browse/KYLIN-2788 > [2] > https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
