[jira] [Updated] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-28459: -- Fix Version/s: 2.4.18 2.7.0 3.0.0-beta-2 2.6.1 2.5.9 > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HBASE-28459: --- Labels: pull-request-available (was: ) > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > Labels: pull-request-available > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28459: -- Description: In hadoop3 there's the s3 magic committer which can speed up s3 writes dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in committer as a FileOutputCommitter. This causes a class cast exception when the s3 magic committer is enabled: {code:java} Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} We can cast to PathOutputCommitter instead, but its only available in hadoop3+. So we will need to use reflection to work around this in branch-2. was: In hadoop3 there's the s3 magic committer which can speed up s3 writes dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in committer as a FileOutputCommitter. This causes a class cast exception when the s3 magic committer is enabled: Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter We can cast to PathOutputCommitter instead, but its only available in hadoop3+. So we will need to use reflection to work around this in branch-2. > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > {code:java} > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter {code} > > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HBASE-28459) HFileOutputFormat2 ClassCastException with s3 magic committer
[ https://issues.apache.org/jira/browse/HBASE-28459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Beaudreault updated HBASE-28459: -- Description: In hadoop3 there's the s3 magic committer which can speed up s3 writes dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in committer as a FileOutputCommitter. This causes a class cast exception when the s3 magic committer is enabled: Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter We can cast to PathOutputCommitter instead, but its only available in hadoop3+. So we will need to use reflection to work around this in branch-2. was: In hadoop3 there's the s3 magic committer which can speed up s3 writes dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in committer as a FileOutputCommitter. This causes a class cast exception when the s3 magic committer is enabled: Error: java.lang.ClassCastException: class org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter We can cast to PathOutputCommitter instead, but its only available in hadoop3+. So we will need to use reflection to work around this in branch-2. > HFileOutputFormat2 ClassCastException with s3 magic committer > - > > Key: HBASE-28459 > URL: https://issues.apache.org/jira/browse/HBASE-28459 > Project: HBase > Issue Type: Bug >Reporter: Bryan Beaudreault >Priority: Major > > In hadoop3 there's the s3 magic committer which can speed up s3 writes > dramatically. In HFileOutputFormat2.createRecordWriter we cast the passed in > committer as a FileOutputCommitter. This causes a class cast exception when > the s3 magic committer is enabled: > > Error: java.lang.ClassCastException: class > org.apache.hadoop.fs.s3a.commit.magic.MagicS3GuardCommitter cannot be cast to > class org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter > We can cast to PathOutputCommitter instead, but its only available in > hadoop3+. So we will need to use reflection to work around this in branch-2. -- This message was sent by Atlassian Jira (v8.20.10#820010)