[jira] [Updated] (HUDI-1607) Decimal handling bug in SparkAvroPostProcessor
[ https://issues.apache.org/jira/browse/HUDI-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1607: -- Status: Open (was: In Progress) > Decimal handling bug in SparkAvroPostProcessor > --- > > Key: HUDI-1607 > URL: https://issues.apache.org/jira/browse/HUDI-1607 > Project: Apache Hudi > Issue Type: Bug >Reporter: Jingwei Zhang >Priority: Major > Labels: core-flow-ds, sev:critical, user-support-issues > > This issue related to > [#[Hudi-1343]|[https://github.com/apache/hudi/pull/2192].] > I think the purpose of Hudi-1343 was to bridge the difference between avro > 1.8.2(used by hudi) and avro 1.9.2(used by upstream system) thru internal > Struct type. In particular, the incompatible form to express nullable type > between those two versions. > It was all good until I hit the type Decimal. Since it can either be FIXED or > BYTES, if an avro schema contains decimal type with BYTES as its literal > type, after this two way conversion its literal type become FIXED instead. > This will cause an exception to be thrown in AvroConversionHelper as the data > underneath is HeapByteBuffer rather than GenericFixed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1607) Decimal handling bug in SparkAvroPostProcessor
[ https://issues.apache.org/jira/browse/HUDI-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1607: -- Labels: core-flow-ds sev:critical user-support-issues (was: sev:critical user-support-issues) > Decimal handling bug in SparkAvroPostProcessor > --- > > Key: HUDI-1607 > URL: https://issues.apache.org/jira/browse/HUDI-1607 > Project: Apache Hudi > Issue Type: Bug >Reporter: Jingwei Zhang >Priority: Major > Labels: core-flow-ds, sev:critical, user-support-issues > > This issue related to > [#[Hudi-1343]|[https://github.com/apache/hudi/pull/2192].] > I think the purpose of Hudi-1343 was to bridge the difference between avro > 1.8.2(used by hudi) and avro 1.9.2(used by upstream system) thru internal > Struct type. In particular, the incompatible form to express nullable type > between those two versions. > It was all good until I hit the type Decimal. Since it can either be FIXED or > BYTES, if an avro schema contains decimal type with BYTES as its literal > type, after this two way conversion its literal type become FIXED instead. > This will cause an exception to be thrown in AvroConversionHelper as the data > underneath is HeapByteBuffer rather than GenericFixed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-1607) Decimal handling bug in SparkAvroPostProcessor
[ https://issues.apache.org/jira/browse/HUDI-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1607: -- Status: In Progress (was: Open) > Decimal handling bug in SparkAvroPostProcessor > --- > > Key: HUDI-1607 > URL: https://issues.apache.org/jira/browse/HUDI-1607 > Project: Apache Hudi > Issue Type: Bug >Reporter: Jingwei Zhang >Priority: Major > Labels: sev:critical, user-support-issues > > This issue related to > [#[Hudi-1343]|[https://github.com/apache/hudi/pull/2192].] > I think the purpose of Hudi-1343 was to bridge the difference between avro > 1.8.2(used by hudi) and avro 1.9.2(used by upstream system) thru internal > Struct type. In particular, the incompatible form to express nullable type > between those two versions. > It was all good until I hit the type Decimal. Since it can either be FIXED or > BYTES, if an avro schema contains decimal type with BYTES as its literal > type, after this two way conversion its literal type become FIXED instead. > This will cause an exception to be thrown in AvroConversionHelper as the data > underneath is HeapByteBuffer rather than GenericFixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1607) Decimal handling bug in SparkAvroPostProcessor
[ https://issues.apache.org/jira/browse/HUDI-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-1607: -- Labels: sev:critical user-support-issues (was: ) > Decimal handling bug in SparkAvroPostProcessor > --- > > Key: HUDI-1607 > URL: https://issues.apache.org/jira/browse/HUDI-1607 > Project: Apache Hudi > Issue Type: Bug >Reporter: Jingwei Zhang >Priority: Major > Labels: sev:critical, user-support-issues > > This issue related to > [#[Hudi-1343]|[https://github.com/apache/hudi/pull/2192].] > I think the purpose of Hudi-1343 was to bridge the difference between avro > 1.8.2(used by hudi) and avro 1.9.2(used by upstream system) thru internal > Struct type. In particular, the incompatible form to express nullable type > between those two versions. > It was all good until I hit the type Decimal. Since it can either be FIXED or > BYTES, if an avro schema contains decimal type with BYTES as its literal > type, after this two way conversion its literal type become FIXED instead. > This will cause an exception to be thrown in AvroConversionHelper as the data > underneath is HeapByteBuffer rather than GenericFixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HUDI-1607) Decimal handling bug in SparkAvroPostProcessor
[ https://issues.apache.org/jira/browse/HUDI-1607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingwei Zhang updated HUDI-1607: Description: This issue related to [#[Hudi-1343]|[https://github.com/apache/hudi/pull/2192].] I think the purpose of Hudi-1343 was to bridge the difference between avro 1.8.2(used by hudi) and avro 1.9.2(used by upstream system) thru internal Struct type. In particular, the incompatible form to express nullable type between those two versions. It was all good until I hit the type Decimal. Since it can either be FIXED or BYTES, if an avro schema contains decimal type with BYTES as its literal type, after this two way conversion its literal type become FIXED instead. This will cause an exception to be thrown in AvroConversionHelper as the data underneath is HeapByteBuffer rather than GenericFixed. was: This issue related to [#[Hudi-1343]|[https://github.com/apache/hudi/pull/2192].] I think the purpose of Hudi-1343 was to bridge the different between avro 1.8.2(used by hudi) and avro 1.9.2(used by upstream system) thru internal Struct type. In particular, the incompatible form to express nullable type between those two versions. It was all good until I hit the type Decimal. Since it can either be FIXED or BYTES, if an avro schema contains decimal type with BYTES as its literal type, after this two way conversion its literal type become FIXED instead. This will cause an exception to be thrown in AvroConversionHelper as the data underneath is ByteBuffer rather than GenericFixed. > Decimal handling bug in SparkAvroPostProcessor > --- > > Key: HUDI-1607 > URL: https://issues.apache.org/jira/browse/HUDI-1607 > Project: Apache Hudi > Issue Type: Bug >Reporter: Jingwei Zhang >Priority: Major > > This issue related to > [#[Hudi-1343]|[https://github.com/apache/hudi/pull/2192].] > I think the purpose of Hudi-1343 was to bridge the difference between avro > 1.8.2(used by hudi) and avro 1.9.2(used by upstream system) thru internal > Struct type. In particular, the incompatible form to express nullable type > between those two versions. > It was all good until I hit the type Decimal. Since it can either be FIXED or > BYTES, if an avro schema contains decimal type with BYTES as its literal > type, after this two way conversion its literal type become FIXED instead. > This will cause an exception to be thrown in AvroConversionHelper as the data > underneath is HeapByteBuffer rather than GenericFixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
