[jira] [Updated] (HUDI-2958) Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type.
[ https://issues.apache.org/jira/browse/HUDI-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2958: -- Fix Version/s: 0.10.1 > Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert > to insert data which contains decimal Type. > > > Key: HUDI-2958 > URL: https://issues.apache.org/jira/browse/HUDI-2958 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Minor > Labels: pull-request-available, query-eng, sev:high > Fix For: 0.11.0, 0.10.1 > > > Now by default ParquetWriteSupport will write DecimalType to parquet as > int32/int64 when the scale of decimalType < Decimal.MAX_LONG_DIGITS(), > but AvroParquetReader which used by HoodieParquetReader cannot support read > int32/int64 as DecimalType. this will lead follow error > Caused by: java.lang.UnsupportedOperationException: > org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary > at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:41) > at > org.apache.parquet.avro.AvroConverters$BinaryConverter.setDictionary(AvroConverters.java:75) > .. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2958) Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type.
[ https://issues.apache.org/jira/browse/HUDI-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2958: -- Status: Resolved (was: Patch Available) > Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert > to insert data which contains decimal Type. > > > Key: HUDI-2958 > URL: https://issues.apache.org/jira/browse/HUDI-2958 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: tao meng >Assignee: tao meng >Priority: Minor > Labels: pull-request-available, query-eng, sev:high > Fix For: 0.11.0 > > > Now by default ParquetWriteSupport will write DecimalType to parquet as > int32/int64 when the scale of decimalType < Decimal.MAX_LONG_DIGITS(), > but AvroParquetReader which used by HoodieParquetReader cannot support read > int32/int64 as DecimalType. this will lead follow error > Caused by: java.lang.UnsupportedOperationException: > org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary > at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:41) > at > org.apache.parquet.avro.AvroConverters$BinaryConverter.setDictionary(AvroConverters.java:75) > .. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2958) Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type.
[ https://issues.apache.org/jira/browse/HUDI-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2958: -- Status: In Progress (was: Open) > Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert > to insert data which contains decimal Type. > > > Key: HUDI-2958 > URL: https://issues.apache.org/jira/browse/HUDI-2958 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: tao meng >Priority: Minor > Labels: pull-request-available, query-eng, sev:high > Fix For: 0.11.0 > > > Now by default ParquetWriteSupport will write DecimalType to parquet as > int32/int64 when the scale of decimalType < Decimal.MAX_LONG_DIGITS(), > but AvroParquetReader which used by HoodieParquetReader cannot support read > int32/int64 as DecimalType. this will lead follow error > Caused by: java.lang.UnsupportedOperationException: > org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary > at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:41) > at > org.apache.parquet.avro.AvroConverters$BinaryConverter.setDictionary(AvroConverters.java:75) > .. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2958) Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type.
[ https://issues.apache.org/jira/browse/HUDI-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2958: -- Status: Patch Available (was: In Progress) > Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert > to insert data which contains decimal Type. > > > Key: HUDI-2958 > URL: https://issues.apache.org/jira/browse/HUDI-2958 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: tao meng >Priority: Minor > Labels: pull-request-available, query-eng, sev:high > Fix For: 0.11.0 > > > Now by default ParquetWriteSupport will write DecimalType to parquet as > int32/int64 when the scale of decimalType < Decimal.MAX_LONG_DIGITS(), > but AvroParquetReader which used by HoodieParquetReader cannot support read > int32/int64 as DecimalType. this will lead follow error > Caused by: java.lang.UnsupportedOperationException: > org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary > at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:41) > at > org.apache.parquet.avro.AvroConverters$BinaryConverter.setDictionary(AvroConverters.java:75) > .. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2958) Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type.
[ https://issues.apache.org/jira/browse/HUDI-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2958: -- Labels: pull-request-available query-eng sev:high (was: pull-request-available) > Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert > to insert data which contains decimal Type. > > > Key: HUDI-2958 > URL: https://issues.apache.org/jira/browse/HUDI-2958 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: tao meng >Priority: Minor > Labels: pull-request-available, query-eng, sev:high > Fix For: 0.11.0 > > > Now by default ParquetWriteSupport will write DecimalType to parquet as > int32/int64 when the scale of decimalType < Decimal.MAX_LONG_DIGITS(), > but AvroParquetReader which used by HoodieParquetReader cannot support read > int32/int64 as DecimalType. this will lead follow error > Caused by: java.lang.UnsupportedOperationException: > org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary > at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:41) > at > org.apache.parquet.avro.AvroConverters$BinaryConverter.setDictionary(AvroConverters.java:75) > .. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2958) Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type.
[ https://issues.apache.org/jira/browse/HUDI-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2958: - Labels: pull-request-available (was: ) > Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert > to insert data which contains decimal Type. > > > Key: HUDI-2958 > URL: https://issues.apache.org/jira/browse/HUDI-2958 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: tao meng >Priority: Minor > Labels: pull-request-available > Fix For: 0.11.0 > > > Now by default ParquetWriteSupport will write DecimalType to parquet as > int32/int64 when the scale of decimalType < Decimal.MAX_LONG_DIGITS(), > but AvroParquetReader which used by HoodieParquetReader cannot support read > int32/int64 as DecimalType. this will lead follow error > Caused by: java.lang.UnsupportedOperationException: > org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary > at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:41) > at > org.apache.parquet.avro.AvroConverters$BinaryConverter.setDictionary(AvroConverters.java:75) > .. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2958) Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type.
[ https://issues.apache.org/jira/browse/HUDI-2958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tao meng updated HUDI-2958: --- Summary: Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert to insert data which contains decimal Type. (was: Automatically set spark.sql.parquet.writelegacyformat. When using bulkinsert to insert data will contains decimal Type.) > Automatically set spark.sql.parquet.writelegacyformat; When using bulkinsert > to insert data which contains decimal Type. > > > Key: HUDI-2958 > URL: https://issues.apache.org/jira/browse/HUDI-2958 > Project: Apache Hudi > Issue Type: Improvement > Components: Spark Integration >Reporter: tao meng >Priority: Minor > Fix For: 0.11.0 > > > Now by default ParquetWriteSupport will write DecimalType to parquet as > int32/int64 when the scale of decimalType < Decimal.MAX_LONG_DIGITS(), > but AvroParquetReader which used by HoodieParquetReader cannot support read > int32/int64 as DecimalType. this will lead follow error > Caused by: java.lang.UnsupportedOperationException: > org.apache.parquet.column.values.dictionary.PlainValuesDictionary$PlainIntegerDictionary > at org.apache.parquet.column.Dictionary.decodeToBinary(Dictionary.java:41) > at > org.apache.parquet.avro.AvroConverters$BinaryConverter.setDictionary(AvroConverters.java:75) > .. -- This message was sent by Atlassian Jira (v8.20.1#820001)
