[
https://issues.apache.org/jira/browse/NIFI-15866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18075171#comment-18075171
]
Sönke Liebau commented on NIFI-15866:
-------------------------------------
[~exceptionfactory] - I think you were heavily involved in the fixes for
https://issues.apache.org/jira/browse/NIFI-15568 , this might be right up your
alley :)
> Inserting Date values in Iceberg tables results in Exception
> ------------------------------------------------------------
>
> Key: NIFI-15866
> URL: https://issues.apache.org/jira/browse/NIFI-15866
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.9.0
> Reporter: Sönke Liebau
> Priority: Minor
>
> The PutIcebergRecord processor currently does not support columns with type
> Date.
> When inserting in a table with a Date column, each time a ClassCastException
> between java.sql.Date (from the Flowfile) and java.time.LocalDate (required
> from PutIcebergRecord) is thrown.
> This behavior occurs with Avro as well as Parquet and their specific readers
> configured in PutIcebergRecord, with a hardcorded Avro Schema referencing the
> Date column.
>
> This bug is very similar to an already fixed bug regarding
> Datetimes/Timestamps: https://issues.apache.org/jira/browse/NIFI-15568.
>
> *How to reproduce*
> * Create an Iceberg table with one Date column
> * Generate a record flowfile with GenerateFlowFile containing a value for
> this column, for example as CSV or JSON. Give an explicit Avro Schema as
> attribute in the GenerateFlowfile:
>
> +CSV:+
>
> {code:java}
> dateCol
> 234234
> {code}
>
>
> +Avro Schema:+
>
> {code:java}
> {
> "type": "record",
> "name": "Document",
> "namespace": "com.example",
> "fields": [
> {
> "name": "dateCol",
> "type": [
> "null",
> {
> "type": "int",
> "logicalType": "date"
> }
> ]
> }
> ]
> }{code}
>
> * Use a ConvertRecord to transform the CSV to an Avro flowfile, using a
> AvroRecordSetWriter with setting „Use ‚Schema Text‘ Property“
> * Write into Iceberg with PutIcebergRecord and a default AvroReader
>
> Stacktrace:
> {code:java}
> PutIcebergRecord[id=8e5f6f95-9ec0-37d7-ad34-9aa4500355f7] Write Rows to Table
> [xyz.test] failed FlowFile[filename=b13a8af8-851e-4f00-93dd-69f40a36e7b5]:
> java.lang.ClassCastException: class java.sql.Date cannot be cast to class
> java.time.LocalDate (java.sql.Date is in module java.sql of loader
> 'platform'; java.time.LocalDate is in module java.base of loader 'bootstrap')
> java.lang.ClassCastException: class java.sql.Date cannot be cast to class
> java.time.LocalDate (java.sql.Date is in module java.sql of loader
> 'platform'; java.time.LocalDate is in module java.base of loader
> 'bootstrap')at
> org.apache.iceberg.data.parquet.GenericParquetWriter$DateWriter.write(GenericParquetWriter.java:91)at
>
> org.apache.iceberg.parquet.ParquetValueWriters$OptionWriter.write(ParquetValueWriters.java:421)at
>
> org.apache.iceberg.parquet.ParquetValueWriters$StructWriter.write(ParquetValueWriters.java:665)at
> org.apache.iceberg.parquet.ParquetWriter.add(ParquetWriter.java:138)at
> org.apache.iceberg.io.DataWriter.write(DataWriter.java:71)at
> org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.write(BaseTaskWriter.java:401)at
>
> org.apache.iceberg.io.BaseTaskWriter$RollingFileWriter.write(BaseTaskWriter.java:384)at
>
> org.apache.iceberg.io.BaseTaskWriter$BaseRollingWriter.write(BaseTaskWriter.java:311)at
>
> org.apache.iceberg.io.UnpartitionedWriter.write(UnpartitionedWriter.java:42)at
>
> org.apache.nifi.services.iceberg.parquet.io.ParquetIcebergRowWriter.write(ParquetIcebergRowWriter.java:39)at
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown
> Source)at java.base/java.lang.reflect.Method.invoke(Unknown Source)at
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:251)at
>
> org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:237)at
> jdk.proxy515/jdk.proxy515.$Proxy706.write(Unknown Source)at
> org.apache.nifi.processors.iceberg.PutIcebergRecord.writeRecords(PutIcebergRecord.java:240)at
>
> org.apache.nifi.processors.iceberg.PutIcebergRecord.processFlowFiles(PutIcebergRecord.java:176)at
>
> org.apache.nifi.processors.iceberg.PutIcebergRecord.onTrigger(PutIcebergRecord.java:156)at
>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)at
>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1274)at
>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:229)at
>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)at
> org.apache.nifi.engine.FlowEngine.lambda$wrap$1(FlowEngine.java:105)at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown
> Source)at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown
> Source)at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source)at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)at java.base/java.lang.Thread.run(Unknown Source) {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
