[
https://issues.apache.org/jira/browse/NIFI-16069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-16069.
-------------------------------------
Fix Version/s: 2.12.0
Resolution: Fixed
> PutIcebergRecord fails with ClassCastException when writing complex types
> (arrays, maps, nested records)
> --------------------------------------------------------------------------------------------------------
>
> Key: NIFI-16069
> URL: https://issues.apache.org/jira/browse/NIFI-16069
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 2.10.0
> Reporter: Sönke Liebau
> Priority: Minor
> Fix For: 2.12.0
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> PutIcebergRecord fails to write FlowFiles whose schema contains
> complex/nested types like Iceberg list, map, or struct columns.
> RecordConverter only translates top-level scalar values (java.sql
> timestamp/date/time -> java.time) and passes complex values through
> unchanged.
> As a result, values reach Iceberg's Parquet writer in NiFi's native
> representation, which is incompatible with what Iceberg expects:
> * Nested records arrive as org.apache.nifi.serialization.record.MapRecord
> but Iceberg requires org.apache.iceberg.StructLike.
> * Array fields arrive as Object[] but Iceberg's writer requires a
> java.util.Collection.
> * Maps and elements/values nested inside these types are likewise not
> converted (e.g. a date inside an array or map value).
> Because conversion is gated on scalar field types only, records consisting
> solely of complex fields skipp conversion entirely.
> h3. Steps to reproduce
> # Create an Iceberg table with a complex column, e.g. a struct (nested
> record), a list<...> (array), or a map<...>.
> # Configure a PutIcebergRecord processor pointing at that table with a
> matching record reader schema.
> # Send a FlowFile containing a record with a value for the complex column
> (e.g. a nested record for the struct, or an array for the list).
> # Observed: the FlowFile routes to failure with a ClassCastException
> (MapRecord -> StructLike for structs, Object[] -> Collection for arrays).
> # Expected: the record is written to the Iceberg table successfully.
> h3. Root cause
> RecordConverter performs only shallow, scalar-only conversion and has no
> knowledge of the target Iceberg types, so complex values are passed through
> in NiFi's native form.
> h3. Proposed Fix
> Make RecordConverter recursive and Iceberg-schema-aware. Convert
> arrays/collections to List, wrap nested records as Iceberg StructLike
> (DelegatedRecord), and convert maps, recursing into element/key/value types
> so scalar conversions still apply at any depth.
> DelegatedRecord now passes the target StructType into the converter.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
