alexkot1394 opened a new pull request, #15498:
URL: https://github.com/apache/iceberg/pull/15498
## Summary
Adds support for the VARIANT data type in the Kafka Connect sink connector.
## Background
When attempting to write records with variant fields to Iceberg tables via
the Kafka Connect sink, the connector would throw:
```
java.lang.UnsupportedOperationException: Unsupported type: VARIANT
at
org.apache.iceberg.connect.data.RecordConverter.convertValue(RecordConverter.java:146)
```
This prevented users from using the Kafka Connect sink with Databricks
tables or any Iceberg tables containing variant columns.
## Changes
- Added `convertVariant()` method to `RecordConverter` that handles
conversion from Kafka Connect data types to Iceberg Variant
- Supports conversion from:
- JSON strings
- Maps (converted to JSON objects)
- Lists (converted to JSON arrays)
- Kafka Connect Structs
- Primitives (numbers, booleans)
- Already-serialized variants (ByteBuffer, byte[])
- Added test coverage in `TestRecordConverter.testVariantConversion()`
## Implementation Details
The variant conversion wraps incoming data as JSON and creates an Iceberg
Variant object with:
- Empty metadata (for simplicity)
- The JSON-serialized value
This approach is consistent with how other complex types are handled in the
connector.
## Testing
- Added unit test `testVariantConversion()` that validates conversion of:
- JSON strings
- Complex nested objects (Maps with Lists)
- Primitive values
- All existing tests pass
- Manually tested with Databricks Unity Catalog tables containing variant
columns
## Related Issues
Closes #15443
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
