yadav-ankit opened a new pull request, #15499:
URL: https://github.com/apache/iceberg/pull/15499
# Enable Upsert and Delete Support in Apache Iceberg Kafka Connect
## Background
The current implementation of Apache Iceberg Kafka Connect supports
**append-only writes** and does not provide native support for **upsert** or
**delete** operations.
Out-of-the-box, the following writers are used:
* `UnpartitionedWriter<T>`
* `PartitionedFanoutWriter<T>`
These writers support only append-based writes, meaning:
* No row-level updates
* No delete handling
* No delta file generation
---
## Problem Statement
The absence of upsert and delete functionality limits support for:
Change Data Capture (CDC) pipelines
---
## Solution Overview
To enable **upsert** and **delete** operations, delta-aware writers were
introduced by extending the existing writer hierarchy.
### New Classes Added
* `BaseDeltaTaskWriter<Record>`
* `PartitionedDeltaWriter` → extends `BaseDeltaTaskWriter`
* `UnpartitionedDeltaWriter` → extends `BaseDeltaTaskWriter`
These writers generate **delta files** and handle row-level mutations.
---
## Design Changes
### Existing Writer Hierarchy (Append-only)
```
TaskWriter<T>
│
▼
BaseTaskWriter<T>
│
┌──────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
UnpartitionedWriter<T> PartitionedFanoutWriter<T>
(Append-only flow)
```
---
### Extended Writer Hierarchy (With Delta Support)
```
TaskWriter<T>
│
▼
BaseTaskWriter<T>
│
┌──────┼───────────────────────────────┐
│ │ │
▼ ▼ ▼
UnpartitionedWriter<T> PartitionedFanoutWriter<T>
BaseDeltaTaskWriter<Record>
│
┌─────────────────┴─────────────────┐
▼ ▼
UnpartitionedDeltaWriter
PartitionedDeltaWriter
```
---
## Functional Impact
### Before This Change
* Only append operations supported
* Upsert operations not handled
* Delete operations not handled
---
### After This Change
* Upsert operations supported
* Delete operations supported
* Delta files generated via Delta Writers
* Compatible with:
* Partitioned tables
* Unpartitioned tables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
